Item management system, item management method, and program

Abstract

To provide an article management system capable of suppressing an article from reducing forgetting to use items to improve user's convenience, an article management method and a program.SOLUTION: An article management system comprises: an imaging device that images the inside of a storage chamber of a storage to generate a captured image, and detects, based on the captured image, an article to be stored in the storage chamber, a storage area in the storage chamber in which the article is stored, and an article to be taken out from the storage area; and a control device that causes an information processing device to display information on articles that are not taken out of the storage area even after a preset period has elapsed from a storage date, based on the detection result of the imaging device.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] The present disclosure relates to an article management system, an article management method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique in which a camera is provided above a refrigerator, photographing is executed when an object passing through the opening of the refrigerator is detected, and food stored in the refrigerator is managed based on the photographing result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides an article management system, an article management method, and a program that suppress forgetting to use articles and improve user convenience.

Means for Solving the Problems

[0005] The present disclosure of The article management system includes an imaging device that images a storage chamber of a storage and generates an imaging image, and based on the imaging image, detects an article stored in the storage chamber, a storage area in the storage chamber where the article is stored, and an article shipped out from the storage area; and a control device that causes an information processing device to display information on an article that has not been shipped out from the storage area even after a preset period has elapsed since the article was stored in the storage chamber, based on the detection result of the imaging device. The control device counts the number of times items are brought in and taken out of storage for each storage area in the storage room during a set period based on the image captured by the imaging device, calculates the ratio of items brought in and taken out during the set period for each storage area, and displays a storage status screen on the information processing device that shows a period change button that allows the set period to be changed, and the ratio of items brought in and taken out for each storage area. The storage ratio is calculated to correspond to the set period selected by the period change button and is changed according to the set period.

[0006] ​​​The item management method includes the steps of: taking an image of the inside of the storage room of a storage facility; detecting, based on the captured image, items to be placed in the storage room, the storage area within the storage room where the items are stored, and items to be taken out from the storage area; Based on the captured image from the imaging device, the number of times items are brought in and taken out during a set period is counted for each storage area in the storage room, and the ratio of items being brought in and taken out during the set period is calculated for each storage area. The process includes the step of displaying information on an information processing device about items that have not been removed from the storage area even after a predetermined period has elapsed since being placed in the storage room. The step of displaying the information is to display a screen on the information processing device showing the loading / unloading status, which includes a period change button that allows the setting period to be changed, and the loading / unloading ratio of the items for each storage area. The loading / unloading ratio is calculated based on the setting period selected by the period change button and is changed according to the setting period.

[0007] This disclosure of The program includes a procedure for a computer to detect, based on captured images of the inside of a storage room, items to be placed into the storage room, the storage area within the storage room where the items are placed, and items to be removed from the storage area. A procedure for calculating the ratio of items being moved in and out for each storage area in the storage room, based on the captured image of the imaging device, during a set period, The procedure involves displaying information on an information processing device about items that have not been removed from the storage area even after a predetermined period has elapsed since being placed in the storage facility. 、 The procedure for displaying the above is: The information processing device displays a screen showing the loading / unloading status, which includes a period change button that allows the setting period to be changed, and the loading / unloading ratio of the items for each storage area. The loading / unloading ratio is calculated based on the setting period selected by the period change button and is changed according to the setting period. [Effects of the Invention]

[0008] The item management system, item management method, and program described in this disclosure display information on items that have not been removed from their storage area even after a predetermined period has elapsed since the date of receipt on the information processing device. This helps to prevent items from being forgotten and reduces item loss. Therefore, it can improve user convenience. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example of the system configuration of an inventory management system. [Figure 2] A diagram showing an example of the configuration of an identification information database. [Figure 3] A diagram showing an example of the structure of a food ingredient management database. [Figure 4] A diagram showing an example of the configuration of an inventory management database. [Figure 5]Figure showing an example of the configuration of the open state management database [Figure 6] Front view of the refrigerator [Figure 7] Block diagram showing the configuration of the imaging device [Figure 8] Top view of the refrigerator [Figure 9] Figure showing an example of the captured image of the camera for the refrigerating chamber [Figure 10] Figure showing an example of the captured image of the drawer camera [Figure 11] Figure showing the boundary lines of the first region, second region, third region and fourth region [Figure 12] Figure for explaining the incoming / outgoing determination image [Figure 13] Flowchart showing the operation of the imaging device [Figure 14] Flowchart showing the operation of the article management system [Figure 15] Flowchart showing the operation of the article management server [Figure 16] Figure showing an example of the initial screen [Figure 17] Figure showing another example of the initial screen [Figure 18] Figure showing the long-term storage incoming / outgoing status screen [Figure 19] Figure showing the incoming / outgoing status screen [Figure 20] Figure showing the opening / closing status screen [Figure 21] Figure showing the selection screen [Figure 22] Figure showing the stored article screen [Figure 23] Figure showing the stored article screen [Figure 24] Flowchart showing the operation of the app execution part [Figure 25] Figure which is the captured image of the drawer camera and shows the boundary lines of the fifth region, sixth region, seventh region and eighth region

Mode for Carrying Out the Invention

[0010] (Knowledge, etc. on which the present disclosure is based) At the time the inventors conceived of this disclosure, there was a technology that used a camera to image storage compartments such as refrigerators and to display information about the items stored in the compartment from the captured images. However, the inventors discovered that conventional technology did not manage the time elapsed since the items were stored in the compartment, leading to problems such as forgetting to use food items and resulting in food waste. To solve this problem, they came to form the subject of this disclosure. Therefore, this disclosure provides an item management system, an item management method, and a program that reduce the loss of items by preventing items stored in a storage facility from being forgotten or forgotten.

[0011] The embodiments will be described in detail below with reference to the drawings. However, some unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. The attached drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) [1. System Configuration] Figure 1 shows an example of the system configuration of the item management system 1. The item management system 1 comprises an imaging device 3, an information processing device 6, and an item management server 8. The imaging device 3, the information processing device 6, and the item management server 8 are interconnected via a network 10. The imaging device 3 and the information processing device 6 are connected to the network 10 by wireless connection, but a wired connection may also be used.

[0013] Figure 1 shows one imaging device 3 and one information processing device 6, but multiple imaging devices 3 and information processing devices 6 are provided in proportion to the number of users registered in the item management system 1. Furthermore, the number of imaging devices 3 and information processing devices 6 are not necessarily equal; for example, multiple users in a household can each use an information processing device 6 and view images and information provided by the item management server 8 on the information processing device 6.

[0014] The imaging device 3 is installed on the top surface of the refrigerator 2. The imaging device 3 is equipped with multiple cameras and, when the door or drawers of the refrigerator 2 are opened, it images the inside of the refrigerator compartment, vegetable compartment, etc., and generates images of items being placed into the refrigerator 2 and items being taken out of the refrigerator 2. The images captured by the cameras may be moving images or still images. Items being placed into the refrigerator 2 or taken out of the refrigerator 2 include food, ingredients, and storage containers containing food or ingredients.

[0015] The imaging device 3 analyzes the generated captured images to detect items and marker IDs attached to the items. The storage containers are marked with markers, and the markers contain marker IDs, which are the identification information for the markers. The imaging device 3 generates upload data including the captured images and marker IDs, and sends the generated upload data to the item management server 8.

[0016] The item management server 8 manages the items stored in the refrigerator 2 based on the uploaded data received from the imaging device 3. For example, the item management server 8 manages the date and time when an item is placed in the refrigerator 2, and the elapsed time since the item was placed in the refrigerator 2. The item management server 8 also determines the number of times the door of the refrigerator 2 has been opened and closed, and the storage area of ​​the refrigerator 2 where an item has been stored for a certain period of time or longer. The item management server 8 transmits this information to the information processing device 6, which displays it on the touch panel 63 of the information processing device 6.

[0017] The information processing device 6 is a portable computer such as a smartphone or tablet computer. Alternatively, a notebook or desktop personal computer may be used as the information processing device 6. For example, the information processing device 6 downloads images and information selected by the user from the item management server 8 and generates an application screen 700 based on the downloaded images and information. The information processing device 6 displays the generated application screen 700 on the touch panel 63.

[0018] [2. Configuration of the Information Processing Device] The information processing device 6 comprises a first communication module 61, a touch panel 63, and a first control unit 65.

[0019] The first communication module 61 is a communication interface having an interface circuit that conforms to a wireless communication standard. The first communication module 61 communicates with the item management server 8 under the control of the first control unit 65. In this embodiment, the communication standard used by the first communication module 61 is exemplified as a wireless communication standard, but a wired communication standard may also be used.

[0020] The touch panel 63 comprises a display panel such as a liquid crystal display panel and a touch sensor. The touch panel 63 outputs coordinate information indicating the position of the touched display panel to the first control unit 65.

[0021] The first control unit 65 is a computer device comprising a first memory 67 and a first processor 69.

[0022] The first memory 67 comprises ROM (Read Only Memory) and RAM (Random Access Memory). Illustrations of ROM and RAM are omitted. ROM stores the item management application program 675 and setting information for the information processing device 6 as control programs executed by the first processor 69. Hereafter, the application program will be referred to as "app". RAM is used as the arithmetic area for the first processor 69.

[0023] The first processor 69 is an arithmetic processing unit such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The first processor 69 may consist of a single processor or multiple processors. The first processor 69 may also consist of a System-on-a-chip (SoC) integrated with part or all of the first memory 67 or other circuits. The first processor 69 may also consist of a combination of a CPU that executes programs and a DSP that performs predetermined arithmetic processing. Furthermore, the first processor 69 may be configured with all its functions implemented in hardware, or it may be configured using programmable devices.

[0024] The first processor 69 functions as an application execution unit 695 by executing the item management application 675. The application execution unit 695 communicates with the item management server 8 via the first communication module 61 and receives data for the application screen 700 to be displayed on the touch panel 63.

[0025] Furthermore, the application execution unit 695 displays the application screen 700 on the touch panel 63 as a user interface for managing items stored in the refrigerator 2. The application execution unit 695 displays the application screen 700 on the touch panel 63, provides the user with various information about the items stored in the refrigerator 2, and also accepts various inputs from the user regarding the items stored in the refrigerator 2.

[0026] [3. Configuration of the inventory management server] The item management server 8 comprises a second communication module 81 and a second control unit 83. The item management server 8 corresponds to a control device.

[0027] The second communication module 81 is a communication interface equipped with an interface circuit and the like that conforms to a predetermined communication standard. The second communication module 81 is wired to the network 10 and communicates data with the imaging device 3 and the information processing device 6.

[0028] The second control unit 83 includes a second memory 85 and a second processor 87.

[0029] The second memory 85 includes ROM, RAM, and auxiliary storage devices such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). Illustrations of the ROM, RAM, and HDD are omitted. The ROM stores the control program 851. The RAM is used as the arithmetic area for the second processor 87.

[0030] The auxiliary storage device stores an identification information database 91, a food ingredient management database 93, an item management database 95, and an open status management database 97. Hereafter, databases will be abbreviated as DB.

[0031] The second processor 87 is an arithmetic processing unit such as a CPU or MPU. The second processor 87 may be composed of a single processor or multiple processors. The second processor 87 may also be composed of part or all of the second memory 85 or an SoC integrated with other circuits. Furthermore, the second processor 87 may be composed of a combination of a CPU that executes programs and a DSP that performs predetermined arithmetic processing. In addition, all the functions of the first processor 69 may be implemented in hardware, or they may be configured using programmable devices. The second processor 87 functions as a processing unit 871 by reading and executing the control program 851 stored in the second memory 85.

[0032] Here, with reference to Figures 2 to 5, the configurations of the identification information database 91, the food ingredient management database 93, the item management database 95, and the open status management database 97 will be described.

[0033] Figure 2 shows an example of the configuration of the identification information database 91. The identification information database 91 is a database in which identification information is registered. The identification information database 91 registers account information 911, communication information 912, and imaging device ID 913 as a single record.

[0034] Account information 911 is information that identifies the account assigned to a user of the item management application 675. Communication information 912 is address information, such as an IP address, used for communication with the information processing device 6. The imaging device ID 913 is an ID that identifies imaging device 3 installed at the user's home.

[0035] Figure 3 shows an example of the configuration of the ingredient management DB93. The ingredient management DB93 is a database generated for each account information 911, and the following information is registered as one record: ingredient name information 931, color information 932, nutrient information 933, receiving date and time information 934, ingredient image 935, storage area 936, and shipping date and time information 937.

[0036] Food ingredient name information 931 is information indicating the name of the food item stored in refrigerator 2. The processing unit 871 determines the food item captured in the image uploaded from the imaging device 3. The processing unit 871 determines the food item based on the color and shape of the item, and registers the determination result as food item name information 931.

[0037] The color information 932 stores the color of the food ingredient from the food ingredient name information 931. When the processing unit 871 determines the food item from the captured image, it obtains the color information of the determined food item. The color of the food item may be obtained by referring to a database of pre-registered food item colors stored in the second memory 85, or the color information of the food item may be obtained from another server (not shown). The processing unit 871 registers the obtained color as color information 932.

[0038] Nutrient information 933 stores information about the nutrients contained in the food ingredient name information 931. When the processing unit 871 determines a food ingredient from the captured image, it obtains the nutrients contained in the determined food ingredient. The nutrients contained in the food ingredient may be obtained by referring to a database that has been pre-registered with the nutrients contained in the food ingredient and stored in the second memory 85, or the nutrient information of the food ingredient may be obtained from another server (not shown). The processing unit 871 registers the obtained nutrients as nutrient information 933.

[0039] The storage date and time information 934 indicates the date and time when the food item specified in the food item name information 931 was stored in refrigerator 2. The processing unit 871 registers the date and time when it received the uploaded data from the imaging device 3 as the storage date and time information 934. The storage date and time information 934 may also be registered based on the date and time information uploaded from the imaging device 3.

[0040] The food ingredient image 935 is registered with the image uploaded from the imaging device 3. The imaging device 3 captures images of the inside of the refrigerator 2 and generates captured images. From the generated captured images, the imaging device 3 extracts images of the areas in which items are captured as item images. The imaging device 3 uploads the uploaded data, including the extracted item images, to the item management server 8. The processing unit 871 retrieves the item images from the uploaded data, and if food is detected from the extracted item images, it registers the item images as food image 935.

[0041] Storage area 936 is information indicating the storage area of ​​refrigerator 2 where food is stored. Processing unit 871 extracts information indicating the storage area from the uploaded data and registers the extracted information indicating the storage area in storage area 936. Multiple storage areas in refrigerator 2 are assigned identification numbers. The identification number is registered in storage area 936 as information indicating the storage area.

[0042] The dispatch date and time information 937 indicates the date and time when the food item specified in the food item name information 931 was dispatched from refrigerator 2. The processing unit 871 registers the date and time when it received the uploaded data from the imaging device 3 in the dispatch date and time information 937. The dispatch date and time information 937 may also be registered based on the date and time information uploaded from the imaging device 3. Food items may be taken out of refrigerator 2 for cooking or other purposes and then put back into refrigerator 2. For this reason, even if the processing unit 871 determines that food items have been dispatched from refrigerator 2, it does not delete the record in the food item management DB 93 until a certain period of time has elapsed. When used food items are re-stored, the processing unit 871 either deletes the record in the dispatch date and time information 937 or registers a new re-store date and time in the re-store date and time information 934.

[0043] Figure 4 shows an example of the configuration of the inventory management DB95. The item management DB95 is a database generated for each account information 911, and the marker ID 951, item image 952, receiving date and time information 953, storage area 954, and shipping date and time information 955 are registered as one record.

[0044] Marker ID 951 is information included in the upload data received from the imaging device 3. The imaging device 3 detects markers attached to items from the captured image and obtains the marker ID assigned to the detected marker. The imaging device 3 sends the obtained marker ID as upload data to the item management server 8. The processing unit 871 extracts the marker ID from the received upload data and registers the extracted marker ID as marker ID 951.

[0045] The item image 952 contains the image uploaded from the imaging device 3. As described above, the imaging device 3 uploads the uploaded data, including the cropped image of the item, to the item management server 8. If the uploaded data contains a marker ID, the processing unit 871 registers the item image extracted from the uploaded data as the item image 952. If the uploaded data includes a marker ID, the marker ID may be stored in advance on the item management server 8, and the corresponding item image may be registered in the item image 952. In this case, the user takes a picture of the item with the marker ID using a camera and sends the captured item image to the item management server 8. The processing unit 871 associates the received item image with the marker ID and registers it in the item management DB 95.

[0046] The storage date and time information 953 indicates the date and time when the item with the marker ID was stored in refrigerator 2. The processing unit 871 registers the date and time when it received the uploaded data from imaging device 3 in the storage date and time information 953.

[0047] Storage area 954 is information indicating the storage area of ​​refrigerator 2 where the items are stored. Processing unit 871 retrieves information indicating the storage area from the uploaded data and registers the retrieved information indicating the storage area in storage area 954.

[0048] The dispatch date and time information 955 indicates the date and time when the item with the marker ID was dispatched from refrigerator 2. The processing unit 871 registers the date and time when it received the uploaded data from the imaging device 3 in the dispatch date and time information 955. Items may be taken out of refrigerator 2 for cooking or other purposes and then put back into refrigerator 2. For this reason, even if the processing unit 871 determines that food has been dispatched from refrigerator 2, it does not delete the record in the food management DB 93 until a certain period of time has elapsed. When an item is re-stored, the processing unit 871 either deletes the record in the dispatch date and time information 955 or registers a new re-store date and time in the re-store date and time information 953.

[0049] Figure 5 shows an example of the configuration of the open state management DB97. The open status management DB97 is a database generated for each account information 911, and includes a column 971 for recording the opening date and time of the refrigerator compartment 21, a column 972 for recording the opening date and time of the vegetable compartment 27, and a column 973 for recording the opening date and time of the freezer compartment 26. Column 971, which records the date and time the refrigerator compartment 21 was opened, registers the date and time when either the left door 22 or the right door 23 of the refrigerator compartment 21 was opened. Column 972, which records the date and time the vegetable compartment 27 was opened, registers the date and time when drawer 27A of the vegetable compartment 27 was pulled out. Column 973, which records the date and time the freezer compartment 26 was opened, registers the date and time when drawer 26A of the freezer compartment 26 was pulled out. Furthermore, data that has been registered in the open status management DB97 for a certain period of time is deleted from the open status management DB97.

[0050] [4. Refrigerator configuration] Here, we will describe the configuration of the refrigerator 2 on which the imaging device 3 is mounted. Figure 6 is a front view of refrigerator 2, showing the left door 22 and right door 23 of refrigerator 2 in the open position. The X, Y, and Z axes shown in Figure 6 are three mutually orthogonal axes. In the view of the drawing, the X-axis corresponds to the left-right direction, and the positive direction of the X-axis corresponds to the right direction. In the view of the drawing, the Y-axis corresponds to the front-back direction, and the positive direction of the Y-axis corresponds to the front direction. In the view of the drawing, the Z-axis corresponds to the up-down direction, and the positive direction of the Z-axis corresponds to the up direction.

[0051] The refrigerator 2 includes a box-shaped member, the box body 20, which has heat insulation properties and forms the main body of the refrigerator 2. The box body 20 has a structure in which thermal insulation material is filled in the space between the outer box and the inner box. The outer box and inner box are not shown in the illustration. The outer box is made of, for example, a metal plate such as iron or a rigid resin such as ABS resin. The inner box is made of, for example, a rigid resin such as ABS resin. For example, rigid polyurethane foam is used as the thermal insulation material. However, the thermal insulation material is not limited to this, and any material that has thermal insulation properties and can be filled in the space between the outer box and the inner box is acceptable.

[0052] The internal space of refrigerator 2 is divided into multiple spaces by partition members. In this embodiment, the internal space is divided into five spaces, which function as a refrigerator compartment 21, an ice-making compartment 24, a fresh freezing compartment 25, a freezer compartment 26, and a vegetable compartment 27, respectively. The refrigerator compartment 21, ice-making compartment 24, fresh freezing compartment 25, freezer compartment 26, and vegetable compartment 27 correspond to storage compartments. The ice-making compartment 24, fresh freezing compartment 25, freezer compartment 26, and vegetable compartment 27 are each equipped with drawers 24A, 25A, 26A, and 27A, respectively, capable of accommodating items. The refrigerator compartment 21 is closed by the left door 22 and the right door 23, the ice-making compartment 24 is closed by the drawer 24A, the fresh freezing compartment 25 is closed by the drawer 25A, the freezer compartment 26 is closed by the drawer 26A, and the vegetable compartment 27 is closed by the drawer 27A.

[0053] The refrigerator compartment 21 is equipped with shelves 211, 213, 215, and 217, and these shelves divide the refrigerator compartment 21 into five spaces. The refrigerator compartment 21 located at the top in the Z-axis direction, separated by shelf 211, is called the first refrigerator compartment 21A. The refrigerator compartment 21 located second from the top in the Z-axis direction, separated by shelf 211 and shelf 213, is called the second refrigerator compartment 21B. The refrigerator compartment 21 located third from the top in the Z-axis direction, separated by shelf 213 and shelf 215, is called the third refrigerator compartment 21C. The refrigerator compartment 21 located fourth from the top in the Z-axis direction, separated by shelf 215 and shelf 217, is called the fourth refrigerator compartment 21D. The refrigerator compartment 21 located at the bottom in the Z-axis direction, separated by shelf 217, is called the chilled compartment 28. The chilled compartment 28 is a storage compartment with a lower internal temperature than the refrigerator compartments 21. The chilled compartment 28 is provided with a drawer 28A capable of storing items.

[0054] The front of the refrigerator 2, which is located in the positive direction of the Y-axis, is provided with a rotating left door 22 and a right door 23 that open like double doors. The left door 22 is provided with four left door pockets. The left door 22 is provided with the first left door pocket 221, the second left door pocket 223, the third left door pocket 225, and the fourth left door pocket 227, in order from top to bottom in the Z-axis direction. The right door 23 is provided with four right door pockets. The right door 23 is provided with the first right door pocket 231, the second right door pocket 233, the third right door pocket 235, and the fourth right door pocket 237, in order from top to bottom in the Z-axis direction.

[0055] [5. Configuration of the imaging device] Figure 7 is a block diagram showing the configuration of the imaging device 3. The configuration of the imaging device 3 will be explained with reference to Figure 7. The imaging device 3 comprises a sensor board 30 and a camera board 40. The sensor board 30 is a signal processing board that processes and converts signals output from various sensors. The sensor board 30 is connected to a left door distance sensor 31, a right door distance sensor 32, a pull-out distance sensor 33, and a human presence sensor 34. The left door distance sensor 31, the right door distance sensor 32, and the pull-out distance sensor 33 correspond to open / close sensors.

[0056] The camera board 40 is a signal processing board that processes signals output by the camera module to generate captured images. The camera board 40 is connected to a third communication module 41, a camera module consisting of a refrigerator camera 42 and a drawer camera 43, and a light-emitting device 44.

[0057] The left door distance measuring sensor 31 detects the distance to the left door 22. The left door distance measuring sensor 31 outputs a signal indicating the detected distance to the left door 22 to the third control unit 430. The right door distance measuring sensor 32 detects the distance to the right door 23. The right door distance measuring sensor 32 outputs a signal indicating the detected distance to the right door 23 to the third control unit 430.

[0058] Figure 8 is a top view of refrigerator 2, seen from above. The left door distance sensor 31 includes a light-emitting element that emits detection light and a light-receiving element that receives reflected light from the left door 22. The right door distance sensor 32 also includes a light-emitting element that emits detection light and a light-receiving element that receives reflected light from the right door 23. The light-emitting element and light-receiving element are not shown in the illustration. The light-emitting element is positioned in the positive direction of the Y-axis relative to the light-receiving element. This allows the detection range of the distance sensor to be widened. The distance measuring sensor 31 for the left door has a light-emitting element and a light-receiving element installed so as to be tilted at a predetermined angle with respect to the positive direction of the Y axis. The same applies to the distance measuring sensor 32 for the right door. When the left door 22 is open by a predetermined angle or more, the detection light emitted from the light-emitting element is reflected on the inside of the left door 22 at the upper part in the Z-axis direction. When the left door 22 is closed, the detection light emitted from the light-emitting element is reflected on the inside of the left door 22 at the upper part in the Z-axis direction. For example, it is reflected on the inside of the left door 22 in the region between the upper end of the left door 22 and the gasket provided on the left door 22. By detecting the reflected light reflected from the upper part of the left door 22 in the Z-axis direction with the light-receiving element, it is determined that the left door 22 is in an open state. For example, if the signal changes by more than a predetermined value when compared with the signal when the left door 22 is closed, it is determined that the left door 22 is in an open state. Similarly, when the right door 23 is open beyond a predetermined angle, the detection light emitted from the light-emitting element is reflected on the inside of the right door 23, specifically at the upper part in the Z-axis direction. When the right door 23 is closed, the detection light emitted from the light-emitting element is reflected on the inside of the right door 23, specifically at the upper part in the Z-axis direction. For example, it is reflected on the inside of the right door 23 in the region between the upper end of the right door 23 and the gasket provided on the right door 23. By detecting the reflected light reflected at the upper part of the right door 23 in the Z-axis direction with a light-receiving element, it is determined that the right door 23 is in an open state. For example, by comparing it with the signal when the right door 23 is closed, if it changes by more than a predetermined value, it is determined that the right door 23 is in an open state. The inside of the left door 22 is the side that faces the interior of the refrigerator 2 when the left door 22 is closed. The same applies to the inside of the right door 23.

[0059] The drawer distance sensor 33 is a sensor that detects the distance to drawers 24A, 25A, 26A, 27A, and 28A. The drawer distance sensor 33 has a light-emitting element that emits detection light and a light-receiving element that receives reflected light from the detection light, both of which are installed facing downwards in the negative direction of the Z-axis. The illustration of the light-emitting element and the light-receiving element is omitted. When drawers 24A, 25A, 26A, and 27A are pulled out from inside the refrigerator 2, the detection light emitted by the light-emitting element is reflected by one of the drawers 24A, 25A, 26A, or 27A. The light-receiving element receives the reflected light from one of the drawers 24A, 25A, 26A, or 27A. The change in time from when the light-emitting element emits the detection light until the light-receiving element receives the reflected light determines which of the drawers 24A, 25A, 26A, or 27A has been pulled out. The drawer distance sensor 33 outputs a signal indicating the distance to the drawers 24A, 25A, 26A, or 27A to the third control unit 430.

[0060] The motion sensor 34 detects the approach of a person. The motion sensor 34 may be of any type: infrared, ultrasonic, or visible light. The motion sensor 34 detects the approach of a person and outputs a sensor signal indicating the approach of a person to the third control unit 430.

[0061] The sensor board 30 is equipped with a first interface circuit 310 and a signal processing unit 330. The signal processing unit 330 includes a third memory 340 and a third processor 350. The first interface circuit 310 and the signal processing unit 330 may be configured as a System-on-a-chip (SoC).

[0062] The first I / F circuit 310 is a communication interface with the camera board 40. The first I / F circuit 310 includes a port for connecting a cable and an interface circuit that conforms to a communication standard.

[0063] The signal processing unit 330 is a computer device that includes a third memory 340 and a third processor 350.

[0064] The third memory 340 includes ROM and RAM. The ROM stores the control program 345 executed by the third processor 350. The RAM is used as the arithmetic area for the third processor 350.

[0065] The third processor 350 is an arithmetic processing unit such as a CPU or MPU. The third processor 350 may be composed of a single processor or multiple processors. Alternatively, the third processor 350 may be composed of part or all of the third memory 340 or an SoC integrated with other circuits. Furthermore, the third processor 350 may be composed of a combination of a CPU that executes programs and a DSP that performs predetermined arithmetic processing. In addition, all the functions of the third processor 350 may be implemented in hardware, or they may be configured using programmable devices.

[0066] The signal processing unit 330 processes the signals input from the left door distance sensor 31, the right door distance sensor 32, the pull-out distance sensor 33, and the human presence sensor 34, for example, by performing A / D conversion, and outputs the processed data to the third control unit 430. The processed data is called sensor data.

[0067] The third communication module 41 is equipped with connectors and communication circuits, and connects to the network 10 in accordance with communication standards such as Wi-Fi (registered trademark), and communicates data with the item management server 8.

[0068] The refrigerator camera 42 is a camera equipped with an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The refrigerator camera 42 takes images according to the control of the third control unit 430 and outputs the captured image generated by the imaging to the third control unit 430. The captured image may be a still image or a moving image.

[0069] Figure 9 shows an example of an image captured by the camera 42 for the refrigerator compartment. The field of view of the refrigerator compartment camera 42 is wider than that of the drawer camera 43. The image captured by the refrigerator compartment camera 42 includes a part of the refrigerator compartment 21, the left door 22 and the right door 23, and the detection area TA. The detection area TA is an area within the front region formed in front of the chiller compartment 28 on the bottom surface of the refrigerator compartment 21. In this embodiment, the detection area TA is exemplified as an area smaller in area than the front region, but it may be the same area. The imaging device 3 detects items being placed in the refrigerator 2 or taken out of the refrigerator in this detection area TA.

[0070] Figure 10 shows an example of an image captured by the drawer camera 43. The pull-out camera 43 is also a camera equipped with an image sensor such as a CCD sensor or a CMOS sensor. The pull-out camera 43 also starts capturing images in accordance with the control of the third control unit 430. The pull-out camera 43 also generates an image and outputs the generated image to the third control unit 430. The image may be a still image or a moving image. Figure 10 shows an example of an image captured by the drawer camera 43, which shows the freezer compartment 26 when it is pulled out from inside the refrigerator 2.

[0071] The light-emitting device 44 is equipped with light-emitting elements such as LEDs (Light Emitting Diodes) and emits light or lights up according to the control of the third control unit 430.

[0072] The camera board 40 is equipped with a second I / F circuit 410 and a third control unit 430. The second I / F circuit 410 and the third control unit 430 may be configured using an SoC.

[0073] The second I / F circuit 410 is a communication interface with the sensor board 30. The second I / F circuit 410 includes a port for connecting a cable and an interface circuit that conforms to a communication standard.

[0074] The third control unit 430 is a computer device comprising a fourth memory 440 and a fourth processor 450.

[0075] The fourth memory 440 includes ROM and RAM. The ROM and RAM are not shown in the diagram. The ROM stores the control program 441 executed by the fourth processor 450, the imaging device ID 443, and the captured images. The imaging device ID 443 is identification information that uniquely identifies the imaging device 3.

[0076] The fourth processor 450 is an arithmetic processing unit such as a CPU or MPU. The fourth processor 450 may be composed of a single processor or multiple processors. Alternatively, the fourth processor 450 may be composed of part or all of the fourth memory 440 or an SoC integrated with other circuits. Furthermore, the fourth processor 450 may be composed of a combination of a CPU that executes programs and a DSP that performs predetermined arithmetic processing. In addition, all the functions of the fourth processor 450 may be implemented in hardware, or they may be configured using programmable devices.

[0077] The third control unit 430 functions as a person presence determination unit 451, an opening / closing determination unit 452, an imaging control unit 453, an entry / exit determination unit 454, an acquisition unit 455, and a data generation unit 456 by executing the control program 441.

[0078] The person presence determination unit 451 determines whether or not a person is present around the refrigerator 2 based on the detection value of the human presence sensor 34. For example, if the detection value of the human presence sensor 34 is above a predetermined value, the person presence determination unit 451 determines that a person is present around the refrigerator 2, and if it is below the predetermined value, it determines that no person is present around the refrigerator 2.

[0079] The opening / closing determination unit 452 determines whether the left door 22 is in the open state or not based on the sensor data input from the left door distance measuring sensor 31. The opening / closing determination unit 452 also determines whether the right door 23 is in the open state or not based on the sensor data input from the right door distance measuring sensor 32. Furthermore, the opening / closing determination unit 452 determines, based on sensor data input from the drawer distance sensor 33, whether or not any of the drawers 24A, 25A, 26A, 27A, or 28A has been pulled out from inside the refrigerator 2.

[0080] If the person presence detection unit 451 determines that a person is present around the refrigerator 2, the imaging control unit 453 starts imaging with the refrigerator compartment camera 42 and the drawer camera 43. Subsequently, the imaging control unit 453 turns on or blinks the light-emitting device 44 to notify the user that imaging has started. If the person presence detection unit 451 determines that no person is present around the refrigerator 2, the imaging control unit 453 stops imaging with the refrigerator compartment camera 42 and the drawer camera 43. Subsequently, the imaging control unit 453 turns off the light-emitting device 44.

[0081] Furthermore, if the opening / closing determination unit 452 determines that either the left door 22 or the right door 23 is open, or that any of the drawers 24A, 25A, 26A, 27A, or 28A is open, the imaging control unit 453 starts recording with the refrigerator camera 42 and the drawer camera 43. The imaging control unit 453 stores the input captured images in the fourth memory 440. Also, if the opening / closing determination unit 452 determines that both the left door 22 and the right door 23 are closed, and any of the drawers 24A, 25A, 26A, 27A, or 28A are closed, the imaging control unit 453 stops recording with the refrigerator camera 42 and the drawer camera 43.

[0082] Furthermore, the imaging control unit 453 may start recording with the refrigerator camera 42 if the open / close determination unit 452 determines that either the left door 22 or the right door 23 is open. The imaging control unit 453 may store the input captured images in the fourth memory 440. Also, the imaging control unit 453 may stop recording with the refrigerator camera 42 if the open / close determination unit 452 determines that either the left door 22 or the right door 23 is closed.

[0083] Furthermore, the imaging control unit 453 may start recording with the drawer camera 43 if the open / close determination unit 452 determines that any of the drawers 24A, 25A, 26A, 27A, or 28A is in the open state. The imaging control unit 453 may also store the input captured image in the fourth memory 440. The imaging control unit 453 may stop recording with the drawer camera 43 if the open / close determination unit 452 determines that any of the drawers 24A, 25A, 26A, 27A, or 28A is in the closed state.

[0084] The in / out determination unit 454 analyzes the captured image to detect items such as food ingredients, food products, and storage containers. The in / out determination unit 454 also determines whether the detected items are to be placed into or removed from the refrigerator 2.

[0085] The process of detecting items from captured images and determining their entry and exit will be explained with reference to Figures 11 and 12. Figure 11 shows the boundaries of the first region R1, the second region R2, the third region R3, and the fourth region R4. The following section describes the case of detecting an item BP with a marker MK.

[0086] First, the entry / exit determination unit 454 sets a first region R1, a second region R2, a third region R3, and a fourth region R4 in the captured image. The first region R1 is the region in the captured image that corresponds to the refrigerator compartment 21. The second region R2 is the region in the captured image that corresponds to the left door 22. The third region R3 is the region in the captured image that corresponds to the right door region 23. The fourth region R4 is the region in the captured image that corresponds to the outside of the refrigerator 2.

[0087] The first region R1 corresponding to the refrigerator compartment 21 remains constant in the captured image, even if the opening degree of the left door 22 and right door 23 opened and closed by the user differs. For this reason, the entry / exit determination unit 454 sets a preset range of the captured image to the first region R1. The boundary line 501 shown in Figure 11 corresponds to the boundary between the first region R1 and the second region R2, third region R3, and fourth region R4.

[0088] Furthermore, the second region R2 corresponding to the left door 22 and the third region R3 corresponding to the right door 23 are changed depending on the degree to which the left door 22 and right door 23 are opened by the user. For this reason, the entry / exit determination unit 454 detects, for example, the edges of the first left door pocket 221 to the fourth left door pocket 227 of the left door 22 and the first right door pocket 231 to the fourth right door pocket 237 of the right door 23.

[0089] The inbound / outbound determination unit 454 sets the ranges of the second region R2 and the third region R3 based on the detected edges of the first left door pocket 221 to the fourth left door pocket 227 and the first right door pocket 231 to the fourth right door pocket 237. Additionally, marks or the like may be placed at the ends of the first left door pocket 221 to the fourth left door pocket 227, and the first right door pocket 231 to the fourth right door pocket 237. The entry / exit determination unit 454 sets the ranges of the second region R2 and the third region R3 based on the positions of the marks detected from the captured image. The entry / exit determination unit 454 also sets the areas other than the set first region R1, second region R2, and third region R3 as the fourth region R4, which is outside the refrigerator compartment 21. Boundary line 502 shown in Figure 11 indicates the boundary of the second region R2, and boundary line 503 indicates the boundary of the third region R3.

[0090] The entry / exit determination unit 454 may set the range of the second region R2 based on sensor data input from the left door distance measuring sensor 31. Furthermore, the entry / exit determination unit 454 may set the range of the third region R3 based on sensor data input from the right door distance measuring sensor 32.

[0091] The inventory determination unit 454 analyzes the captured image to detect images of item BP and marker MK. The inventory determination unit 454 detects item BP and marker MK from the region corresponding to the detection region TA of the captured image. For example, the inventory determination unit 454 detects item BP and marker MK based on changes in brightness or color in the detection region TA.

[0092] Furthermore, the inventory entry / exit determination unit 454 detects the trajectory of moving items BP and markers MK by performing image analysis on the continuously captured images. The inbound / outbound determination unit 454 determines which of the first area R1, second area R2, or third area R3 the trajectory of the detected marker MK ended in, and then determines which area the item BP was inbound into. Furthermore, if the trajectory of the detected marker MK ends in the fourth area R4, the inbound / outbound determination unit 454 determines that the item BP has been removed from the refrigerator compartment 21.

[0093] Figure 12 is a diagram illustrating the inbound / outbound determination image. The inventory determination unit 454 superimposes the inventory determination image HG onto the captured image to determine whether the item BP is being received or sent out. Figure 12 shows an image of an item BP marked with marker MK being placed into the refrigerated room 21, with the entry / exit determination image HG superimposed on it.

[0094] The inbound / outbound determination image HG has a boundary line L1. The boundary line L1 is a line that extends from the left edge to the right edge of the entry / exit determination image HG, located approximately in the center of the image. The boundary line L1 is located in front of the refrigerator 2, beyond the opening of the refrigerator compartment 21 as captured in the image taken by the refrigerator compartment camera 42.

[0095] The image area of ​​the in / out detection image HG is divided into a first area HA1 and a second area HA2 by a boundary line L1. The first area HA1 corresponds to the front side of the refrigerator 2 from the opening of the refrigerator compartment 21 captured in the image captured by the refrigerator compartment camera 42 when the in / out detection image HG is superimposed on the captured image. The second area HA2 corresponds to the rear side of the refrigerator 2 from the opening of the refrigerator compartment 21 captured in the image captured by the refrigerator compartment camera 42 when the in / out detection image HG is superimposed on the captured image.

[0096] The inbound / outbound determination unit 454 analyzes the superimposed image of the inbound / outbound determination image HG to determine whether the marker MK detected in the detection area TA has moved to the first area HA1 or the second area HA2. If the inbound / outbound determination unit 454 determines that the marker MK detected in the detection area TA has moved to the first area HA1, it determines that the detected item BP is an item BP that has been taken out of the refrigerator 21. On the other hand, if the marker MK detected in the detection area TA has moved to the second area HA2, the inbound / outbound determination unit 454 determines that the detected item BP is an item BP that has been taken into the refrigerator 21.

[0097] If the In / Out Determination Unit 454 determines that item BP has been placed inside the storage area, it determines the storage area where item BP was placed. The In / Out Determination Unit 454 analyzes the captured image to identify the storage area in the refrigerated compartment 21 where item BP was placed. Then, based on the direction of movement of the marker MK and the arrangement of the multiple storage areas provided inside the storage area, the In / Out Determination Unit 454 determines the storage area where item BP was placed. For example, each of the shelves 211, 213, 215, and 217 of refrigerator 2, as well as the front of the left door 22 or right door 23, and each of the door pockets, are marked to identify the storage area. The entry / exit determination unit 454 detects the marks that identify the storage area from the captured image and identifies the storage area based on the positional relationship between the detected marks and the marker MK attached to the item BP.

[0098] Furthermore, the in / out determination unit 454 may determine whether or not the item BP is an item BP that has been re-stored in the refrigerator 2. For example, this could include cases where user P takes food out of the refrigerator 2 for use, uses some of the food, and then re-stores the remaining food in the refrigerator 2, or where user P uses food stored in a storage container which is an item BP, replenishes the storage container with new food, and then re-stores it in the refrigerator 2.

[0099] The above explanation described the case where the object to be detected from the captured image is an item BP with a marker MK attached. However, the same method can be used to detect food ingredients and other items that do not have a marker MK attached. The inbound / outbound determination unit 454 detects food ingredients and food products based on the color and shape of the captured image, and determines whether the detected food ingredients or food products have been inbound or outbound, in the same manner as in the case of items BP with markers MK.

[0100] The acquisition unit 455 analyzes the marker MK detected by the inbound / outbound determination unit 454 and obtains the marker ID. The marker ID is the identification information of the marker MK that is assigned to it. For example, if the marker MK is a two-dimensional code such as a QR code (registered trademark), the acquisition unit 455 obtains the marker ID contained in the QR code. The acquisition unit 455 temporarily stores the obtained marker ID in the fourth memory 440.

[0101] The data generation unit 456 generates upload data. Upload data is the data to be uploaded to the item management server 8. The data generation unit 456 sends the generated upload data to the item management server 8.

[0102] Uploaded data includes item images, storage area information, imaging device ID, information indicating whether the item was received or released, the date and time of receipt or release, and information about opened or pulled-out doors or drawers. Information regarding the receipt of an item is called receipt information, and information regarding the release of an item is called release information. Item images are cropped images from the captured image, showing the area in which the item was photographed.

[0103] Furthermore, if the item is marked with a marker (MK), the uploaded data will include the marker ID in addition to the item image, storage area information, imaging device ID, storage date and time or delivery date and time, storage information or delivery information, and information on opened or pulled-out doors or drawers.

[0104] Furthermore, if at least one of the left door 22 and the right door 23 is opened, or if any of the drawers 24A, 25A, 26A, or 27A is pulled out but no goods are stored in or out, the uploaded data will include information about the opened or pulled-out door or drawer, as well as date and time information.

[0105] [6. Operation of the Imaging Device] Figure 13 is a flowchart showing the operation of the imaging device 3. The operation of the imaging device 3 will be explained with reference to the flowchart shown in Figure 13. The third control unit 430 determines whether or not a person is present around the refrigerator 2 based on the sensor data from the human presence sensor 34 (step SA1).

[0106] If the third control unit 430 determines that a person is present around the refrigerator 2 (step SA1 / YES), it instructs the refrigerator compartment camera 42 and the drawer camera 43 to start imaging (step SA2).

[0107] Next, the third control unit 430 determines whether at least one of the left door 22 and the right door 23 is open, or whether any of the drawers 24A, 25A, 26A, 27A, and 28A is open (step SA3).

[0108] The third control unit 430 performs the process in step SA13 if the left door 22 and the right door 23 are not open, and none of the drawers 24A, 25A, 26A, 27A, and 28A are open (step SA3 / NO).

[0109] If at least one of the left door 22 and the right door 23 is open, or if any of the drawers 24A, 25A, 26A, 27A, or 28A is open (step SA3 / YES), the third control unit 430 stores the images captured by the refrigerator camera 42 and the drawer camera 43 in the fourth memory 440 and starts recording the captured images (step SA4).

[0110] Next, the third control unit 430 reads the captured image from the fourth memory 440 and determines whether or not a marker and an object have been detected within the detection area TA based on the read captured image (step SA5).

[0111] If the third control unit 430 determines that it has detected a marker and an item within the detection area TA (step SA5 / YES), it reads the marker ID from the detected marker (step SA7). Then, the third control unit 430 extracts an item image from the captured image, which is the image in which the item was captured (step SA8).

[0112] Furthermore, if no marker is detected within the detection area TA and only an object is detected (step SA6 / YES), the third control unit 430 extracts an image of the object from the captured image (step SA8).

[0113] Next, the third control unit 430 makes an inbound / outbound determination based on the captured image recorded in the fourth memory 440 (step SA9). If the third control unit 430 determines that an item has been inbound, it stores the item image of the inbound item, the storage area where the item was inbound, and the time of inbound in the fourth memory 440, associating them. If the third control unit 430 determines that an item has been outbound, it stores the item image of the outbound item and the time of outbound in the fourth memory 440, associating them. Furthermore, if at least one of the left door 22 and the right door 23 is opened, or if any of the drawers 24A, 25A, 26A, or 27A is pulled out but no items are being stored or taken out, the third control unit 430 stores information about the opened or pulled-out door or drawer, along with date and time information, in the fourth memory 440.

[0114] Next, the third control unit 430 determines whether the open door and drawer have been closed (step SA11). If the third control unit 430 determines that the door and drawer are not closed (step SA11 / NO), it proceeds to step SA5.

[0115] On the other hand, if the third control unit 430 determines that the open door and drawer have been closed (step SA11 / YES), it terminates recording of the captured image to the fourth memory 440 (step SA12).

[0116] Next, the third control unit 430 determines whether or not there is a person around the refrigerator 2 (step SA13). If the third control unit 430 determines that there is a person around the refrigerator 2 (step SA13 / YES), it moves the process to step SA3.

[0117] On the other hand, if the third control unit 430 determines that there are no people around the refrigerator 2 (step SA13 / NO), it will terminate imaging with the refrigerator compartment camera 42 and the drawer camera 43 (step SA14).

[0118] When the third control unit 430 completes the processing in step SA14, it moves the processing to step SA1 and then performs the processing from step SA1 onward again.

[0119] [7. Operation of the imaging device and the operation of the inventory management server] Figure 14 is a flowchart showing the operation of the inventory management system 1. In Figure 14, flowchart FB shows the operation of the imaging device 3, and flowchart FC shows the operation of the item management server 8.

[0120] As shown in the flowchart FB, the third control unit 430 determines whether a trigger has occurred to send the uploaded data to the item management server 8 (step SB1). A trigger for sending the uploaded data may be the determination that an item has been received or dispatched, and the determination result is stored in the fourth memory 440. Other triggers may include the completion of recording of images captured by the refrigerator camera 42, or the current time reaching a predetermined time.

[0121] If a trigger occurs (step SB1 / YES), the third control unit 430 retrieves the upload data from the fourth memory 440 (step SB2). The uploaded data includes item images, storage area information, imaging device ID, receiving or shipping information, receiving or shipping date and time, and information about opened or pulled-out doors or drawers. If the item is marked with a marker (MK), the marker ID is also included in addition to the above uploaded data. Furthermore, if at least one of the left door 22 and the right door 23 is opened, or if any of the drawers 24A, 25A, 26A, or 27A is pulled out but no goods are stored in or out, the uploaded data will include information about the opened or pulled-out door or drawer, as well as date and time information.

[0122] The third control unit 430 sends the generated upload data to the item management server 8 (step SB3). After that, the third control unit 430 returns to the determination in step SB1.

[0123] Next, the operation of the item management server 8 will be explained with reference to flowchart FC. The second control unit 83 of the item management server 8 determines whether or not it has received the uploaded data (step SC1).

[0124] If the second control unit 83 determines that it has not received the uploaded data (step SC1 / NO), it performs the determination in step SC1 again.

[0125] On the other hand, if the second control unit 83 determines that it has received the uploaded data (step SC1 / YES), it stores the received uploaded data in the second memory 85 and updates either the ingredient management DB 93, the item management DB 95, or the open status management DB 97 (step SC2). The second control unit 83 may also update any of the ingredient management DB 93, the item management DB 95, or the open status management DB 97.

[0126] The second control unit 83 obtains account information 911 associated with the imaging device ID included in the received uploaded data from the identification information DB 91. The second control unit 83 updates one of the following: the food ingredient management DB 93, the item management DB 95, or the open status management DB 97, corresponding to the acquired account information 911. The second control unit 83 may also update any of the food ingredient management DB 93, the item management DB 95, or the open status management DB 97.

[0127] The second control unit 83 updates the item management DB 95 if the received uploaded data includes a marker ID. The second control unit 83 registers the marker ID, item image, receiving date and time or shipping date and time, and storage area included in the uploaded data in the item management DB 95.

[0128] Furthermore, if the received uploaded data does not contain a marker ID, the second control unit 83 identifies the item from the item image and obtains the color information and nutritional information of the identified item. The second control unit 83 registers the name of the identified food item in the food item name information 931 of the food item management DB 93. The second control unit 83 also registers the acquired color information and nutritional information of the item in the color information 932 and nutritional information 933 of the food item management DB 93. In addition, the second control unit 83 registers the item image included in the uploaded data in the food item image 935, and registers the receiving date and time or the shipping date and time, and the storage area in the receiving date and time 934 or shipping date and time 937 and storage area 936, respectively.

[0129] Furthermore, the second control unit 83 updates the open state management DB 97 based on the information of the opened or pulled-out door or drawer included in the uploaded data.

[0130] [8. Operation of the inventory management server] Figure 15 is a flowchart showing the operation of the item management server 8. The second control unit 83 of the item management server 8 determines whether or not there is a request from the information processing device 6 to acquire data accompanied by account information (step SD1). If there is no request to acquire data (step SD1 / NO), the second control unit 83 waits to start processing until a request to acquire data is received.

[0131] When the second control unit 83 receives a data acquisition request (step SD1 / YES), it determines whether the data registered in the ingredient management DB 93, item management DB 95, and open status management DB 97 corresponding to the received account information has been updated (step SD2).

[0132] If there are no updates to the data registered in the ingredient management DB 93, item management DB 95, and open status management DB 97 (step SD2 / NO), the second control unit 83 notifies the information processing device 6 that there are no update data (step SD4). Alternatively, if there is no update data, the previous data may be sent.

[0133] Furthermore, if there are updates to the data registered in the food ingredient management DB 93, item management DB 95, and open status management DB 97 (step SD2 / YES), the second control unit 83 retrieves the updated data from the food ingredient management DB 93, item management DB 95, or open status management DB 97 and transmits the retrieved data to the information processing device 6 as updated data (step SD3).

[0134] [9. Configuration of App Screen 700] Figures 16 and 17 show examples of the initial screens 710A and 710B. Here, we will explain the application screen 700 displayed on the touch panel 63 of the information processing device 6. Figures 16 and 17 show the initial screens 710A and 710B that appear immediately after the item management application 675 is launched. The initial screens 710A and 710B are included in the application screen 700.

[0135] Figure 16 shows an example of the initial screen 710A displaying an image of the inside of refrigerator 2. Figure 17 shows an example of the initial screen 710B displaying an image of an empty refrigerator 2 with an image of an item acquired from the item management server 8. Initial screens 710A and 710B can be switched, for example, by operating the menu button 711. Hereinafter, initial screens 710A and 710B will be collectively referred to as initial screen 710.

[0136] Furthermore, if only the left door 22 is open, the initial screen 710A may display an image of the left door 22 and the area of ​​the refrigerator compartment 21 corresponding to the left door 22. Alternatively, the initial screen 710B may display an image of an empty refrigerator 2 showing the updated item image in the area of ​​the refrigerator compartment 21 corresponding to the left door 22. Similarly, the initial screen 710A may display an image of the right door 23 and the area of ​​the refrigerator compartment 21 corresponding to the right door 23, if only the right door 23 is open. Alternatively, the initial screen 710B may display an image of an empty refrigerator 2 showing the updated item image in the area of ​​the refrigerator compartment 21 corresponding to the right door 23.

[0137] The initial screen 710 displays the menu button 711, the refrigerator button 712, the vegetable compartment button 713, the freezer compartment button 714, and the recipe search button 715, among others.

[0138] By pressing the menu button 711, you can select the screen to display as the app screen 700. The app screens 700 that can be selected by the menu button 711 include the long-term storage notification screen 730, the loading / unloading status screen 750, the opening / closing status screen 770, and the stored items screen 820.

[0139] The refrigerator compartment button 712, vegetable compartment button 713, and freezer compartment button 714 are buttons that switch the displayed area. On the initial screen 710, when the refrigerator compartment button 712 is selected, the displayed area changes to the refrigerator compartment 21. When the vegetable compartment button 713 is selected, the displayed area changes to the vegetable compartment 27. Also, when the freezer compartment button 714 is pressed, the displayed area changes to the freezer compartment 26.

[0140] Recipe search button 715 is a button for searching for recipes. By pressing the recipe search button 715, a list of ingredients stored in refrigerator 2 will be displayed. The user can then select the desired ingredients from the list to search for recipes using those ingredients.

[0141] Figure 18 shows the long-term storage notification screen 730. The long-term storage notification screen 730 is also included in the app screen 700. The long-term storage notification screen 730 displays information about items that have not been removed from their storage area even after a predetermined period has elapsed since being placed in refrigerator 2. The long-term storage notification screen 730 shows an example where the storage area is displayed as information about items that have not been removed. The long-term storage notification screen 730 displays the refrigerator compartment button 731, the vegetable compartment button 733, the freezer compartment button 735, and the area display area 740. The refrigerator compartment button 731, the vegetable compartment button 733, and the freezer compartment button 735 are buttons that switch the displayed area.

[0142] The storage area of ​​refrigerator 2 is displayed in the area display area 740. The storage area displayed in the area display area 740 can be changed by pressing the refrigerator compartment button 731, the vegetable compartment button 733, or the freezer compartment button 735. Area display area 740 displays the storage areas of refrigerator 2 where items have been stored for 30 days or more. Figure 18 shows an example where 30 days is set as the preset period. The preset period can be changed at will by the user. Figure 18 shows an example where the refrigerator compartment 21 is divided into 18 storage areas in the area display region 740. The number of divisions of the refrigerator compartment 21 is arbitrary. The refrigerator compartment 21 is divided into a left area and a right area along the X-axis. Furthermore, the left area is divided into five sections along the Z-axis, with each of the five sections corresponding to the first refrigerator compartment 21A, the second refrigerator compartment 21B, the third refrigerator compartment 21C, the fourth refrigerator compartment 21D, and the chiller compartment 28. Similarly, the right area is divided into five sections along the Z-axis, with each of the five sections corresponding to the first refrigerator compartment 21A, the second refrigerator compartment 21B, the third refrigerator compartment 21C, the fourth refrigerator compartment 21D, and the chiller compartment 28.

[0143] The area of ​​the left door 22 is divided into four areas corresponding to the first left door pocket 221, the second left door pocket 223, the third left door pocket 225, and the fourth left door pocket 227. The area of ​​the right door 23 is divided into four areas corresponding to the first right door pocket 231, the second right door pocket 233, the third right door pocket 235, and the fourth right door pocket 237.

[0144] As shown in Figure 18, the long-term storage notification screen 730 displays the storage area containing items that have been stored in the refrigerator 2 for 30 days or more. Figure 18 shows an example of displaying the number of days elapsed, such as 30 days or more, but images such as pre-set shapes, symbols, or marks may also be displayed.

[0145] Figure 19 shows the loading / unloading status screen 750. The loading / unloading status screen 750 is also included in the application screen 700. The item management server 8, based on the item management DB 95, counts the number of items stored in refrigerator 2 and the number of items taken out of refrigerator 2 for each storage area during a pre-set period. The loading / unloading status screen 750 displays information about storage areas where the total number of items loaded and unloaded is less than or equal to a pre-set value.

[0146] The loading / unloading status screen 750 displays the refrigerator compartment button 751, the vegetable compartment button 752, the freezer compartment button 753, the period selection buttons 754 and 755, the unit selection buttons 756 and 757, and the loading / unloading status 760.

[0147] The refrigerator compartment button 751, vegetable compartment button 752, and freezer compartment button 753, like the refrigerator compartment button 712, vegetable compartment button 713, and freezer compartment button 714, are buttons that switch the displayed content.

[0148] The period selection buttons 754 and 755 are used to select the period for which the in / out status 760 will be displayed. Selecting period selection button 754 selects a period of one month, and selecting period selection button 755 selects a period of one week.

[0149] The unit selection buttons 756 and 757 are for selecting the display unit. Unit selection button 756 selects "1 day" as the display unit, while unit selection button 757 selects a unit that divides 1 day into three parts: morning, noon, and evening. The time periods for dividing morning, noon, and evening can be arbitrarily changed by the user. For example, one could set 5am to 10am as morning, 11am to 4pm as noon, and 4pm to midnight as evening.

[0150] The loading / unloading status screen 760, like the long-term storage notification screen 730, divides the refrigerator compartment 21 into 18 storage areas and displays information on the loading and unloading of items in each divided storage area. For example, suppose the refrigerator button 751 is selected, the period selection button 754 is selected and set to 1 month as the selected period, and the unit selection button 756 is selected and set to 1 day as the display unit. In this case, the percentage of items that have been taken in and out over the past month, including the current day, will be displayed for each of the 18 storage areas. The total number of times items were stored in the refrigerator compartment 21, left door 22, and right door 23, and the total number of times items were taken out from the refrigerator compartment 21, left door 22, and right door 23 over the past month, is set to 100%, and the ratio of the number of times items were stored in and out for each of the 18 storage areas is displayed. In Figure 19, the ratio for all storage areas is displayed, but it is also possible to display only the storage areas whose ratio is below a preset value.

[0151] Figure 20 shows the open / closed status screen 770. The open / closed status screen 770 is also included in the application screen 700. The open / close status screen 770 displays the period selection buttons 771 and 772, the unit selection buttons 773 and 774, and the open / close status 780.

[0152] The period selection buttons 771 and 772 are used to select the period for which the open / closed status 780 will be displayed. Selecting period selection button 771 selects a period of one month, and selecting period selection button 772 selects a period of one week.

[0153] The unit selection buttons 773 and 774 are used to select the display unit. Unit selection button 773 selects "1 day" as the display unit, while unit selection button 774 selects a unit that divides 1 day into three parts: morning, noon, and evening.

[0154] The opening / closing status 780 displays the count value of the number of times at least one of the left door 22 and the right door 23 has been opened. For example, suppose the period selection button 772 is selected, setting the period to one week, and the unit selection button 774 is selected, setting the display units to morning, noon, and evening. In this case, the number of times the refrigerator compartment 21, vegetable compartment 27, and freezer compartment 26 are opened for one week, each for morning, noon, and evening, will be displayed.

[0155] The item management server 8, if it fails to detect the opening or closing of a door or drawer after a pre-set period has elapsed, causes the information processing device 6 to perform a notification operation. For example, the application execution unit 695 may display the opening / closing status screen 770 in a different display mode than the normal state in which the opening or closing of a door or drawer is detected. Alternatively, the application execution unit 695 may change the display color of the opening / closing status screen 770, or display on the initial screen 710 that the opening or closing of a door or drawer has not been detected.

[0156] Figure 21 shows the selection screen 790, and Figures 22 and 23 show the item storage screens 820A and 820B. The selection screen 790 and the item storage screens 820A and 820B are also included in the application screen 700. The selection screen 790 is used to select the storage area to be displayed on the storage item screen 820B. The selection screen 790 displays the display selection buttons 791 and 792, the area selection display 800, and the confirm button 810.

[0157] Display selection buttons 791 and 792 are used to select how to display images of items stored in refrigerator 2. Selecting display selection button 791 displays items in descending order of the number of days since they were stored. Selecting display selection button 792 allows you to select the area in which the items are stored.

[0158] The area selection display 800 is a screen that allows the user to select a storage area where images of the stored items will be displayed when the display selection button 792 is selected. When the user selects one of the storage areas and presses the confirm button 810, the stored items screen 820B shown in Figure 23 is displayed.

[0159] Figure 22 shows the storage item screen 820A. The stored items screen 820A displays all items stored in refrigerator 2, sorted by the time elapsed since the date of storage, from longest to longest. The stored items screen 820A displays a delete button 821, a new add button 823, and an item display area 830. The delete button 821 deletes the item displayed in the item display area. The new add button 823 registers a new item in the item display area 830.

[0160] The item display area 830 displays the item image 831, the date and time of receipt 833, and storage area information 835. The item image 831 is an image of the item. The storage date and time 833 indicates the date and time when the item in item image 831 was stored in refrigerator 2. The storage area information 835 indicates the storage area of ​​refrigerator 2 in which the item is stored. Figure 22 shows an example in which the storage area information 835 displays an image showing the location of the storage area in the refrigerator compartment 21.

[0161] Figure 23 shows the contents screen 820B. The stored items screen 820B displays the items stored in the storage area selected in the area selection display 800, in descending order of the time elapsed since the storage date. The item storage screen 820B also displays a delete button 821 and a new add button 823.

[0162] The stored items column 840 also displays the item image 841, the date and time of storage 843, and storage area information 845. The item image 841 is an image of the item. The storage date and time 843 indicates the date and time when the item in item image 841 was stored in refrigerator 2. The storage area information 845 indicates the storage area of ​​refrigerator 2 in which the item is stored. In the case of the stored item screen 820B shown in Figure 23, the same storage area is shown for all items as the storage area information 845. Hereafter, the containment item screens 820A and 820B will be collectively referred to as containment item screen 820.

[0163] [10. Operation of the Information Processing Device] Figure 24 is a flowchart showing the operation of the application execution unit 695. The operation of the application execution unit 695 will be explained with reference to the flowchart shown in Figure 24. When the item management application 675 is selected and starts operating, the application execution unit 695 communicates with the item management server 8 and obtains updated data (step SE1). Next, the application execution unit 695 displays the initial screen 710 shown in Figure 16 or Figure 17 on the touch panel 63 (step SE2).

[0164] Next, the application execution unit 695 determines whether the long-term storage notification screen 730 has been selected (step SE3). If the long-term storage notification screen 730 has been selected (step SE3 / YES), the application execution unit 695 displays the long-term storage notification screen 730 shown in Figure 18 on the touch panel 63 (step SE4). After that, the application execution unit 695 returns to the determination in step SE3.

[0165] Furthermore, if the long-term storage notification screen 730 is not selected (step SE3 / NO), the application execution unit 695 determines whether the loading / unloading status screen 750 is selected (step SE5). If the loading / unloading status screen 750 is selected (step SE5 / YES), the application execution unit 695 displays the loading / unloading status screen 750 shown in Figure 19 on the touch panel 63 (step SE6). After that, the application execution unit 695 returns to the determination in step SE3.

[0166] If the insertion / removal status screen 750 is not selected (step SE5 / NO), the application execution unit 695 determines whether the opening / closing status screen 770 is selected (step SE7). If the opening / closing status screen 770 is selected (step SE7 / YES), the application execution unit 695 displays the opening / closing status screen 770 shown in Figure 20 on the touch panel 63 (step SE8). After that, the application execution unit 695 returns to the determination in step SE3.

[0167] If the open / close status screen 770 is not selected (step SE8 / NO), the application execution unit 695 determines whether the stored items screen 820 is selected (step SE9). If the stored items screen 820 is selected (step SE9 / YES), the application execution unit 695 displays the selection screen 790 shown in Figure 21 and determines whether it displays items for each storage area (step SE11).

[0168] If the display is not for each storage area (step SE11 / NO), the application execution unit 695 displays the stored items screen 820A shown in Figure 22, and displays all items stored in refrigerator 2 in descending order of the time elapsed since the date of storage (step SE12). After that, the application execution unit 695 returns to the determination in step SE3.

[0169] Furthermore, if the display is for each storage area (step SE11 / YES), the application execution unit 695 accepts the selection of a storage area on the selection screen 790 (step SE13). Then, the application execution unit 695 displays the storage items screen 820B shown in Figure 23, and displays the items stored in the selected storage area in descending order of elapsed time since the storage date (step SE14).

[0170] Furthermore, if the determination in step SE9 is not the selection of the contained items screen 820B (step SE9 / NO), the application execution unit 695 executes the process corresponding to the received operation (step SE10) and returns to the determination in step SE3.

[0171] [11. Effects, etc.] As described above, the item management system 1 comprises an imaging device 3 and an item management server 8. The imaging device 3 captures images of the inside of the refrigerator compartment 21 of the refrigerator 2 and generates an image. The item management server 8 detects items being stored in the refrigerator 21, the storage area within the refrigerator 21 where the items are stored, and items being taken out of the storage area, based on the images captured by the imaging device 3. Based on the detection results, the item management server 8 stores in the second memory 85 the date on which the item was placed in the refrigerated room 21 and the storage area in which the item was stored. The item management server 8 displays information on the information processing device 6 about items that have not been removed from their storage area even after a predetermined period has elapsed since the date of receipt.

[0172] According to this system, information on items that have not been taken out of the storage area even after a predetermined period has elapsed since the date of arrival is displayed on the information processing device 6. This helps to prevent items from being forgotten and reduces the loss of goods.

[0173] The item management server 8 displays information about the storage area of ​​items that have not been removed from the storage area even after a predetermined period has elapsed since the date of receipt on the information processing device 6.

[0174] This approach helps prevent items from being forgotten and reduces waste. It also allows users to recognize storage areas that are not used frequently.

[0175] The item management server 8 counts the number of items stored in refrigerator 2 and the number of items taken out of refrigerator 2 for each storage area during a pre-configured period. The inventory management server 8 displays information on the information processing device 6 for storage areas where the total number of items received and items sent out is less than or equal to a pre-set value.

[0176] This allows users to be aware of storage areas that are not used very often.

[0177] The item management server 8 displays the date of entry of items stored in the refrigerator 2, along with an image showing the storage area in which the items are stored, on the information processing device in chronological order from the time elapsed since the items were entered into the refrigerator 2.

[0178] This allows users to be aware of the date items were received, their storage area, and the elapsed time, thereby reducing item loss.

[0179] The imaging device 3 and the item management server 8 are equipped with a left door distance sensor 31, a right door distance sensor 32, and a drawer distance sensor 33 that detect the open / closed state of at least one of the doors and drawers of the refrigerator 2. If the left door distance sensor 31, the right door distance sensor 32, and the drawer distance sensor 33 fail to detect the opening or closing of a door or drawer even after a pre-set period of time has elapsed, the item management server 8 causes the information processing device 6 to perform a notification operation.

[0180] This allows the information processing device 6 to perform a notification action if the opening or closing of the refrigerator door and drawers is not detected, thereby preventing items from being forgotten.

[0181] (Other embodiments) As described above, Embodiment 1 has been presented as an example disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in Embodiment 1. Therefore, other embodiments are illustrated below.

[0182] In the embodiments described above, a refrigerator 2 was given as an example of the storage facility of the present disclosure. However, the storage facility of the present disclosure is not limited to a refrigerator 2, and may be other storage facilities such as a freezer or a room-temperature storage facility.

[0183] In the embodiments described above, the refrigerator compartment 21 was given as an example of the storage compartment of the present disclosure. However, the storage compartment of the present disclosure is not limited to the refrigerator compartment 21, and may also be the ice-making compartment 24, the fresh freezing compartment 25, the freezer compartment 26, and the vegetable compartment 27. Furthermore, if the storage facility of the present disclosure is a storage facility other than the refrigerator 2, the storage compartment of the present disclosure corresponds to the storage compartment of the storage facility other than the refrigerator 2.

[0184] The process of detecting items from captured images and determining the storage area will be explained with reference to Figure 25. Figure 25 shows the boundary lines of the 5th area R5, the 6th area R6, the 7th area R7, and the 8th area R8. The following explanation will use the vegetable compartment 27, which has been pulled out from the inside of the storage unit, as an example.

[0185] The inbound / outbound determination unit 454 analyzes the captured image to identify the storage area of ​​the vegetable compartment 27 into which the item was stored. For example, it determines that the storage area of ​​an item stored in the fifth area R5 is the fifth area R5. Similarly, it makes the same determination for the sixth area R6, the seventh area R7, and the eighth area R8. In other words, it identifies the storage area from a single image capturing the entire vegetable compartment 27. This simplifies the system configuration of the item management system 1. Alternatively, the processing unit 871 of the item management server 8 may identify the storage area. In this configuration, the image of the vegetable compartment 27 is transmitted from the imaging device 3 to the item management server 8, and based on the received image, the processing unit 871 identifies the storage area and registers it in the food management DB or item management DB. The vegetable compartment 27 may be divided into two or other arbitrary regions.

[0186] In the embodiment described above, a configuration was illustrated in which the person presence determination unit 451 determines whether or not a person is present around the refrigerator 2 based on the detection value of the human presence sensor 34. However, the person presence determination unit 451 may also make the determination based on the detection value of an illuminance sensor or an electrostatic sensor provided on the gripping part of the refrigerator 2, instead of the human presence sensor 34. In the case of this electrostatic sensor, the refrigerator 2 and the imaging device 3 can communicate with each other, and the refrigerator 2 transmits the detection value of the electrostatic sensor to the imaging device 3.

[0187] In other embodiments, the refrigerator compartment camera 42 and the drawer camera 43 may be configured as a single camera. In this configuration, the camera's field of view may be changed depending on whether the refrigerator door 2 is open or closed. This eliminates the need for the imaging device 3 to have multiple cameras, simplifying the system configuration.

[0188] In other embodiments, the information processing device 6 may operate as a control device. In this configuration, the information processing device 6 stores data from the ingredient management DB 93, the item management DB 95, and the open status management DB 97. The information processing device 6 updates the data from the ingredient management DB 93, the item management DB 95, and the open status management DB 97 as appropriate, similar to the item management server 8 described above. This simplifies the system configuration as the item management system 1 does not need to include the item management server 8.

[0189] In other embodiments, the types of compartments formed in the refrigerator 2's casing 20 may include compartments other than the refrigerator compartment 21, ice-making compartment 24, fresh freezing compartment 25, freezer compartment 26, and vegetable compartment 27, or at least one of the ice-making compartment 24, fresh freezing compartment 25, freezer compartment 26, and vegetable compartment 27 may be omitted. Also, in other embodiments, the number of doors provided at the front opening of the refrigerator compartment 21 may be just one.

[0190] Furthermore, the first processor 69, the second processor 87, and the fourth processor 450 may be composed of a single processor or multiple processors. The first processor 69, the second processor 87, and the fourth processor 450 may also be hardware programmed to realize the corresponding functional units. That is, the first processor 69, the second processor 87, and the fourth processor 450 may be composed of, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0191] The configurations of the imaging device 3, information processing device 6, and item management server 8 shown in Figures 1 and 5 are examples, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually, and it is certainly possible to configure the system so that a single processor executes a program to realize the functions of each part. Furthermore, some of the functions realized by software in the above-described embodiment may be implemented by hardware, or some of the functions realized by hardware may be realized by software. In addition, the specific detailed configurations of other parts of the imaging device 3, information processing device 6, and item management server 8 can also be arbitrarily changed without departing from the spirit of this disclosure.

[0192] The control program 851 executed by the second control unit 83 can also be implemented by recording the control program 851 on a portable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, and semiconductor storage devices such as USB (Universal Serial Bus) memory and SSDs (Solid State Drives), but other recording media can also be used.

[0193] The first processor 69, the second processor 87, the third processor 350, and the fourth processor 450 may be composed of a single processor or multiple processors. These processors may also be hardware programmed to implement the corresponding functional units. That is, these processors may be composed of, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0194] The configurations of the information processing device 6 and the item management server 8 shown in Figure 1, and the configuration of the imaging device 3 shown in Figure 7, are merely examples, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually; it is also possible to have a configuration in which a single processor executes a program to realize the functions of each part. Furthermore, some of the functions realized by software in the above-described embodiment may be implemented as hardware, or some of the functions realized by hardware may be implemented as software.

[0195] The operational step units shown in Figures 13, 14, 15, and 24 are divided according to the main processing content to facilitate understanding of the operation, and the operation is not limited by the way the processing units are divided or the names of the processing units. Depending on the processing content, it may be further divided into more step units. Alternatively, it may be divided so that one step unit includes even more processing. Furthermore, the order of the steps may be changed as appropriate, as long as it does not impede the intent of this disclosure.

[0196] Since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents. [Industrial applicability]

[0197] As described above, the refrigerator and storage unit according to the present invention can be used for applications that utilize photographic images showing the conditions inside the storage unit. [Explanation of Symbols]

[0198] 1. Inventory Management System 2 Refrigerator 3. Imaging device 6. Information Processing Device 8. Item Management Server 10 Networks 20 box body 21 Refrigerator 22 Left door 23 Right door 30 Sensor board 31. Distance measuring sensor for the left door 32. Distance measuring sensor for the right door 33. Distance measuring sensor for drawers 34 motion sensors 35 Sensor board 40 Camera circuit board 41 Third Communication Module 42 Camera for Refrigerator 43 Drawer Camera 44 Light-emitting device 61. First Communication Module 63 Touch Panel 65 First Control Unit 67 First Memory 69 First Processor 81 Second Communication Module 83 Second Control Unit 85 Second Memory 87 Second Processor 91 Identification Information Database 93. Food Ingredient Management Database 95. Inventory Management Database 97 Open State Management Database 310 1st I / F circuit 330 Signal Processing Unit 340 Third Memory 345 Control Program 350 Third Processor 410 2nd I / F circuit 430 Third Control Unit 440 4th memory 441 Control Program 443 Imaging device ID 450 Fourth Processor 451 Person presence determination unit 452 Open / Close Determination Unit 453 Imaging Control Unit 454 Enter Departure Determination Unit 455 Acquisition Department 456 Data Generation Unit 459 Fourth Processor 675 Inventory Management App 695 Application Execution Unit 700 App screen

Claims

1. An imaging device that captures images of the inside of a storage room in a storage facility and generates captured images, and based on the captured images, detects articles being placed into the storage room, the storage area within the storage room where the articles are stored, and articles being removed from the storage area. A control device that, based on the detection results of the imaging device, displays information on an information processing device about items that have not been removed from the storage area even after a predetermined period has elapsed since being placed in the storage room, Equipped with, The control device, based on the image captured by the imaging device, counts the number of times items are brought in and taken out during the set period for each storage area in the storage room, and calculates the ratio of items being brought in and taken out during the set period for each storage area. The information processing device displays a screen showing the loading / unloading status, which includes a period change button that allows the setting period to be changed, and the loading / unloading ratio of the items for each storage area. The aforementioned inflow / outflow ratio is calculated based on the setting period selected by the period change button and is changed according to the setting period, in an item management system.

2. The article management system according to claim 1, wherein the control device causes the information processing device to display information about the storage area of ​​an article that has not been taken out of the storage area even after a predetermined period of time has elapsed since it was placed in the storage room.

3. The control device counts the number of items entered into the storage facility and the number of items taken out of the storage facility for each storage area during a preset period. The item management system according to claim 1 or 2, wherein the information processing device displays information on storage areas where the total number of items received and the total number of items dispatched is less than or equal to a preset value.

4. The item management system according to any one of claims 1 to 3, wherein the control device causes the information processing device to display the date of entry of an item into the storage facility and an image showing the storage area in which the item is stored, in order of elapsed time since the item was entered into the storage facility.

5. The imaging device includes an open / close sensor that detects the open / closed state of at least one of the doors and drawers of the storage unit. The article management system according to any one of claims 1 to 4, wherein the control device causes the information processing device to perform a notification operation if the opening or closing of the door or drawer cannot be detected by the opening / closing sensor even after a preset period of time has elapsed.

6. The steps include: imaging the inside of the storage room of the storage facility, A step of detecting, based on the captured image, an article to be placed in the storage room, a storage area within the storage room where the article is stored, and an article to be taken out of the storage area, Based on the captured images, the number of times items are brought in and taken out during the set period is counted for each storage area in the storage room, and the ratio of items being brought in and taken out during the set period is calculated for each storage area. The steps include displaying information on an information processing device about items that have not been removed from the storage area even after a predetermined period has elapsed since being placed in the storage room, It has, The step of displaying the above involves displaying a screen on the information processing device that shows a period change button that allows the setting period to be changed, and the percentage of items being taken in and out for each storage area. The aforementioned return / exit ratio is calculated based on the setting period selected by the period change button, and is changed according to the setting period, in this item management method.

7. On the computer, A procedure for detecting, based on an image captured from inside the storage room of a storage facility, an item being placed into the storage room, a storage area within the storage room where the item is stored, and an item being removed from the storage area. A procedure for calculating the ratio of items being moved in and out for each storage area within the storage room, based on the captured images, during the set period, and for each storage area, during the set period, A procedure for displaying information on an information processing device about items that have not been removed from the storage area even after a predetermined period has elapsed since being placed in the storage facility, Make it run, The procedure for displaying the above involves displaying a screen on the information processing device that shows a period change button that allows the setting period to be changed, and the percentage of items being taken in and out for each storage area. The program calculates the in / out ratio based on the setting period selected by the period change button, and modifies it according to the setting period.

Images