Inventory management system and inventory management method

Abstract

A loading / unloading management system (10) comprises: an imaging unit (31) for capturing a moving image when an opening portion of a refrigerator (20) is viewed from above; a food product detecting unit (37) for subjecting the moving image to image processing to detect whether a stationary object other than the refrigerator (20) appears in a first region of the moving image; a hand / arm detecting unit (38) for subjecting the moving image to image processing to detect whether a hand or arm of a user positioned in front of the opening portion appears in a second region of the moving image; and a determining unit (39) which, on the condition that a stationary object other than the refrigerator (20) appears in the first region in a period in which the hand or arm of the user does not appear in the second region, determines that the stationary object is an object to be subjected to inventory management.

Description

[Technical Field]

[0001] The present disclosure relates to an inventory management system and an inventory management method. [Background technology]

[0002] Conventionally, technologies relating to storage cabinets for storing objects have been proposed. Patent Document 1 discloses a refrigerator that can manage inventory inside the cabinet while reducing the user's workload, and can prevent a decrease in effective storage space inside the cabinet and an increase in manufacturing costs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-81762 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an inventory management system that can determine that an item is an object of inventory management based on an intentional action of a user. [Means for solving the problem]

[0005] An inventory management system according to one embodiment of the present disclosure includes a photographing unit that photographs a video of the opening of a storage unit as viewed from above, a first detection unit that performs image processing on the video to detect whether a stationary object other than the storage unit is visible in a first area of ​​the video, a second detection unit that performs image processing on the video to detect whether a user's hand or arm positioned in front of the opening is visible in a second area of ​​the video, and a determination unit that determines that a stationary object other than the storage unit is an object of inventory management when a stationary object other than the storage unit is detected to be visible in the first area during a period when the user's hand or arm is not visible in the second area. [Effects of the Invention]

[0006] A warehousing / retrieval management system according to one aspect of the present disclosure can determine that an item is a target of inventory management based on an intentional action of a user. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of a warehousing / shipping management system according to an embodiment. [Figure 2] FIG. 2 is an external view of a refrigerator and an incoming / outgoing management device included in the incoming / outgoing management system according to the embodiment. [Figure 3] FIG. 3 is a first diagram showing a moving image captured by the image capturing section. [Figure 4] FIG. 4 is a second diagram showing the moving image captured by the image capturing section. [Figure 5] FIG. 5 is a third diagram showing a moving image captured by the image capturing section. [Figure 6] FIG. 6 is a fourth diagram showing a moving image captured by the image capturing section. [Figure 7] FIG. 7 is a state transition diagram of four states used in managing food inventory. [Figure 8] FIG. 8 is a block diagram showing the functional configuration of the information processing unit. [Figure 9] FIG. 9 is a flowchart of a first operation example of the inventory management system according to the embodiment. [Figure 10] FIG. 10 is a diagram showing an example of food management information. [Figure 11] FIG. 11 is a sequence diagram of a second operation example of the inventory management system according to the embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a display screen showing the contents of food management information. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0009] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0010] (Embodiment) [composition] First, the configuration of the inventory management system according to the embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of the inventory management system according to the embodiment.

[0011] 1 is a system for managing food stored in a refrigerator 20. The receiving / shipping management system 10 includes the refrigerator 20, a receiving / shipping management device 30, a server device 40, and an information terminal 50.

[0012] Refrigerator 20 is an example of a storage cabinet, and is installed in a user's home or the like to refrigerate food. FIG. 2 is an external view of refrigerator 20 (and inventory management device 30). As shown in FIG. 2, refrigerator 20 comprises a main body 21, a door 22 that covers the opening of main body 21, and a plurality of shelves 24 provided in a storage chamber 23 within main body 21. When door 22 is opened, refrigerator 20 is ready to place food on each of the plurality of shelves 24 (i.e., store food in storage chamber 23). In the example of FIG. 2, refrigerator 20 comprises two doors 22 that cover storage chamber 23, but it may also comprise a single door 22 that covers storage chamber 23.

[0013] The receiving / shipping management device 30 is attached, for example, to the top surface of the refrigerator 20, and performs information processing to manage food stored in the refrigerator 20. The receiving / shipping management device 30 is, for example, retrofitted to an existing refrigerator 20, but may also be built into the refrigerator 20. In other words, the receiving / shipping management device 30 may be a device separate from the refrigerator 20, or may be a device manufactured integrally with the refrigerator 20. As shown in FIG. 1 , the receiving / shipping management device 30 includes an imaging unit 31, an opening / closing sensor 32, an information processing unit 33, a storage unit 34, a communication unit 35, and an alarm unit 36.

[0014] The image capturing unit 31 captures video from above the refrigerator 20 showing how food is being taken in and out of the refrigerator 20. That is, the image capturing unit 31 captures video of the refrigerator 20 with the door 22 open (video of the opening of the refrigerator 20) as viewed from above the refrigerator 20. The image capturing unit 31 is implemented, for example, by a camera with a fisheye lens, but may also be implemented by a camera with a normal lens. The video includes multiple images captured continuously, and each of the multiple images is also referred to as a frame.

[0015] Although the photographing unit 31 is attached to the housing of the receiving / receiving management device 30, it may be attached to the refrigerator 20 separately from the housing of the receiving / receiving management device 30. In other words, the photographing unit 31 may be retrofitted to the refrigerator 20. In this case, the photographing unit 31 has a structure that allows it to be retrofitted to the refrigerator 20, such as a magnet, a suction cup, or a clip.

[0016] The opening / closing sensor 32 is located at the top of the refrigerator 20 and detects whether the door 22 is open or closed (how open it is) by measuring the distance from the opening / closing sensor 32 to the door 22. The opening / closing sensor 32 is realized by, for example, an infrared distance measuring sensor. The opening / closing sensor 32 can individually detect whether the right door 22 and the left door 22 are open or closed by separately measuring the distance from the opening / closing sensor 32 to the right door 22 and the distance from the opening / closing sensor 32 to the left door 22.

[0017] The opening / closing sensor 32 may be realized by an angle sensor or the like instead of the infrared distance measuring sensor. The opening / closing sensor 32 may also be provided in the refrigerator 20, in which case the inventory management device 30 does not need to be provided with the opening / closing sensor 32.

[0018] The information processing unit 33 performs information processing for managing food stored in the refrigerator 20. The information processing unit 33 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 33 has, as functional components, a food detection unit 37, a hand / arm detection unit 38, and a determination unit 39. The functions of the food detection unit 37, the hand / arm detection unit 38, and the determination unit 39 are realized, for example, by the microcomputer or the like constituting the information processing unit 33 executing a computer program stored in the memory unit 34. The functions of the food detection unit 37, the hand / arm detection unit 38, and the determination unit 39 will be described in detail below.

[0019] The storage unit 34 is a storage device that stores the computer program executed by the information processing unit 33 and various information (such as food management information described below) required for the above information processing. The storage unit 34 is realized by, for example, a semiconductor memory.

[0020] The communication unit 35 is a communication circuit that enables the inventory management device 30 to communicate with the server device 40 via the wide area communication network 60. The communication unit 35 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard used for communication by the communication unit 35.

[0021] The notification unit 36 ​​notifies the user that a food item has been recognized as a target for inventory management, that an error has occurred related to the photographing unit 31, etc. The notification unit 36 ​​is realized, for example, by a speaker or a buzzer, and notifies the user by emitting sound, but may also be realized by a light-emitting element such as an LED (Light Emitting Diode), and notifies the user by emitting light.

[0022] Server device 40 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 40 performs information processing to present information about food stored in refrigerator 20. Server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0023] The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the inventory management device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used for communication by the communication unit 41.

[0024] Information processing unit 42 performs information processing to present information about food stored in refrigerator 20. Information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. Information processing unit 42 has a presentation unit 44 as a functional component. The function of presentation unit 44 is realized, for example, by the microcomputer or the like constituting information processing unit 42 executing a computer program stored in memory unit 43. The function of presentation unit 44 will be described in detail later.

[0025] The storage unit 43 is a storage device that stores the computer programs executed by the information processing unit 42 and various information required for the information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0026] Information terminal 50 is an information terminal owned by a user. The user uses information terminal 50 to check the food items stored in refrigerator 20 (contents of food management information) while away from home. Information terminal 50 is, for example, a portable information terminal such as a smartphone or tablet terminal, but may also be a stationary information terminal such as a personal computer.

[0027] [Determining the four states] Next, the inventory management operation of food items in refrigerator 20 performed by inventory management system 10 will be described. When a user wants to store or remove food items from refrigerator 20 for inventory management, the user temporarily holds the food items still above a specific area provided on the bottom surface of storage compartment 23. That is, the user intentionally holds the food items above the specific area. Inventory management device 30 can detect that the user has held the food items still above the specific area by performing image processing on the video captured by imaging unit 31. FIG. 3 shows an example of the video captured by imaging unit 31.

[0028] FIG. 3 is a diagram showing the entire imaging range of imaging unit 31. The video shown in FIG. 3 shows the opening of refrigerator 20 and storage room 23 located behind the opening. The video also shows specific area P. Specific area P is an area on the bottom surface of storage room 23 (the inner surface of refrigerator 20) where a predetermined pattern is drawn, and is a real area. Although a user (person) is not shown in FIG. 3, the video also shows a user located in front of the opening, as shown in FIGS. 4 to 6, which will be described later.

[0029] In the description of image processing of moving images to be given later, the image processing is performed on a predetermined range A0 corresponding to a portion of the shooting range of the image capturing unit 31, rather than the entire shooting range of the image capturing unit 31. In other words, in the description of image processing to be given later, the term frame, image, or moving image will refer to the portion of the predetermined range A0, rather than the entire shooting range of the image capturing unit 31.

[0030] A first area A1 and a second area A2 are set within the predetermined range A0. The first area A1 and the second area A2 are virtual areas used in image processing and are shown by dashed lines in Fig. 3. The first area A1 is set to include a specific area P. The first area A1 and the specific area P are set at locations suitable for a user to hold over an item that they wish to use as an inventory management target.

[0031] The shapes and arrangements of the specific area P and the first area A1 are merely examples, and the shapes and arrangements of the specific area P and the first area A1 may be appropriately determined empirically or experimentally by a designer of the inventory management system 10. Note that an explicit specific area P may not be provided, and the first area A1 may be set in at least a part of the bottom surface of the storage room 23 (a surface of a single color, such as white).

[0032] When a user puts food into storage, the video captured by the image capturing unit 31 is expected to change as shown in FIGS.

[0033] 3 shows a video before the user places food in the storage. This state is defined as the STANDBY state. In the STANDBY state, the user's hands or arms, or the food held by the user (hereinafter referred to as the user's arms, etc.), are not visible within the predetermined area A0.

[0034] 4 shows a video in which a user is moving food above a specific area, and this state is defined as a READY state. The READY state is a state in which the user's arm or the like is visible in the first area A1 of the video and the user's arm or the like is moving.

[0035] 5 shows a video in which a user holds food still above a specific area, and this state is defined as the SET state. The SET state is a state in which the user's arm (in this case, the food) is displayed in the first area A1 of the video and the user's arm is still (or, more accurately, moves very little).

[0036] FIG. 6 is a diagram showing a video of a user storing food in storage room 23 (placing it on shelf 24), and this state is defined as the INOUT state. The INOUT state is a state in which the user's arm or the like is visible in first area A1 of the video and the user's arm or the like is moving. After the INOUT state, when the user has finished placing the food, the state returns to the STANDBY state. The STANDBY state, READY state, SET state, and INOUT state described above can be represented in a state transition diagram as shown in FIG. 7. FIG. 7 is a state transition diagram of the STANDBY state, READY state, SET state, and INOUT state.

[0037] Although no example animation is shown, when a user takes food out, the state transitions in the same order as when food is stored: STANDBY state, READY state, SET state, INOUT state, and STANDBY state.

[0038] [Image processing to determine the four states] The inventory management device 30 can manage food by determining the above four states based on the difference in pixel values ​​between frames included in the video (presence or absence of movement) and the difference in pixel values ​​between the first reference image and frames included in the video (presence or absence of the user's arm, etc.). The method for determining the four states will be described below with reference to FIG. 8. FIG. 8 is a block diagram showing the functional configuration of the information processing unit 33 included in the inventory management device 30. FIG. 8 also shows the imaging unit 31 and the open / close sensor 32.

[0039] 8 below (description of image processing for determining the four states), the terms frame, image, video, etc. refer to a portion of the predetermined range A0 rather than the entire shooting range of the imaging unit 31. In image processing, the image is temporarily stored in the storage unit 34, but the following description of this point will be omitted.

[0040] 8, the information processing unit 33 has a food detection unit 37 and a determination unit 39. The food detection unit 37 has an inter-frame difference calculation unit 37a, a reference image acquisition unit 37b, and a reference image difference calculation unit 37c.

[0041] First, the determination of the presence or absence of movement of an object in a video by the inter-frame difference calculation unit 37a will be described. The inter-frame difference calculation unit 37a calculates the difference between the pixel value of a pixel included in a frame captured by the imaging unit 31 at time t−1 and the pixel value of a pixel included in a frame at the current time t. For example, the inter-frame difference calculation unit 37a calculates the difference in pixel value of corresponding pixels between a frame at time t−1 and a frame at the current time t. When the number of pixels included in a frame is m (m is a natural number), m differences are calculated. The inter-frame difference calculation unit 37a determines that there is movement of an object (such as a user's arm) in the video if n / m is equal to or greater than a predetermined ratio, and determines that there is no movement of an object (such as a user's arm) in the video if n / m is less than the predetermined ratio. The predetermined threshold and the predetermined ratio may be determined empirically or experimentally as appropriate.

[0042] Next, a description will be given of how reference image acquisition unit 37b and reference image difference calculation unit 37c determine whether an object such as a user's arm is captured in first area A1 of the video. Reference image acquisition unit 37b acquires a first reference image captured by imaging unit 31. The first reference image is an image in which no object other than refrigerator 20 is captured in first area A1, as shown in FIG. 3 .

[0043] For example, when open / close sensor 32 detects that at least one of two doors 22 is open, reference image acquisition unit 37b causes photographing unit 31 to photograph a first reference image and acquires the first reference image. In other words, reference image acquisition unit 37b acquires a new first reference image each time door 22 is opened. This makes it possible to acquire a first reference image that shows the latest interior of refrigerator 20. Since it is a rule that food should not be placed on specific area P when storing food in refrigerator 20, the image captured immediately after door 22 is opened does not show any objects other than refrigerator 20 in first area A1, which includes specific area P, and can therefore be used as the first reference image.

[0044] It is not essential that the first reference image is acquired in response to the opening of door 22, and for example, the first reference image may be acquired by photographing unit 31 when inventory management device 30 is newly installed (at the time of initial setup), and this first reference image may be used thereafter. The first reference image may also be acquired at any timing, and for example, the first reference image acquired in response to the opening of door 22 may be updated periodically while refrigerator door 22 is open.

[0045] When the first reference image is captured at any timing, reference image acquisition unit 37b acquires (uses) the first reference image after confirming that no objects other than refrigerator 20 appear in first region A1 of the captured image. For example, if specific region P is an area in which a predetermined pattern is drawn, reference image acquisition unit 37b can determine that no objects other than refrigerator 20 appear in first region A1 if the predetermined pattern is detected in first region A1 by pattern matching. Also, if there is no explicit specific region P and first region A1 is set in part of the bottom surface (a surface of a single color, such as white) of storage room 23, reference image acquisition unit 37b can determine that no objects other than refrigerator 20 appear in first region A1 if no edges are detected in first region A1 by edge detection.

[0046] The reference image difference calculation unit 37c calculates the difference between the pixel values ​​of pixels included in the first region A1 of the first reference image acquired by the reference image acquisition unit 37b and the pixel values ​​of pixels included in the first region A1 of the frame at the current time t. For example, the reference image difference calculation unit 37c calculates the difference between the pixel values ​​of corresponding pixels between the first region A1 of the first reference image and the first region A1 of the frame at the current time t. If the number of pixels included in the first region A1 is p (p is a natural number), p differences are calculated. The reference image difference calculation unit 37c determines that the number of differences greater than a predetermined threshold, q (q is a natural number), of the p differences, is q (q is a natural number). If q / p is equal to or greater than a predetermined ratio, it determines that the user's arm, etc. is present in the first region A1. If q / p is less than the predetermined ratio, it determines that the user's arm, etc. is not present. The predetermined threshold and the predetermined ratio may be determined empirically or experimentally as appropriate.

[0047] Next, we will explain the determination of the four states by the determination unit 39. The determination unit 39 determines which of the four states the current state is based on the determination result of the inter-frame difference calculation unit 37a (presence or absence of an object in the video), the determination result of the reference image difference calculation unit 37c (presence or absence of the user's arm or the like in the first area A1), and the state transition diagram (FIG. 7).

[0048] Immediately after door 22 is opened, the user's arm or the like is not visible in the video (predetermined range A0), so there is no movement of objects visible in the video, and the user's arm or the like is not located in first area A1. Therefore, determination unit 39 determines that the refrigerator is in the STANDBY state. After the STANDBY state, when the user places their arm or the like inside refrigerator 20, the user's arm or the like visible in the video moves and is located in first area A1, so determination unit 39 determines that the refrigerator has transitioned to the READY state.

[0049] Thereafter, when the user intends to make the food a management target and stops the device above the specific area P, the movement of the user's arms, etc. shown in the video stops (becomes less) and the user's arms, etc. (in this case, the food) are located in the first area A1, so the judgment unit 39 judges that the device has transitioned to the SET state.

[0050] When the determination unit 39 determines that the SET state has continued for a certain period of time, it determines that the state is in the INOUT state. The INOUT state is a state in which the user is performing an action to put food in or take food out, and the user's arm, etc. shown in the video is moving and is positioned in the first area A1. On the other hand, when the determination unit 39 determines that the SET state has not continued for a certain period of time (the duration of the SET state has not reached the certain period of time), it determines that the state has returned to the READY state.

[0051] After the INOUT state, when the user pulls his / her hand out of the refrigerator 20, the user's arm or the like is not captured in the video (predetermined range A0), so there is no movement of the objects captured in the video, and the user's arm or the like is not located in the first area A1. Therefore, the determination unit 39 determines that the refrigerator is in the STANDBY state.

[0052] As described above, information processing unit 33 has food detection unit 37 that performs image processing on the video captured by imaging unit 31 to detect whether or not a stationary object other than refrigerator 20 is captured in first area A1 of the video (specifically, whether food has stopped in first area A1). This allows information processing unit 33 (determination unit 39) to determine that a stationary object other than refrigerator 20 is an object subject to inventory management when it is detected that the object is captured in first area A1. More specifically, information processing unit 33 can determine that a food object subject to inventory management has been stored (or removed) when the state transitions from STANDBY state to READY state, SET state, INOUT state, and STANDBY state in this order.

[0053] [Configuration to suppress false positives] Incidentally, when a user stores food that is not subject to inventory management, the user does not stop the food above specific area P, and the food does not transition from the READY state to the SET state, but remains determined to be in the READY state, and the user places their hand inside refrigerator 20 as shown in Figure 6. Here, if the user stops with their hand inside refrigerator 20 as shown in Figure 6, determination unit 39 may erroneously determine that the state has transitioned from the READY state to the SET state.

[0054] Therefore, the information processing unit 33 sets a second area A2 in the video (predetermined range A0), and when the user's arm or the like is shown in the second area A2, the information processing unit 33 does not determine that the SET state has occurred even if the user's arm or the like in the video stops moving and the user's arm or the like is located in the first area A1. In other words, the determination unit 39 determines whether or not the SET state has been transitioned to by adding the requirement that the user's arm or the like in the video stops moving and the user's arm or the like is located in the first area A1, as well as the requirement that the user's arm or the like is not shown in the second area A2.

[0055] As shown in FIGS. 3 to 6, second area A2 is located further back than first area A1 in the depth direction of refrigerator 20 (the vertical direction in FIGS. 3 to 6). The length of second area A2 in the depth direction of refrigerator 20 is shorter than the length of first area A1 in the depth direction, and the length of second area A2 in the width direction of refrigerator 20 (the horizontal direction in FIGS. 3 to 6) is longer than the length of first area A1 in the width direction. Second area A2 is set in a location where it is expected that the user's hands or arms will not be visible when the user holds an item that they wish to manage inventory and places the item in first area A1.

[0056] It should be noted that such second area A2 is merely an example, and the shape and arrangement of the second area A2 may be appropriately determined empirically or experimentally by the designer of the inventory management system 10 or the like.

[0057] As shown in FIG. 8 above, information processing unit 33 has hand / arm detection unit 38 as a component for determining whether the user's arm or the like is captured in second area A2. Hand / arm detection unit 38 has reference image acquisition unit 38a and reference image difference calculation unit 38b. Below, we will explain how reference image acquisition unit 38a and reference image difference calculation unit 38b determine whether an object such as the user's arm is captured in second area A2 of the video. Reference image acquisition unit 38a acquires a second reference image captured by imaging unit 31. The second reference image is an image in which no objects other than refrigerator 20 are captured in second area A2, as shown in FIG. 3.

[0058] For example, when the open / close sensor 32 detects that at least one of the two doors 22 is open, the reference image acquisition unit 38a causes the photographing unit 31 to photograph a second reference image and acquires the second reference image. In other words, the reference image acquisition unit 38a acquires a new second reference image every time the door 22 is opened. This makes it possible to acquire a second reference image that shows the latest interior of the refrigerator 20.

[0059] It is not essential that the second reference image is acquired in response to the opening of door 22. For example, the second reference image may be acquired by photographing unit 31 when inventory management device 30 is newly installed (at the time of initial setup), and this reference image may continue to be used thereafter. The second reference image may be acquired at any timing. For example, the reference image acquired in response to the opening of door 22 may be periodically updated while refrigerator door 22 is open. The first reference image may also be used as the second reference image. In this case, acquisition of the second reference image may be omitted.

[0060] The reference image difference calculation unit 38b calculates the difference between the pixel values ​​of pixels included in the second region A2 of the second reference image acquired by the reference image acquisition unit 38a and the pixel values ​​of pixels included in the second region A2 of the frame at the current time t. For example, the reference image difference calculation unit 38b calculates the difference between the pixel values ​​of corresponding pixels between the second region A2 of the second reference image and the second region A2 of the frame at the current time t. If the number of pixels included in the second region A2 is x (x is a natural number), x differences are calculated. The reference image difference calculation unit 38b determines that the number of differences greater than a predetermined threshold, y (y is a natural number), out of the x differences, is y (y is a natural number). If y / x is equal to or greater than a predetermined ratio, it determines that a user's arm, etc. is present in the second region A2. If y / x is less than the predetermined ratio, it determines that a user's arm, etc. is not present. The predetermined threshold and the predetermined ratio may be determined empirically or experimentally as appropriate.

[0061] In this way, hand and arm detection unit 38 performs image processing on the video captured by imaging unit 31 to detect whether or not a hand or arm is captured in second area A2 of the video. As a result, determination unit 39 can determine that a stationary object other than refrigerator 20 is an object subject to inventory management when it is detected that the object is captured in first area A1 during a period when no hand or arm is captured in second area A2 (specifically, food has stopped in first area A1). This prevents erroneous determination that the refrigerator has transitioned from the READY state to the SET state.

[0062] [Example 1] Next, a food inventory management operation will be described as a specific operation example 1 of the receiving / delivery management system 10. FIG.

[0063] When the information processing unit 33 of the inventory management device 30 determines that at least one of the two doors 22 is open based on the detection result of the opening / closing sensor 32 (S11), it activates the photographing unit 31 and starts photographing a video using the photographing unit 31 (S12).

[0064] Next, the information processing unit 33 performs a food detection process (S13). In the food detection process, the information processing unit 33 determines whether food is being stored or removed by determining the above four states.

[0065] When the user stops a food item above the specific area P and the state transitions from the SET state to the INOUT state, the information processing unit 33 may use the notification unit 36 ​​to notify the user that the food item has been recognized as a target for inventory management. For example, if the notification unit 36 ​​is realized by a speaker, the notification unit 36 ​​outputs a sound or a voice message indicating that the food item has been recognized as a target for inventory management. If the notification unit 36 ​​is realized by a light-emitting element, the notification unit 36 ​​emits light to notify the user that the food item has been recognized as a target for inventory management.

[0066] This allows the user to easily understand that the food item has been successfully frozen (the food item has been included in inventory management).

[0067] Furthermore, the information processing unit 33 temporarily stores in the storage unit 34, for example, a frame when the state transitions from the SET state to the INOUT state.

[0068] Thereafter, when the information processing unit 33 determines that both doors 22 are closed based on the detection result of the open / close sensor 32 (S14), it performs an update process of the food management information (S15). Fig. 10 is a diagram showing an example of the food management information, in which food IDs and images of the foods are stored in association with each other. The food management information is information that lists the foods in the refrigerator 20 at the time of step S11, in other words, inventory list information.

[0069] In step S15, the information processing unit 33 cuts out the portion of the frame temporarily stored in step S13 in which the food appears (the portion near the first area A1) as an image of the food. When a new food item is received in steps S11 to S14, the information processing unit 33 assigns an ID to the cut-out image of the food and adds it to the food management information. On the other hand, when a food item is removed from the food management information in steps S11 to S14, the information processing unit 33 deletes from the food management information the image of the food that is most similar to the cut-out image of the food, from among one or more food images included in the food management information. In this case, the information processing unit 33 performs a process of determining the similarity of the images using various existing algorithms.

[0070] Whether food has been stored or removed is determined, for example, in the INOUT state, based on the presence or absence of a user's arm or the like in a third area (not shown, for example, an area near shelf 24) in the video. For example, determination unit 39 determines that food has been stored if a user's arm or the like is present in the third area, and determines that food has been removed if a user's arm or the like is not present in the third area. The method using the reference image described above can be used to determine the presence or absence of a user's arm or the like in the third area.

[0071] As described above, the receiving / dispatching management system 10 can manage the inventory of food items in the refrigerator 20.

[0072] If an error occurs in the photographing unit 31, such as when the photographing unit 31 is unable to shoot a video, the information processing unit 33 may use the notification unit 36 ​​to notify the user that an error has occurred in the photographing unit 31. For example, if the notification unit 36 ​​is implemented by a speaker, the notification unit 36 ​​outputs a voice message or the like indicating that an error has occurred. If the notification unit 36 ​​is implemented by a light-emitting element, the notification unit 36 ​​notifies the user that an error has occurred by emitting light.

[0073] In this way, if an error related to the imaging unit 31 occurs and the user is notified of the error, the user can take measures to deal with the error.

[0074] [Example 2] Next, a presentation operation of food stored in the refrigerator 20 will be described as a second operational example of the receiving / delivery management system 10. FIG.

[0075] The communication unit 35 of the inventory management device 30 transmits the food management information to the server device 40 (S21). The communication unit 41 of the server device 40 receives the food management information, and the presentation unit 44 stores the received food management information in the memory unit 43 (S22). The communication unit 35 of the inventory management device 30 transmits the food management information to the server device 40, for example, every time the food management information is updated. This causes the latest food management information to be stored in the memory unit 43 of the server device 40. The food management information may also be transmitted from the inventory management device 30 to the server device 40 periodically, such as once at a predetermined time.

[0076] On the other hand, when the user performs a predetermined operation on the information terminal 50 while away from home to check the food stored in the refrigerator 20 (the contents of the food management information), the information terminal 50 accepts the predetermined operation (S23) and transmits a presentation request to the server device 40 based on the accepted predetermined operation (S24).

[0077] The communication unit 41 of the server device 40 receives the presentation request. Based on the received presentation request, the presentation unit 44 transmits presentation information for presenting the contents of the food management information stored in the storage unit 43 to the information terminal 50 (S25). More specifically, the transmission of the presentation information is performed by the communication unit 41.

[0078] The information terminal 50 receives the presentation information and displays a display screen showing the contents of the food management information on the display unit provided in the information terminal 50 based on the received presentation information (S26). Fig. 12 is a diagram showing an example of the display screen showing the contents of the food management information. On the display screen in Fig. 12, images of the food included in the food management information (images cut out from video captured by the image capture unit 31) are superimposed on a background image simulating the contents of the refrigerator 20.

[0079] In this way, the user of the inventory management system 10 can check the food stored in the refrigerator 20 at home while away from home.

[0080] [Variations] In the above embodiment, a configuration for suppressing erroneous determination of an item to be subject to inventory management has been described. The inventory management system 10 may also be provided with a configuration for excluding food items that have been determined to be items to be subject to inventory management from the items to be subject to inventory management.

[0081] As described above, when a food item is determined to be a target item, a notification to that effect is issued by the notification unit 36. At this time, the inventory management system 10 may exclude from the target items the food item that has been determined to be a target item for inventory management, on the condition that the user has made a predetermined gesture in the first area A1 (such as making a peace sign with their hand and holding it still) within a predetermined period of time after the notification.

[0082] [Other variations] In the above embodiment, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0083] Furthermore, the refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be implemented as other storage cabinets that do not have a refrigeration function. Furthermore, the items (in other words, objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0084] [Effects, etc.] Below, examples of techniques that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these techniques will be described.

[0085] Technique 1 is an inventory management system 10 including a photographing unit 31 that photographs a video of the opening of a storage cabinet as viewed from above, a food detection unit 37 that performs image processing on the video to detect whether a stationary object other than the storage cabinet is visible in a first area A1 of the video, a hand / arm detection unit 38 that performs image processing on the video to detect whether a user's hand or arm positioned in front of the opening is visible in a second area A2 of the video, and a determination unit 39 that determines that a stationary object other than the storage cabinet is an object subject to inventory management when a stationary object other than the storage cabinet is detected to be visible in the first area A1 during a period when the user's hand or arm is not visible in the second area A2. The food detection unit 37 is an example of a first detection unit, and the hand / arm detection unit 38 is an example of a second detection unit.

[0086] Such an inventory control system 10 can determine that an item that the user places in the first area A1 is an object of inventory control. In other words, the inventory control system 10 can determine that an item is an object of inventory control based on the user's intentional action. Furthermore, the second area A2 is set, for example, in a location where the user's hand or arm is not expected to be captured when the user intentionally places an item that the user wants to be an object of inventory control in the first area A1. By setting the second area A2 in this manner, the inventory control system 10 can prevent erroneous determination of an object of inventory control.

[0087] Technique 2 is the inventory management system 10 of Technique 1, in which the hand / arm detection unit 38 acquires a second reference image captured in advance by the photographing unit 31, and detects whether a hand or arm is captured in the second area A2 based on the difference between the pixel values ​​of the pixels in the second area A2 in the acquired second reference image and the pixel values ​​of the pixels in the second area A2 in the image included in the video.

[0088] Such a warehousing / retrieval management system 10 can detect whether or not a hand or arm is captured in the second area A2 by comparing the second reference image with the current image (frame).

[0089] Technique 3 is the inventory management system 10 of technique 2, in which the hand / arm detection unit 38 can change the second reference image.

[0090] Such a warehousing / retrieval control system 10 can change (update) the second reference image.

[0091] Technology 4 is the inventory management system 10 of Technology 3, in which a door 22 is provided at the opening, and the hand / arm detection unit 38 uses the image captured by the photographing unit 31 when the door 22 is opened as a second reference image, and updates the second reference image every time the door 22 is opened.

[0092] Such a warehousing / retrieval control system 10 can update the second reference image every time the door 22 is opened.

[0093] Technology 5 is an inventory management system 10 according to any one of technologies 1 to 4, in which the food detection unit 37 acquires a first reference image previously photographed by the photographing unit 31, and detects the presence or absence of stationary objects other than storage units in the first area A1 based on the difference between the pixel values ​​of pixels in the first area A1 in the acquired first reference image and the pixel values ​​of pixels in the first area A1 in an image included in the video, and the difference in pixel values ​​between multiple images included in the video.

[0094] Such an inventory management system 10 can detect the presence or absence of stationary objects by comparing a first reference image with the current image (frame) and by comparing multiple images (frames) contained in a video.

[0095] Technique 6 is inventory control system 10 of technique 5, in which food detection unit 37 can change the first reference image.

[0096] Such a warehousing / retrieval management system 10 can change (update) the first reference image.

[0097] Technology 7 is the inventory management system 10 of Technology 6, in which a door 22 is provided at the opening, and the food detection unit 37 uses an image captured by the photographing unit 31 when the door 22 is opened as a first reference image, and updates the first reference image each time the door 22 is opened.

[0098] Such a warehousing / retrieval control system 10 can update the first reference image every time the door 22 is opened.

[0099] Technique 8 is the inventory management system 10 of any of Techniques 1 to 7, in which the first area A1 is an area including the inner surface of the storage cabinet shown in the video.

[0100] Such a warehousing / retrieval management system 10 can use the area in the video showing the inner surface of the storage as the first area.

[0101] Technique 9 is an inventory management system 10 according to any of techniques 1 to 8, in which a specific area P having a predetermined pattern drawn thereon is provided on the inner surface of the storage unit shown in the video, and the first area A1 includes the specific area P.

[0102] Such a specific area P is useful as a mark indicating the position where the user should stop an article that he or she wishes to manage in inventory.

[0103] Technique 10 is the incoming / outgoing inventory management system 10 of any of Techniques 1 to 9, in which the second area A2 is located further back than the first area A1 in the depth direction of the storage.

[0104] If the second area A2 is set in this manner, when a user holds an item to be managed as an inventory control target and positions the item in the first area A1, the user's hand or arm is not likely to be seen in the second area A2. Therefore, the inventory control system 10 can prevent erroneous determination of an item to be managed as an inventory control target.

[0105] Technique 11 is the incoming / outgoing inventory management system 10 of any one of techniques 1 to 10, in which the length of the second area A2 in the depth direction of the storage is shorter than the length of the first area A1 in the depth direction.

[0106] The incoming / outgoing inventory management system 10 can prevent erroneous determination of an object of inventory management based on the second area A2 whose length in the depth direction is shorter than that of the first area A1.

[0107] Technique 12 is the incoming / outgoing inventory management system 10 of any one of techniques 1 to 11, in which the length of the second area A2 in the width direction of the storage cabinet is longer than the length of the first area A1 in the width direction.

[0108] The incoming / outgoing inventory management system 10 can prevent erroneous determination of an object of inventory management based on the second area A2 that is longer in the width direction than the first area A1.

[0109] Technique 13 is the inventory management system 10 of any one of techniques 1 to 12, further comprising a notification unit 36 ​​that notifies that an error related to the photographing unit 31 has occurred.

[0110] Such a warehousing / retrieval management system 10 can notify that an error related to the photographing unit 31 has occurred.

[0111] Technique 14 is the inventory management system 10 of any of techniques 1 to 13, in which the storage cabinet is a refrigerator 20.

[0112] The incoming / outgoing inventory management system 10 can determine whether or not an item is an object of inventory management of the refrigerator 20.

[0113] Technique 15 is a computer-implemented inventory management method. The inventory management method includes a photographing step of photographing a video of the opening of a storage cabinet viewed from above, a first detection step of detecting whether a stationary object other than the storage cabinet is captured in a first area A1 of the video by performing image processing on the video, a second detection step of detecting whether a user's hand or arm positioned in front of the opening is captured in a second area A2 of the video by performing image processing on the video, and a determination step of determining that a stationary object other than the storage cabinet is an object of inventory management when a stationary object other than the storage cabinet is detected to be captured in the first area A1 during a period when the user's hand or arm is not captured in the second area A2.

[0114] Such an inventory management method can determine that an item is an object of inventory management based on an intentional action of a user.

[0115] Technique 16 is a program for causing a computer to execute the inventory management method described in Technique 15.

[0116] Such a program can determine that an item is an object of inventory control based on the intentional actions of a user.

[0117] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0118] For example, in the above embodiment, the inventory management system is realized by multiple devices, but it may also be realized by a single device. For example, the inventory management system may be realized as a single device corresponding to the inventory management device of the above embodiment, or as a single device corresponding to the server device. When the inventory management system is realized by multiple devices, the components (especially functional components) of the inventory management system may be allocated in any way to the multiple devices. For example, some or all of the processing described in the above embodiment as being executed by the inventory management device may also be executed by the server device.

[0119] Furthermore, for example, the communication method between the devices in the above-described embodiment is not particularly limited. Furthermore, a relay device (not shown) may be involved in the communication between the devices. Furthermore, the information transmission path described in the above-described embodiment is not limited to the transmission path shown in the sequence diagram.

[0120] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0121] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0122] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0123] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0124] For example, the present disclosure may be realized as a method for managing the receipt and delivery of objects executed by a computer, or as a program for causing a computer to execute the method for managing the receipt and delivery of objects. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0125] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0126] The inventory management system of the present disclosure is useful as a device capable of managing items in a storage facility. [Explanation of symbols]

[0127] 10. Inventory management system 20 Refrigerator (storage) 21 Main body 22 doors 23 Storage Room 24 shelves 30 Warehouse entry / exit management device 31 Photography Department 32 Open / close sensor 33, 42 Information Processing Department 34, 43 Storage section 35, 41 Communications Department 36 Information Department 37 Food detection unit (first detection unit) 38 Hand / arm detection unit (second detection unit) 39 Judgment section 40 Server device 44 Presentation section 50 Information terminal 60 Wide Area Communication Network

Claims

1. a camera unit that captures a video of the opening of the storage cabinet as seen from above; a first detection unit that performs image processing on the video to detect whether a stationary object other than the storage unit is captured in a first area of ​​the video; a second detection unit that performs image processing on the video to detect whether a hand or arm of a user positioned in front of the opening is captured in a second region of the video; a determination unit that determines that a stationary object other than the storage cabinet is an object to be managed in inventory when a stationary object other than the storage cabinet is detected in the first area during a period when the user's hand or arm is not displayed in the second area. Inventory management system.

2. The second detection unit acquiring a second reference image captured in advance by the imaging unit; Whether a hand or an arm is captured in the second region is detected based on a difference between pixel values ​​of pixels in the second region in the acquired second reference image and pixel values ​​of pixels in the second region in an image included in the video. The inventory management system according to claim 1.

3. The second detection unit can change the second reference image. The inventory management system according to claim 2.

4. A door is provided at the opening, The second detection unit uses the image captured by the image capturing unit when the door is opened as the second reference image, and updates the second reference image every time the door is opened. The inventory management system according to claim 3.

5. The first detection unit acquiring a first reference image captured in advance by the imaging unit; The presence or absence of a stationary object other than the storage unit in the first area is detected based on a difference between the pixel values ​​of pixels in the first area in the acquired first reference image and the pixel values ​​of pixels in the first area in an image included in the video, and based on a difference in pixel values ​​between a plurality of images included in the video. The inventory management system according to claim 1.

6. The first detector can change the first reference image. The inventory management system according to claim 5.

7. A door is provided at the opening, The first detection unit uses the image captured by the image capturing unit when the door is opened as the first reference image, and updates the first reference image every time the door is opened. The inventory management system according to claim 6.

8. The first area is an area including the inner surface of the storage cabinet shown in the video. The inventory management system according to any one of claims 1 to 7.

9. A specific area on which a predetermined pattern is drawn is provided on the inner surface of the storage cabinet shown in the video, The first region includes the specific region. The inventory management system according to claim 8.

10. In the depth direction of the storage cabinet, the second area is located further back than the first area. The inventory management system according to any one of claims 1 to 7.

11. The length of the second area in the depth direction of the storage cabinet is shorter than the length of the first area in the depth direction. The inventory management system according to any one of claims 1 to 7.

12. The length of the second area in the width direction of the storage cabinet is longer than the length of the first area in the width direction. The inventory management system according to any one of claims 1 to 7.

13. The image capturing device further includes a notification unit that notifies the user that an error has occurred in the image capturing unit. The inventory management system according to any one of claims 1 to 7.

14. The storage cabinet is a refrigerator. The inventory management system according to any one of claims 1 to 7.

15. A computer-implemented inventory management method, comprising: a photographing step of photographing a video of the opening of the storage shed viewed from above; a first detection step of detecting whether a stationary object other than the storage unit is captured in a first region of the video by performing image processing on the video; a second detection step of detecting whether a hand or arm of a user positioned in front of the opening is captured in a second region of the video by performing image processing on the video; and a determining step of determining that a stationary object other than the storage cabinet is an object to be managed in inventory, based on the condition that a stationary object other than the storage cabinet is detected to be captured in the first area during a period in which the user's hand or arm is not captured in the second area. Inventory management method.

16. A program for causing the computer to execute the inventory management method according to claim 15.

Images