Warehouse in-out management system and warehouse in-out management method

Abstract

The warehouse-in and warehouse-out management system (10) is provided with: an imaging unit (31) that captures a moving image when an opening of a refrigerator (20) is observed from above; a food detection unit (37) that detects whether a stationary object other than the existing refrigerator (20) is reflected in a first region of the moving image by performing image processing on the moving image; an arm detection unit (38) that detects whether or not a user's hand or arm located in front of the opening is present in a second region of the moving image by performing image processing on the moving image; and a determination unit (39) that determines that a stationary object other than the refrigerator (20) is present in the first area as an object to be subjected to inventory management on the condition that the stationary object is detected while the hand or arm of the user is not present in the second area.

Description

Technical Field

[0001] The present disclosure relates to an inbound and outbound management system and an inbound and outbound management method. Background Art

[0002] Conventionally, technologies related to a storage for storing objects have been proposed. Patent Document 1 discloses a refrigerator that can manage the inventory inside the refrigerator while reducing the trouble for the user, and can suppress a reduction in the effective storage space inside the refrigerator and an increase in the manufacturing cost.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2015-81762 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] The present disclosure provides an inbound and outbound management system that can determine an object to be managed in inventory based on a deliberate action of a user.

[0008] Means for Solving the Problems

[0009] An inbound and outbound management system according to one aspect of the present disclosure includes: a imaging unit that captures a moving image when observing the opening of a storage from above; a first detection unit that detects whether a stationary object other than the storage is reflected in a first region of the moving image by performing image processing on the moving image; a second detection unit that detects whether a hand or an arm of a user located in front of the opening is reflected in a second region of the moving image by performing image processing on the moving image; and a determination unit that determines the stationary object as an object to be managed in inventory on the condition that the stationary object other than the storage is detected in the first region during a period when the hand or the arm of the user is not reflected in the second region.

[0010] Effects of the Invention

[0011] An inbound and outbound management system according to one aspect of the present disclosure can determine an object to be managed in inventory based on a deliberate action of a user. Brief Description of the Drawings

[0012] Figure 1 is a block diagram showing the functional configuration of the inbound and outbound management system according to the embodiment.

[0013] Figure 2 is an external view of the refrigerator and the inbound and outbound management device included in the inbound and outbound management system according to the embodiment.

[0014] Figure 3It is the first figure showing the moving image captured by the imaging unit.

[0015] Figure 4 It is the second figure showing the moving image captured by the imaging unit.

[0016] Figure 5 It is the third figure showing the moving image captured by the imaging unit.

[0017] Figure 6 It is the fourth figure showing the moving image captured by the imaging unit.

[0018] Figure 7 It is a state transition diagram for managing the four states of food inventory.

[0019] Figure 8 It is a block diagram showing the functional configuration of the information processing unit.

[0020] Figure 9 It is a flowchart of operation example 1 of the inventory management system of the embodiment.

[0021] Figure 10 It is a figure showing an example of food management information.

[0022] Figure 11 It is a timing diagram of operation example 2 of the inventory management system of the embodiment.

[0023] Figure 12 It is a figure showing an example of a display screen showing the content of food management information. Detailed Embodiment

[0024] Hereinafter, the embodiments will be specifically described with reference to the accompanying drawings. In addition, all the embodiments described below represent general or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions of the constituent elements, connection methods, steps, order of steps, etc. shown in the following embodiments are all examples, and their gist is not to limit the present disclosure. In addition, regarding the constituent elements in the following embodiments that are not described in the independent claims, they will be described as optional constituent elements.

[0025] In addition, each figure is a schematic diagram and is not necessarily strictly illustrated. Also, in each figure, the same reference numerals are given to substantially the same structures, and repeated descriptions may be omitted or simplified.

[0026] (Embodiment)

[0027] [Configuration]

[0028] First, the structure of the inventory management system of the embodiment will be described. Figure 1 It is a block diagram showing the functional structure of the inventory management system of the embodiment.

[0029] Figure 1 The shown incoming and outgoing management system 10 is a system for managing the food stored in the refrigerator 20. The incoming and outgoing management system 10 includes a refrigerator 20, an incoming and outgoing management device 30, a server device 40, and an information terminal 50.

[0030] The refrigerator 20 is an example of a storage compartment, which is installed in a user's residence or the like and refrigerates food. Figure 2 It is an external view of the refrigerator 20 (and the incoming and outgoing management device 30). As Figure 2 shown, the refrigerator 20 includes a main body 21, a door 22 that covers the opening of the main body 21, and a plurality of shelves 24 in a storage compartment 23 provided in the main body 21. When the door 22 is opened, the refrigerator 20 becomes a state where food can be placed on each of the plurality of shelves 24 (i.e., food is stored in the storage compartment 23). In Figure 2 the example of, the refrigerator 20 includes two doors 22 that cover the storage compartment 23, but the number of doors 22 that cover the storage compartment 23 can also be one.

[0031] The incoming and outgoing management device 30 is installed, for example, on the upper surface of the refrigerator 20 and performs information processing for managing the food stored in the refrigerator 20. The incoming and outgoing management device 30 is, for example, retrofitted to an existing refrigerator 20, but can also be built into the refrigerator 20. That is, the incoming and outgoing management device 30 can be a device separate from the refrigerator 20 or a device integrally manufactured with the refrigerator 20. As Figure 1 shown, the incoming and outgoing management device 30 includes a camera unit 31, an opening and closing sensor 32, an information processing unit 33, a storage unit 34, a communication unit 35, and a notification unit 36.

[0032] The camera unit 31 captures a moving image showing the situation when food enters and exits the refrigerator 20 from above the refrigerator 20. That is, the camera unit 31 captures, for example, a moving image of the refrigerator 20 (a moving image of the opening of the refrigerator 20) in a state where the door 22 is opened as viewed from above the refrigerator 20. The camera unit 31 is realized, for example, by a camera with a fish-eye lens, but can also be realized by a camera with a normal lens. In addition, the moving image includes a plurality of continuously captured images, and each of the plurality of images is also recorded as a frame.

[0033] The camera unit 31 is installed in the housing of the incoming and outgoing management device 30, but can also be installed on the refrigerator 20 separately from the housing of the incoming and outgoing management device 30. That is, the camera unit 31 can also be retrofitted to the refrigerator 20. In this case, the camera unit 31 has a magnet, a suction cup, or a clip or the like as a structure that can be retrofitted to the refrigerator 20.

[0034] The opening / closing sensor 32 measures the distance from the opening / closing sensor 32 to the door 22 at the upper part of the refrigerator 20, thereby detecting the opening / closing (open state) of the door 22. The opening / closing sensor 32 is implemented by, for example, an infrared distance measuring sensor. The opening / closing sensor 32 can separately detect the opening / closing of the right door 22 and the left door 22 by separately measuring the distance from the opening / closing sensor 32 to the right door 22 and the distance from the opening / closing sensor 32 to the left door 22.

[0035] In addition, the opening / closing sensor 32 can also be implemented by an angle sensor or the like instead of the infrared distance measuring sensor. Further, the opening / closing sensor 32 can also be provided in the refrigerator 20. In this case, the incoming / outgoing management device 30 may not be provided with the opening / closing sensor 32.

[0036] The information processing unit 33 performs information processing for managing the foods stored in the refrigerator 20. The information processing unit 33 is implemented by, for example, a microcomputer, but can also be implemented by a processor or a dedicated circuit. The information processing unit 33 has a food detection unit 37, an arm detection unit 38, and a determination unit 39 as functional components. The functions of the food detection unit 37, the arm detection unit 38, and the determination unit 39 are realized, for example, by the microcomputer or the like constituting the information processing unit 33 executing the computer program stored in the storage unit 34. Details of the functions of the food detection unit 37, the arm detection unit 38, and the determination unit 39 will be described later.

[0037] The storage unit 34 is a storage device that stores the computer program executed by the information processing unit 33 and various information (such as the food management information described later) required for the above information processing. The storage unit 34 is implemented by, for example, a semiconductor memory.

[0038] The communication unit 35 is a communication circuit for the incoming / outgoing management device 30 to communicate with the server device 40 via the wide area communication network 60. The communication unit 35 is, for example, a wireless communication circuit that performs wireless communication, but can also be a wired communication circuit that performs wired communication. The communication standard of the communication performed by the communication unit 35 is not particularly limited.

[0039] The notification unit 36 performs notifications such as when a food is identified as an object of inventory management and when an error related to the imaging unit 31 occurs. The notification unit 36 is implemented by, for example, a speaker or a buzzer, and performs notifications by outputting sounds, but can also be implemented by a light emitting element such as an LED (Light Emitting Diode) and performs notifications by emitting light.

[0040] The server device 40 is a computer located outside the facility where the refrigerator 20 is installed. Specifically, it is a cloud server. The server device 40 performs information processing for presenting information related to the food stored in the refrigerator 20. The server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0041] The communication unit 41 is a communication circuit for the server device 40 to communicate with the warehousing and outbound management device 30 and the information terminal 50 via a wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but it can also be a wireless communication circuit that performs wireless communication. The communication standard of the communication performed by the communication unit 41 is not particularly limited.

[0042] The information processing unit 42 performs information processing for presenting information related to the food stored in the refrigerator 20. The information processing unit 42 is implemented by, for example, a microcomputer, but it can also be implemented by a processor or a dedicated circuit. The information processing unit 42 has a presentation unit 44 as a functional component. The function of the presentation unit 44 is realized, for example, by a microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the storage unit 43. Details of the function of the presentation unit 44 will be described later.

[0043] The storage unit 43 is a storage device that stores the computer program executed by the information processing unit 42 and various information required for the above information processing. The storage unit 43 is implemented by, for example, a semiconductor memory.

[0044] The information terminal 50 is an information terminal possessed by the user. The user uses the information terminal 50 to confirm the food (the content of the food management information) stored in the refrigerator 20 from a destination outside the home or the like. The information terminal 50 is, for example, a portable information terminal such as a smart phone or a tablet terminal, but it can also be a fixed information terminal such as a personal computer.

[0045] [Determination of Four States]

[0046] Next, the management operation of the inventory of the food in the refrigerator 20 performed by the warehousing and outbound management system 10 will be described. When the user warehouses food in the refrigerator 20 or takes food out of the refrigerator 20 and wants to manage the food as an object of inventory management, the user temporarily stops the food above a specific area provided on the bottom surface of the storage room 23. That is, the user intentionally performs an action of lifting the food above the specific area. The warehousing and outbound management device 30 can detect the situation where the user stops the food above the specific area by performing image processing on the moving image captured by the imaging unit 31. Figure 3 It is a diagram showing an example of the moving image captured by the imaging unit 31.

[0047] Figure 3This is a diagram showing the entire shooting range of the imaging unit 31. In Figure 3 the shown moving image, the opening of the refrigerator 20 and the storage chamber 23 located behind the opening are reflected. In addition, a specific area P is also reflected in the moving image. The specific area P is an area on the bottom surface of the storage chamber 23 (the inner surface of the refrigerator 20) where a specified pattern is depicted, and it is an actual existing area. In Figure 3 no user (person) is reflected, but as shown in Figures 4 - 6 etc., a user located in front of the opening is also reflected in the moving image.

[0048] In addition, in the description of the image processing of the moving image described later, instead of the entire shooting range of the imaging unit 31, a specified range A0 equivalent to a part of the shooting range is taken as the object of image processing. That is, in the description of the image processing described later, it refers to the part of the specified range A0 instead of the entire shooting range of the imaging unit 31, and is described as a frame, an image, a moving image, etc.

[0049] A first area A1 and a second area A2 are set in the specified range A0. The first area A1 and the second area A2 are virtual areas used in image processing and are shown by dashed lines in Figure 3 The first area A1 is set to include the specific area P. The first area A1 and the specific area P are set as places suitable for the user to lift an item to be used as an object for inventory management.

[0050] The shape and configuration of the specific area P and the first area A1 are an example, and the shape and configuration of the specific area and the first area A1 can be appropriately determined by the designer of the warehousing and outbound management system 10, etc. according to experience or experiments. In addition, instead of setting a clear specific area P, the first area A1 can be set in at least a part of the bottom surface of the storage chamber 23 (a surface of a single color such as white).

[0051] When the user puts food into storage, it is considered that the moving image captured by the imaging unit 31 changes as Figures 3 - 6 shown.

[0052] Figure 3 This is a diagram showing the moving image before the user puts food into storage, and this state is defined as the standby state (STANBY). In the standby state (STANBY), the user's hand or arm, or the food held by the user (hereinafter, referred to as the user's arm, etc.) is not reflected in the specified range A0.

[0053] Figure 4This is a diagram showing a moving image when the user moves the food above a specific area, and this state is defined as the READY state. The READY state is when the user's arm or the like is reflected in the first area A1 of the moving image and the user's arm or the like is in a moving state.

[0054] Figure 5 This is a diagram showing a moving image when the user holds the food stationary above a specific area, and this state is defined as the SET state. The SET state is when the user's arm or the like (in this case, the food) is reflected in the first area A1 of the moving image and the user's arm or the like is stationary (more precisely, with little movement).

[0055] Figure 6 This is a diagram showing a moving image when the user stores the food in the storage room 23 (places it on the shelf 24), and this state is defined as the INOUT state. The INOUT state is when the user's arm or the like is reflected in the first area A1 of the moving image and the user's arm or the like is in a moving state. After the INOUT state, when the placement of the food by the user is completed, it returns to the STANBY state. If the above-described STANBY state, READY state, SET state, and INOUT state are represented in the form of a state transition diagram, it is as Figure 7 shown. Figure 7 This is a state transition diagram of the STANBY state, READY state, SET state, and INOUT state.

[0056] In addition, although an example of the moving image is not shown, when the user takes out the food, it also migrates in the order of the STANBY state, READY state, SET state, INOUT state, and STANBY state in the same way as when storing the food.

[0057] [Image processing for determining 4 states]

[0058] The storage and retrieval management device 30 can manage the food by determining the above 4 states based on the difference in pixel values between frames included in the moving image (presence or absence of movement) and the difference in pixel values between the first reference image and the frames included in the moving image (whether the user's arm or the like is reflected). Hereinafter, with reference to Figure 8 the method for determining the 4 states will be described. Figure 8 This is a block diagram showing the functional structure of the information processing unit 33 provided in the storage and retrieval management device 30. Also shown in Figure 8 are the imaging unit 31 and the opening / closing sensor 32.

[0059] In addition, in the following Figure 8 description (description of image processing for determining four states), it refers to a part of the specified range A0 rather than the entire shooting range of the imaging unit 31, and is described as a frame, an image, a moving image, etc. In image processing, the image is temporarily stored in the storage unit 34, but the description of this point is omitted below.

[0060] As Figure 8 shown, the information processing unit 33 includes a food detection unit 37 and a determination unit 39. The food detection unit 37 includes an inter-frame difference calculation unit 37a, a reference image acquisition unit 37b, and a reference image difference calculation unit 37c.

[0061] First, an explanation will be given of whether the object reflected in the moving image is moving as determined by the inter-frame difference calculation unit 37a. The inter-frame difference calculation unit 37a calculates the difference between the pixel values of the pixels included in the frame at time t - 1 captured by the imaging unit 31 and the pixel values of the pixels included in the frame at the current time t. The inter-frame difference calculation unit 37a, for example, calculates the difference in pixel values of the corresponding pixels between the frame at time t - 1 and the frame at the current time t. When the number of pixels included in the frame is m (m is a natural number), m such differences are calculated. The inter-frame difference calculation unit 37a sets the number of differences among the m differences that are greater than a specified threshold as n (n is a natural number). When n / m is equal to or greater than a specified ratio, it can be determined that the object (such as the user's arm) reflected in the moving image is moving. When n / m is less than the specified ratio, it can be determined that the object (such as the user's arm) reflected in the moving image is not moving. The specified threshold and the specified ratio can be appropriately determined based on experience or experiments.

[0062] Next, an explanation will be given of whether an object such as the user's arm is reflected in the first area A1 of the moving image as determined by the reference image acquisition unit 37b and the reference image difference calculation unit 37c. The reference image acquisition unit 37b acquires a first reference image captured by the imaging unit 31. The first reference image is an image in which no object other than the refrigerator 20 is reflected in the first area A1 as Figure 3 shown.

[0063] The reference image acquisition unit 37b causes the imaging unit 31 to capture a first reference image and acquire the first reference image, for example, when at least one of the two doors 22 is detected as being opened by the opening / closing sensor 32. That is, the reference image acquisition unit 37b acquires a new first reference image each time the door 22 is opened. Thereby, the latest first reference image reflecting the inside of the refrigerator 20 can be acquired. When storing food in the refrigerator 20, since there is a rule not to place food above the specific area P, for the image captured immediately after the door 22 is opened, no object outside the refrigerator 20 is reflected in the first area A1 including the specific area P, and it can be used as the first reference image.

[0064] In addition, it is not essential to acquire the first reference image triggered by the opening of the door 22. For example, the first reference image may be captured by the imaging unit 31 when newly introduced into the warehousing and outbound management device 30 (at the time of initial setting), and the first reference image may be continuously used thereafter. In addition, the first reference image may be captured at an arbitrary timing. For example, the first reference image acquired triggered by the opening of the door 22 may be periodically updated during the period when the door 22 of the refrigerator is opened.

[0065] When the first reference image is captured at an arbitrary timing, the reference image acquisition unit 37b acquires (uses) it as the first reference image on the basis of confirming that no object outside the refrigerator 20 is reflected in the first area A1 of the captured image. For example, when the specific area P is an area in which a prescribed pattern is drawn, the reference image acquisition unit 37b can determine that no object outside the refrigerator 20 is reflected in the first area A1 when the prescribed pattern is detected in the first area A1 by pattern matching. In addition, when the first area A1 is set in a part of the bottom surface (a surface of a single color such as white) of the storage chamber 23 without setting a clear specific area P, the reference image acquisition unit 37b can determine that no object outside the refrigerator 20 is reflected in the first area A1 when no edge is detected in the first area A1 by edge detection.

[0066] The reference image difference calculation unit 37c calculates the difference between the pixel values of the pixels included in the first region A1 of the first reference image acquired by the reference image acquisition unit 37b and the pixel values of the pixels included in the first region A1 of the frame at the current time t. For example, the reference image difference calculation unit 37c calculates the difference between the pixel values of the corresponding pixels between the first region A1 of the first reference image and the first region A1 of the frame at the current time t. When the number of pixels included in the first region A1 is p (p is a natural number), p such differences are calculated. The reference image difference calculation unit 37c sets the number of differences greater than a specified threshold among the p differences as q (q is a natural number). When q / p is equal to or greater than a specified ratio, it can be determined that there is a user's arm or the like in the first region A1. When q / p is less than the specified ratio, it can be determined that there is no user's arm or the like. The specified threshold and the specified ratio can be appropriately determined based on experience or experiments.

[0067] Next, the determination of the four states of the determination unit 39 will be described. The determination unit 39 determines which of the four states the current state is based on the determination result of the inter-frame difference calculation unit 37a (the presence or absence of an object shown in the moving image), the determination result of the reference image difference calculation unit 37c (the presence or absence of a user's arm or the like in the first region A1), and the state transition diagram ( Figure 7 ).

[0068] Immediately after the door 22 is opened, no user's arm or the like is shown in the moving image (the specified range A0). Therefore, the object shown in the moving image does not move, and the user's arm or the like is not located in the first region A1. Therefore, the determination unit 39 determines that it is in the standby state (STANBY). After the standby state (STANBY), if the user puts an arm or the like into the refrigerator 20, the user's arm or the like shown in the moving image moves, and the user's arm or the like is located in the first region A1. Therefore, the determination unit 39 determines that it has migrated to the ready state (READY).

[0069] After that, when the user wants to make the food to be managed stationary above the specific region P, the movement of the user's arm or the like shown in the moving image disappears (becomes less), and the user's arm or the like (in this case, the food) is located in the first region A1. Therefore, the determination unit 39 determines that it has migrated to the set state (SET).

[0070] When the determination unit 39 determines that the set state (SET) has continued for a certain period, it determines that it is the in-out state (INOUT). The in-out state (INOUT) is a state in which the user performs an action of storing or taking out food, and is a state in which the movement of the user's arm or the like is reflected in the dynamic image and the user's arm or the like is located in the first area A1. On the other hand, when the determination unit 39 determines that the set state (SET) has not continued for a certain period (the duration of the set state (SET) has not reached a certain period), it determines that it returns to the ready state (READY).

[0071] After the in-out state (INOUT), if the user pulls the hand out of the refrigerator 20 to the outside of the refrigerator 20, since the user's arm or the like is not reflected in the dynamic image (the specified range A0), the object reflected in the dynamic image does not move, and the user's arm or the like is not located in the first area A1. Therefore, the determination unit 39 determines that it is the standby state (STANBY).

[0072] As described above, the information processing unit 33 includes a food detection unit 37 that detects whether a stationary object other than the refrigerator 20 is reflected in the first area A1 of the dynamic image by performing image processing on the dynamic image captured by the imaging unit 31. Thus, the information processing unit 33 (the determination unit 39) can determine the stationary object as an object to be managed for inventory when it is detected that a stationary object other than the refrigerator 20 is reflected in the first area A1. More specifically, when the states have migrated in the order of the ready state (READY), the set state (SET), the in-out state (INOUT), and the standby state (STANBY) after the standby state (STANBY), the information processing unit 33 can determine that the food to be managed for inventory has been stored (or taken out).

[0073] [Structure for suppressing misjudgment]

[0074] However, when the user stores food that is not an object to be managed for inventory, since the user does not keep the food stationary above the specific area P, it is not migrated from the ready state (READY) to the set state (SET) and is determined to be the ready state (READY), as Figure 6 shown, the user puts the hand inside the refrigerator 20. Here, as Figure 6 shown, when the user is stationary with the hand inside the refrigerator 20, the determination unit 39 may misjudge that it has migrated from the ready state (READY) to the set state (SET).

[0075] Therefore, the information processing unit 33 sets the second area A2 in the moving image (prescribed range A0). When a user's arm or the like is reflected in the second area A2, even if the movement of the user's arm or the like reflected in the moving image disappears and the user's arm or the like is located in the first area A1, it is not determined as the set state (SET). In other words, the determination unit 39 determines whether to migrate to the set state (SET) by taking, in addition to the necessary condition that the movement of the user's arm or the like reflected in the moving image disappears and the user's arm or the like is located in the first area A1, the condition that the user's arm or the like is not reflected in the second area A2 as a necessary condition.

[0076] As Figures 3 - 6 shown, in the depth direction of the refrigerator 20 ( Figures 3 - 6 in the up-down direction), the second area A2 is located at a position deeper than the first area A1. The length of the second area A2 in the depth direction of the refrigerator 20 is shorter than the length of the first area A1 in the depth direction, and the length of the second area A2 in the width direction of the refrigerator 20 ( Figures 3 - 6 in the left-right direction) is longer than the length of the first area A1 in the width direction. The second area A2 is set as a place where it is considered that the user's hand or arm is not reflected when the user holds an item to be the object of inventory management and places the item in the first area A1.

[0077] In addition, such a second area A2 is an example, and the shape and configuration of the second area A2 may be appropriately determined by the designer or the like of the warehousing and inventory management system 10 through experience or experiments.

[0078] As described above Figure 8 shown, the information processing unit 33 includes an arm detection unit 38 as a component for determining whether the user's arm or the like is not reflected in the second area A2. The arm detection unit 38 includes a reference image acquisition unit 38a and a reference image difference calculation unit 38b. Hereinafter, it will be described how the reference image acquisition unit 38a and the reference image difference calculation unit 38b determine whether an object such as the user's arm is reflected in the second area A2 of the moving image. The reference image acquisition unit 38a acquires a second reference image captured by the imaging unit 31. The second reference image is an image in which no object other than the refrigerator 20 is reflected in the second area A2 as Figure 3 such.

[0079] The reference image acquisition unit 38a causes the imaging unit 31 to capture the second reference image and acquire the second reference image, for example, when at least one of the two doors 22 is detected to be opened by the opening / closing sensor 32. That is, the reference image acquisition unit 38a acquires a new second reference image every time the door 22 is opened. Thereby, the latest second reference image reflecting the inside of the refrigerator 20 can be acquired.

[0080] In addition, the second reference image does not necessarily need to be obtained triggered by the opening of door 22. For example, the second reference image can also be captured by the imaging unit 31 when newly imported into the warehousing and inventory management device 30 (during initial setting), and the reference image can be continuously used thereafter. In addition, the second reference image can also be captured at an arbitrary timing. For example, the reference image obtained triggered by the opening of door 22 can also be periodically updated during the period when the door 22 of the refrigerator is open. Additionally, the first reference image can also be used as the second reference image. In this case, the acquisition of the second reference image can also be omitted.

[0081] The reference image difference calculation unit 38b calculates the difference between the pixel values of the pixels included in the second area A2 of the second reference image obtained by the reference image acquisition unit 38a and the pixel values of the pixels included in the second area A2 of the frame at the current time t. The reference image difference calculation unit 38b, for example, calculates the difference between the pixel values of the corresponding pixels between the second area A2 of the second reference image and the second area A2 of the frame at the current time t. When the number of pixels included in the second area A2 is x (x is a natural number), x such differences are calculated. The reference image difference calculation unit 38b sets the number of differences among the x differences that are greater than a specified threshold to y (y is a natural number). When y / x is equal to or greater than a specified ratio, it can be determined that there is a user's arm or the like in the second area A2. When y / x is less than the specified ratio, it can be determined that there is no user's arm or the like. The specified threshold and the specified ratio can be appropriately determined based on experience or experiments.

[0082] In this way, the arm detection unit 38 performs image processing on the moving image captured by the imaging unit 31, thereby detecting whether a hand or an arm is reflected in the second area A2 of the moving image. Thereby, the determination unit 39 can determine the stationary object other than the refrigerator 20 reflected in the first area A1 (specifically, the food stops in the first area A1) as an object to be managed for inventory when it is detected during the period when no hand or arm is reflected in the second area A2. Thereby, false determination of the transition from the ready state to the set state is suppressed.

[0083] [Operation Example 1]

[0084] Next, as a specific operation example 1 of the warehousing and inventory management system 10, the management operation of the food inventory will be described. Figure 9 It is a flowchart of the operation example 1 of the warehousing and inventory management system 10.

[0085] When the information processing unit 33 of the warehousing and inventory management device 30 determines that at least one of the two doors 22 is opened based on the detection result of the opening and closing sensor 32 (S11), it activates the imaging unit 31 and starts capturing the moving image of the imaging unit 31 (S12).

[0086] Next, the information processing unit 33 performs food detection processing (S13). In the food detection processing, the information processing unit 33 determines the warehousing or outwarehousing of food by making the above-mentioned four-state determinations.

[0087] In addition, when the user makes the food stationary above the specific area P and migrates from the set state (SET) to the in / out state (INOUT), the information processing unit 33 can also use the notification unit 36 to notify the user that the food is recognized as an object of inventory management. For example, when the notification unit 36 is implemented by a speaker, the notification unit 36 outputs a sound or a voice message indicating that the food is recognized as an object of inventory management. When the notification unit 36 is implemented by a light-emitting element, the notification unit 36 notifies that the food is recognized as an object of inventory management by emitting light.

[0088] Thereby, the user can easily grasp the successful stationary of the food (the food can be used as an object of inventory management).

[0089] In addition, the information processing unit 33 temporarily stores, for example, the frame when migrating from the set state (SET) to the in / out state (INOUT) in the storage unit 34.

[0090] After that, when the information processing unit 33 determines that both of the two doors 22 are closed based on the detection result of the opening / closing sensor 32 (S14), it performs food management information update processing (S15). Figure 10 This is a diagram showing an example of food management information. In the food management information, the ID of the food is stored in association with the image of the food. The food management information is information obtained by listing the foods located in the refrigerator 20 at the time of the above-mentioned step S11. In other words, it is inventory list information.

[0091] In step S15, the information processing unit 33 cuts out the part showing the food (the part near the first area A1) from the frame temporarily stored in step S13 as the image of the food. When a new food is placed in steps S11 to S14, the information processing unit 33 assigns an ID to the cut-out image of the food and adds it to the food management information. On the other hand, when the food is outwarehoused in steps S11 to S14, the information processing unit 33 deletes the image of the food that is most similar to the cut-out image of the food from the one or more food images included in the food management information. In this case, the information processing unit 33 performs processing to determine the similarity of the images using various existing algorithms.

[0092] In addition, for example, in the in-out state (INOUT), it is determined whether the food has been put into the refrigerator or taken out based on the presence or absence of a user's arm, etc. in the third area (not shown in the figure. For example, the area near the shelf 24) within the moving image. The determination unit 39, for example, determines that the food has been put into the refrigerator when there is a user's arm, etc. in the third area, and determines that the food has been taken out when there is no user's arm, etc. in the third area. For the determination of the presence or absence of a user's arm, etc. in the third area, the method using the above-mentioned reference image can be cited.

[0093] As described above, the in-out management system 10 can manage the inventory of food in the refrigerator 20.

[0094] In addition, when an error related to the imaging unit 31 occurs, such as when the imaging unit 31 cannot capture a moving image, the information processing unit 33 can also use the notification unit 36 to notify that an error related to the imaging unit 31 has occurred. For example, when the notification unit 36 is implemented by a speaker, the notification unit 36 outputs a voice message indicating that an error has occurred. When the notification unit 36 is implemented by a light-emitting element, the notification unit 36 notifies that an error has occurred by emitting light.

[0095] In this way, if the occurrence of an error related to the imaging unit 31 is notified, the user can take measures against the error.

[0096] [Operation Example 2]

[0097] Next, as Operation Example 2 of the in-out management system 10, the prompting operation of the food stored in the refrigerator 20 will be described. Figure 11 It is a timing chart of Operation Example 2 of the in-out management system 10.

[0098] The communication unit 35 of the in-out management device 30 sends the food management information to the server device 40 (S21). The communication unit 41 of the server device 40 receives the food management information, and the prompting unit 44 stores the received food management information in the storage unit 43 (S22). The communication unit 35 of the in-out management device 30, for example, sends the food management information to the server device 40 each time the food management information is updated. Thus, the latest food management information is stored in the storage unit 43 of the server device 40. In addition, the food management information can also be sent from the in-out management device 30 to the server device 40 once at regular intervals, etc.

[0099] On the other hand, when the user performs a specified operation on the information terminal 50 from the destination of going out, etc. to confirm the food (the content of the food management information) stored in the refrigerator 20, the information terminal 50 accepts such a specified operation (S23), and based on the accepted specified operation, sends a prompting request to the server device 40 (S24).

[0100] The communication unit 41 of the server device 40 receives a prompt request. Based on the received prompt request, the prompting unit 44 sends a prompt message for prompting the content of the food management information stored in the storage unit 43 to the information terminal 50 (S25). More specifically, the communication unit 41 sends the prompt message.

[0101] The information terminal 50 receives the prompt message and, based on the received prompt message, displays a display screen representing the content of the food management information on the display unit provided in the information terminal 50 (S26). Figure 12 It is a diagram showing an example of a display screen representing the content of food management information. In Figure 12 In the display screen, an image of the food included in the food management information (an image cut out from the moving image captured by the imaging unit 31) is superimposed on a background image simulating the contents of the refrigerator 20 and displayed.

[0102] In this way, the user of the inventory management system 10 can confirm the food stored in the refrigerator 20 of the house from the destination of going out or the like.

[0103] [Modification example]

[0104] In the above-described embodiment, a structure for suppressing misjudgment of the object of inventory management has been described. The inventory management system 10 may also be provided with a structure for excluding the food determined to be the object of inventory management from the objects.

[0105] As described above, when the food is determined to be an object, this is notified by the notification unit 36. At this time, the inventory management system 10 may also exclude the food determined to be the object of inventory management from the objects on the condition that the user has made a specified gesture (such as making the hand into the shape of a cloth and keeping it still) in the first area A1 within a specified period from the notification.

[0106] [Other modification examples]

[0107] In the above-described embodiment, the refrigerator 20 may be a refrigerator for general household use, a refrigerator for displaying products used in a retail store such as a convenience store, or other business-use refrigerators.

[0108] In addition, the refrigerator 20 is an example of a storage compartment, and the present disclosure can also be implemented as other storage compartments without a refrigeration function. In addition, the items (in other words, objects) stored in the storage compartment are not limited to food, and may be other items. The case of storing items other than food in the refrigerator 20 is also considered.

[0109] [Effects, etc.]

[0110] Hereinafter, the technology obtained from the disclosure of this specification will be exemplified, and the effects obtained from this technology will be described.

[0111] Technology 1 is an inbound and outbound management system 10, which includes: a camera unit 31 that captures a moving image when observing the opening of the storage bin from above; a food detection unit 37 that detects whether a stationary object other than the storage bin is reflected in a first area A1 of the moving image by performing image processing on the moving image; an arm detection unit 38 that detects whether a user's hand or arm located in front of the opening is reflected in a second area A2 of the moving image by performing image processing on the moving image; and a determination unit 39 that determines the stationary object as an object for inventory management on the condition that a stationary object other than the storage bin is detected in the first area A1 during a period when the user's hand or arm is not reflected in the second area A2. The food detection unit 37 is an example of a first detection unit, and the arm detection unit 38 is an example of a second detection unit.

[0112] Such an inbound and outbound management system 10 can determine an item stationary in the first area A1 by the user as an object for inventory management. That is, the inbound and outbound management system 10 can determine an item as an object for inventory management based on the user's intentional action. In addition, the second area A2 is set, for example, as a place where it is considered that the user's hand or arm is not reflected when the user intentionally places an item to be an object for inventory management in the first area A1. By setting the second area A2 in this way, the inbound and outbound management system 10 can suppress misjudgment of the object for inventory management.

[0113] Technology 2 is in the inbound and outbound management system 10 of Technology 1. The arm detection unit 38 obtains a second reference image pre-captured by the camera unit 31, and detects whether a hand or arm is reflected in the second area A2 based on the difference between the pixel values of the pixels in the second area A2 in the obtained second reference image and the pixel values of the pixels in the second area A2 in the image included in the moving image.

[0114] Such an inbound and outbound management system 10 can detect whether a hand or arm is reflected in the second area A2 by comparing the second reference image with the current image (frame).

[0115] Technology 3 is in the inbound and outbound management system 10 of Technology 2. The arm detection unit 38 can change the second reference image.

[0116] Such an inbound and outbound management system 10 can change (update) the second reference image.

[0117] In Technique 4, in the warehousing and inventory management system 10 of Technique 3, a door 22 is provided at the opening. The arm detection unit 38 uses the image captured by the imaging unit 31 when the door 22 is opened as the second reference image, and updates the second reference image each time the door 22 is opened.

[0118] Such a warehousing and inventory management system 10 can update the second reference image each time the door 22 is opened.

[0119] In Technique 5, in the warehousing and inventory management system 10 of Techniques 1 to 4, the food detection unit 37 obtains the first reference image pre-captured by the imaging unit 31, and based on the difference between the pixel values of the pixels in the first region A1 in the obtained first reference image and the pixel values of the pixels in the first region A1 in the images included in the moving image, as well as the difference between the pixel values among the multiple images included in the moving image, detects whether there is a stationary object other than the storage bin in the first region A1.

[0120] Such a warehousing and inventory management system 10 can detect the presence or absence of a stationary object by comparing the first reference image with the current image (frame) and comparing the multiple images (frames) included in the moving image.

[0121] In Technique 6, in the warehousing and inventory management system 10 of Technique 5, the food detection unit 37 can change the first reference image.

[0122] Such a warehousing and inventory management system 10 can change (update) the first reference image.

[0123] In Technique 7, in the warehousing and inventory management system 10 of Technique 6, a door 22 is provided at the opening. The food detection unit 37 uses the image captured by the imaging unit 31 when the door 22 is opened as the first reference image, and updates the first reference image each time the door 22 is opened.

[0124] Such a warehousing and inventory management system 10 can update the first reference image each time the door 22 is opened.

[0125] In Technique 8, in any of the warehousing and inventory management systems 10 of Techniques 1 to 7, the first region A1 is a region including the inner surface of the storage bin reflected in the moving image.

[0126] Such a warehousing and inventory management system 10 can use the region including the inner surface of the storage bin reflected in the moving image as the first region.

[0127] In Technique 9, in any of the warehousing and inventory management systems 10 of Techniques 1 to 8, a specific region P with a specified pattern depicted is provided on the inner surface of the storage bin reflected in the moving image, and the first region A1 includes the specific region P.

[0128] Such a specific area P can be used as a marker for the position where an item that a user wants to be an object of inventory management should be stationary.

[0129] In the warehousing and shipping management system 10 of any one of Technologies 1 to 9, in the depth direction of the storage bin, the second area A2 is located on the inner side relative to the first area A1.

[0130] If the second area A2 is set in this way, it is considered that when a user holds an item that is to be an object of inventory management and places the item in the first area A1, the user's hand or arm will not be reflected in the second area A2. Therefore, the warehousing and shipping management system 10 can suppress misjudgment of the object of inventory management.

[0131] In the warehousing and shipping management system 10 of any one of Technologies 1 to 10, the length of the second area A2 in the depth direction of the storage bin is shorter than the length of the first area A1 in the depth direction.

[0132] Such a warehousing and shipping management system 10 can suppress misjudgment of the object of inventory management based on the second area A2 whose length in the depth direction is shorter than that of the first area A1.

[0133] In the warehousing and shipping management system 10 of any one of Technologies 1 to 11, the length of the second area A2 in the width direction of the storage bin is longer than the length of the first area A1 in the width direction.

[0134] Such a warehousing and shipping management system 10 can suppress misjudgment of the object of inventory management based on the second area A2 whose length in the width direction is longer than that of the first area A1.

[0135] In the warehousing and shipping management system 10 of any one of Technologies 1 to 12, it further includes a notification unit 36 that notifies an error related to the imaging unit 31.

[0136] Such a warehousing and shipping management system 10 can notify an error related to the imaging unit 31.

[0137] In the warehousing and shipping management system 10 of any one of Technologies 1 to 13, the storage bin is a refrigerator 20.

[0138] The warehousing and shipping management system 10 can determine whether an item is an object of inventory management of the refrigerator 20.

[0139] Technique 15 is an inbound and outbound management method executed by a computer. The inbound and outbound management method includes: a photographing step of photographing a dynamic image when observing the opening of the storage bin from above; a first detection step of detecting whether a stationary object other than the storage bin is reflected in a first area A1 of the dynamic image by performing image processing on the dynamic image; a second detection step of detecting whether a user's hand or arm located in front of the opening is reflected in a second area A2 of the dynamic image by performing image processing on the dynamic image; and a determination step of determining the stationary object as an object for inventory management on the condition that a stationary object other than the storage bin is detected in the first area A1 during a period when the user's hand or arm is not reflected in the second area A2.

[0140] Such an inbound and outbound management method can determine an item as an object for inventory management based on the user's intentional actions.

[0141] Technique 16 is a program for causing a computer to execute the inbound and outbound management method described in Technique 15.

[0142] Such a program can determine an item as an object for inventory management based on the user's intentional actions.

[0143] (Other Embodiments)

[0144] The above describes the embodiments, but the present disclosure is not limited to the above embodiments.

[0145] For example, in the above embodiment, the inbound and outbound management system is implemented by multiple devices, but it can also be implemented by a single device. For example, the inbound and outbound management system can be implemented as a single device equivalent to the inbound and outbound management device in the above embodiment, or as a single device equivalent to the server device. When the inbound and outbound management system is implemented by multiple devices, the components (especially functional components) included in the inbound and outbound management system can be arbitrarily distributed among the multiple devices. For example, a part or all of the processing described as being executed by the inbound and outbound management device in the above embodiment can also be executed by the server device.

[0146] In addition, for example, there is no particular limitation on the communication method between the devices in the above embodiment. In addition, a relay device (not shown) can be interposed during the communication between the devices. In addition, the information transfer path described in the above embodiment is not limited to the transfer path shown in the timing diagram.

[0147] For example, in the above embodiment, the processing executed by a specific processing unit can also be executed by other processing units. In addition, the order of multiple processes can be changed, and multiple processes can also be executed in parallel.

[0148] In addition, in the above-described embodiments, each component may also be implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor that reads and executes a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0149] In addition, each component may also be implemented by hardware. For example, each component may be a circuit (or an integrated circuit). These circuits may be configured as a single circuit as a whole, or may be separate and different circuits. In addition, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0150] In addition, the overall or specific aspects of the present disclosure may also be implemented by a system, a device, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. In addition, the overall or specific aspects of the present disclosure may also be implemented by any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium.

[0151] For example, the present disclosure may also be implemented as a method for managing the entry and exit of objects executed by a computer, or may be implemented as a program for causing a computer to execute the method for managing the entry and exit. The present disclosure may also be implemented as a computer-readable non-transitory recording medium recording such a program.

[0152] In addition, modes obtained by implementing various modifications conceived by those skilled in the art for each embodiment, and modes implemented by arbitrarily combining the components and functions in each embodiment without departing from the gist of the present disclosure are all included in the present disclosure.

[0153] Industrial Applicability

[0154] The entry and exit management system of the present disclosure is useful as a device capable of managing articles in a storage.

[0155] Reference Signs

[0156] 10 Entry and exit management system; 20 Refrigerator (storage); 21 Main body; 22 Door; 23 Storage room; 24 Shelf; 30 Entry and exit management device; 31 Imaging unit; 32 Opening / closing sensor; 33, 42 Information processing unit; 34, 43 Storage unit; 35, 41 Communication unit; 36 Notification unit; 37 Food detection unit (first detection unit); 38 Arm detection unit (second detection unit); 39 Determination unit; 40 Server device; 44 Prompt unit; 50 Information terminal; 60 Wide area communication network.

Claims

1. An in-out warehouse management system, comprising: a camera unit that captures a moving image when observing the opening of a storage warehouse from above; a first detection unit that detects whether a stationary object other than the storage warehouse is reflected in a first region of the moving image by performing image processing on the moving image; a second detection unit that detects whether a user's hand or arm located in front of the opening is reflected in a second region of the moving image by performing image processing on the moving image; and a determination unit that determines the stationary object as an object to be managed for inventory when it is detected that a stationary object other than the storage warehouse is reflected in the first region during a period when the user's hand or arm is not reflected in the second region.

2. The in-out warehouse management system according to claim 1, wherein the second detection unit acquires a second reference image previously captured by the camera unit, and detects whether a hand or arm is reflected in the second region based on the difference between the pixel values of the pixels in the second region of the acquired second reference image and the pixel values of the pixels in the second region of the image included in the moving image.

3. The in-out warehouse management system according to claim 2, wherein the second detection unit can change the second reference image.

4. The in-out warehouse management system according to claim 3, wherein a door is provided at the opening, and the second detection unit uses the image captured by the camera unit when the door is opened as the second reference image and updates the second reference image each time the door is opened.

5. The in-out warehouse management system according to claim 1, wherein the first detection unit acquires a first reference image previously captured by the camera unit, and detects the presence or absence of a stationary object other than the storage warehouse in the first region based on the difference between the pixel values of the pixels in the first region of the acquired first reference image and the pixel values of the pixels in the first region of the image included in the moving image, and the difference between the pixel values among the multiple images included in the moving image.

6. The in-out warehouse management system according to claim 5, wherein the first detection unit can change the first reference image.

7. The in-out warehouse management system according to claim 6, wherein a door is provided at the opening, and the first detection unit uses the image captured by the camera unit when the door is opened as the first reference image and updates the first reference image each time the door is opened.

8. The in-out warehouse management system according to any one of claims 1 to 7, wherein the first region is a region including the inner surface of the storage warehouse reflected in the moving image.

9. The in-out warehouse management system according to claim 8, wherein a specific region depicting a specified pattern is provided on the inner surface of the storage warehouse reflected in the moving image, and the first region includes the specific region.

10. The in-out warehouse management system according to any one of claims 1 to 7, wherein in the depth direction of the storage warehouse, the second region is located at a position closer to the inside than the first region.

11. The in-out warehouse management system according to any one of claims 1 to 7, The length of the storage in the depth direction in the second area is shorter than the length in the depth direction in the first area.

12. The warehousing and outwarehousing management system according to any one of claims 1 to 7, The length of the storage in the width direction in the second area is longer than the length in the width direction in the first area.

13. The warehousing and outwarehousing management system according to any one of claims 1 to 7, further includes a notification unit that notifies when an error related to the imaging unit occurs.

14. The warehousing and outwarehousing management system according to any one of claims 1 to 7, The storage is a refrigerator.

15. A warehousing and outwarehousing management method executed by a computer, The warehousing and outwarehousing management method includes: a shooting step of shooting a moving image when observing the opening of the storage from above; a first detection step of detecting whether a stationary object other than the storage is reflected in a first area of the moving image by performing image processing on the moving image; a second detection step of detecting whether a user's hand or arm located in front of the opening is reflected in a second area of the moving image by performing image processing on the moving image; and a determination step of determining the stationary object as an object for inventory management on the condition that the stationary object other than the storage is detected in the first area during the period when the user's hand or arm is not reflected in the second area.

16. A program for causing a computer to execute the warehousing and outwarehousing management method according to claim 15.

Images