Detection device, detection method, and detection system

Abstract

To detect a shelf inside a warehouse with a simple configuration.SOLUTION: A detection device comprises a communication interface connected with a sensor for acquiring an image including depth information corresponding to a distance from a reference position to a shelf, and a processor for identifying a gravity center position of the shelf by detecting a shelf frame indicating an outer shape of the shelf from the image obtained from the sensor. The processor detects a shelf when the gravity center position is present inside a first area of the image; and detects a second shelf after detecting a first shelf, when a gravity center position of the second shelf adjacent to the first shelf is detected in the first area according to movement of the sensor.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a detection device, a detection method, and a detection system.

Background Art

[0002] Patent Document 1 discloses a measurement system configured by a computer. This measurement system includes a processor that executes a program and a storage device that stores this program. The storage device holds shelf shape data, shelf area data representing the area occupied by the shelf, and luggage area data representing the area in the shelf where luggage can be stored. The processor receives the input of the shape data inside the warehouse measured by the measurement sensor, collates the shelf shape data with the shape data inside the warehouse, and specifies the position of the shelf inside the warehouse. The processor extracts the shelf shape data from the shape data inside the warehouse based on the specified position of the shelf and the shelf area data. The processor extracts the shape data of the luggage area inside the shelf from the shelf shape data based on the luggage area data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By grasping the degree to which the goods stored on the shelves in the warehouse occupy the shelves, it is expected that the utilization efficiency of the warehouse can be improved. In order to grasp the degree to which the goods occupy the shelves in the warehouse, it is required to accurately detect the shelves in the warehouse. In the configuration of Patent Document 1, it is premised on using a measurement sensor to create shape data of the entire warehouse, and further performing character recognition of the characters pre-written on the shelves or reading barcodes pre-attached to the shelves in order to grasp which goods are stored at which positions in the warehouse. Therefore, there is room for improvement in easily detecting the shelves at specific positions in the warehouse.

[0005] The present disclosure has been devised in view of the above-described conventional situation, and aims to detect the shelves in the warehouse with a simple configuration.

Means for Solving the Problem

[0006] The present disclosure includes a communication interface connected to a sensor that acquires an image including depth information corresponding to the distance from a reference position to a shelf, and a processor that detects a shelf frame indicating the outer shape of the shelf from the image from the sensor and specifies the center-of-gravity position of the shelf. When the center-of-gravity position exists within a first region of the image, the processor detects the shelf. After detecting a first shelf, when the center-of-gravity position of a second shelf adjacent to the first shelf in the first region is detected in response to the movement of the sensor, the processor detects the second shelf. The present disclosure provides a detection device.

[0007] The present disclosure also provides a detection method executed by a detection device communicably connected to a sensor that acquires an image including depth information corresponding to the distance from a reference position to a shelf. The method includes detecting a shelf frame indicating the outer shape of the shelf from the image from the sensor and specifying the center-of-gravity position of the shelf, detecting the shelf when the center-of-gravity position exists within a first region of the image, and after detecting a first shelf, detecting the second shelf when the center-of-gravity position of a second shelf adjacent to the first shelf in the first region is detected in response to the movement of the sensor.

[0008] The present disclosure also provides a detection system including a communication interface connected to a sensor that acquires an image including depth information corresponding to the distance from a reference position to a shelf, a processor that detects a shelf frame indicating the outer shape of the shelf from the image from the sensor and specifies the center-of-gravity position of the shelf, a warehouse management system that stores luggage information regarding the luggage stored in the warehouse, and a display device. The processor specifies the center-of-gravity position of the shelf in each of a plurality of images including the shelf acquired by the sensor, selects, as a best-shot image, an image in which the center-of-gravity position of the shelf among the plurality of images is closest to the center of the image, calculates a filling rate indicating a ratio of the volume of the luggage stored in the shelf occupied by the volume of the shelf included in the best-shot image, acquires the luggage information from the warehouse management system via the communication interface, and causes the display device to display a shelf recommended for storing the luggage scheduled to be carried in based on the luggage information and the filling rate.

[0009] In addition, any combination of the above components, and those obtained by converting the expression of the present disclosure among a method, an apparatus, a system, a storage medium, a computer program, etc. are also effective as aspects of the present disclosure.

Effects of the Invention

[0010] According to the present disclosure, a shelf in a warehouse can be detected with a simple configuration.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0012] Hereinafter, with reference to the drawings as appropriate, embodiments specifically disclosing a detection device, a detection method, and a detection system according to the present disclosure will be described in detail. However, a more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims thereby.

[0013] (Embodiment 1) [System Configuration] First, with reference to FIG. 1, a configuration example of the detection system 1 according to Embodiment 1 will be described. FIG. 1 is a block diagram showing a configuration example of the detection system 1 according to Embodiment 1. The detection system 1 includes a detection device 10, a sensor 20, and a warehouse management system 21.

[0014] The detection device 10 is installed in the warehouse 2. The detection device 10 is a device for detecting the shelves installed in the warehouse 2. The detection device 10 is configured using a general-purpose computer device such as a personal computer or a server computer. The detection device 10 includes a processor 11, a memory 12, a communication interface 13, an input device 14, and a display device 15. Each part included in the detection device 10 is communicably connected by an internal bus (not shown).

[0015] The processor 11 is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Graphics Processing Unit (hereinafter referred to as "GPU"), a Micro Processing Unit (hereinafter referred to as "MPU"), a Digital Signal Processor (hereinafter referred to as "DSP"), or a Field Programmable Gate Array (hereinafter referred to as "FPGA"). The processor 11 realizes the functions of the detection device 10 by reading and executing various data and programs stored in the memory 12.

[0016] The memory 12 is a storage area for storing and holding various data and programs. The memory 12 is composed of, for example, a non-volatile storage area, Read Only Memory (hereinafter referred to as "ROM"), a Hard Disk Drive (hereinafter referred to as "HDD"), and a volatile storage area, Random Access Memory (hereinafter referred to as "RAM"). RAM is, for example, a work memory used during the operation of the detection device 10. ROM stores and holds in advance programs for controlling the detection device 10, for example.

[0017] The memory 12 stores a shelf number table 16, shelf size information 17, a captured image 18, and filling rate information 19.

[0018] The shelf number table 16 is a table showing the shelf rows arranged in the warehouse 2 and the shelf numbers of each shelf. In this specification, a row composed of a plurality of continuously arranged shelves may be referred to as a "shelf row", and a number for identifying a shelf may be referred to as a "shelf number". The shelf number table 16 is displayed on the display device 15, for example, in the form of an image of the layout of each shelf installed in the warehouse 2. Details of the shelf number table 16 will be described later with reference to FIG. 2.

[0019] The shelf size information 17 includes data on the area where luggage can be stored, in other words, the volume, of each shelf installed in the warehouse 2. The data included in the shelf size information 17 will be described later with reference to FIG. 8.

[0020] The photographed image 18 is an image photographed by the sensor 20. The image photographed by the sensor 20 is stored in the memory 12.

[0021] The filling rate information 19 is the filling rate measured by the processor 11. The filling rate measured by the processor 11 is stored in the memory 12. Note that the filling rate will be described later.

[0022] The communication interface 13 communicates with the warehouse management system 21 via a network (not shown) to transmit and receive data and the like. In addition, the communication interface 13 communicates with the sensor 20 arranged in the warehouse 2 to acquire the image photographed by the sensor 20. The communication interface 13 may support either a wired communication or a wireless communication method. The communication method by the communication interface 13 may be, for example, a Wide Area Network (hereinafter referred to as "WAN"), a Local Area Network (hereinafter referred to as "LAN"), a Long Term Evolution (hereinafter referred to as "LTE"), a mobile communication such as 5G, a power line communication, a short-range wireless communication such as Wi-Fi (registered trademark) and Bluetooth (registered trademark), or a combination thereof.

[0023] The input device 14 is configured to include a keyboard, a mouse, a touch panel, or other input devices. The input device 14 receives inputs such as various types of data. The input device 14 is operated by a user such as, for example, the administrator of the warehouse 2 or the administrator of the warehouse management system 21.

[0024] The display device 15 is a device that displays information, images, etc. Examples of the display device 15 include a liquid crystal display or an organic Electro Luminescence (hereinafter referred to as "EL") display.

[0025] The input device 14 and the display device 15 may be configured as an integrated device. An example of the input device 14 and the display device 15 configured as an integrated device is a touch panel display. When the input device 14 and the display device 15 are not configured as an integrated device, the display device 15 may be provided separately from the detection device 10.

[0026] The sensor 20 photographs the shelves installed in the warehouse 2. The sensor 20 is installed so as to be movable within the warehouse 2. For example, the sensor 20 moves linearly along a predetermined route within the warehouse 2. The sensor 20 is installed, for example, so as to be movable by a human hand. For example, the sensor 20 may be fixed to a trolley. Thereby, by the user moving the trolley, the sensor 20 can move within the warehouse 2. Also, for example, the sensor 20 may be fixed to a forklift. The sensor 20 may be configured integrally with the detection device 10. In this case, the detection device 10 is installed so as to be movable.

[0027] The sensor 20 generates and acquires an image including depth information corresponding to the distance from a reference position to an object. The reference position is defined, for example, within the housing of the sensor 20. Hereinafter, the image including depth information may be referred to as a depth image. Also, hereinafter, the depth information may be referred to as depth data. The depth image may be referred to as a distance image. Further, it is preferable that the image generated by the sensor 20 includes color data. The sensor 20 may be constituted by one or more sensors and cameras. Examples of the sensor 20 include a combination of a depth camera and a Red-Green-Blue color model (hereinafter referred to as "RGB") camera, or an RGB-Depth (hereinafter referred to as "RGB-D") camera, etc. The RGB camera acquires an image including color information, in other words, RGB data. Hereinafter, the image including RGB data may be referred to as an RGB image. The RGB-D camera acquires an RGB image and a depth image including depth data.

[0028] The sensor 20 is configured to have at least a lens (not shown) and an image sensor (not shown) as optical elements in order to generate an image. The lens forms an optical image of the object on the light receiving surface of the image sensor, in other words, on the imaging surface, by allowing the light reflected by the object within the angle of view of the area photographed by the sensor 20 to enter. The image sensor is a solid-state imaging device such as, for example, a Charged Coupled Device (hereinafter referred to as "CCD") or a Complementary Metal Oxide Semiconductor (hereinafter referred to as "CMOS"). The image sensor converts the optical image formed on the imaging surface via the lens into an electrical signal every predetermined time. For example, when the predetermined time is 1 / 30 of a second, the frame rate of the sensor 20 is 30 fps. Also, the sensor 20 may generate an image by performing predetermined signal processing on the electrical signal every predetermined time described above. The images generated by the sensor 20 include still images and moving images.

[0029] Hereinafter, the part where the loading and unloading of the goods on the shelf is carried out may be referred to as the "opening" of the shelf. Also, the shelf may be a rectangular parallelepiped or a cube. In the following description, it is assumed that the shelf is a rectangular parallelepiped. Hereinafter, the surface having an opening among the six surfaces of the shelf may be referred to as the "opening surface" of the shelf. When the sensor 20 photographs the shelf, it is desirable that the posture of the sensor 20 is such that the photographing surface is perpendicular to the floor surface of the warehouse 2 and the photographing surface is parallel to the opening surface.

[0030] In Embodiment 1, the image acquired by the sensor 20 is displayed on the display device 15 in real time. For example, when the user moves the sensor 20, the image displayed on the display device 15 also moves in accordance with the movement of the sensor 20.

[0031] The warehouse management system 21 stores information on the layout of the shelves installed in the warehouse 2, information on the goods stored in the warehouse 2, information on the goods to be carried into the warehouse 2, and information on the goods to be carried out of the warehouse 2, etc. The warehouse management system may be referred to as a Warehouse Management System (hereinafter referred to as "WMS").

[0032] The warehouse management system 21 may be configured using a general-purpose computer device such as a personal computer or a server computer. The warehouse management system 21 may have, for example, the same device configuration as the detection device 10. For the sake of simplicity of explanation, FIG. 1 shows that the warehouse management system 21 includes at least a processor 22 and a memory 23.

[0033] The memory 23 stores at least location data 24 and goods information 25. The location data 24 is information indicating which shelf number of the shelf is arranged at which position in the warehouse 2.

[0034] The package information 25 includes information indicating the size, type, and location in the warehouse 2 of the packages stored in the warehouse 2. The information on the location of the package in the warehouse 2 is information indicating which shelf number the package is stored on. The package information 25 further includes information on packages to be carried into the warehouse 2 in the future and packages that have already been carried into the warehouse 2 and are scheduled to be stored on any shelf.

[0035] Note that in FIG. 1, the warehouse management system 21 is shown as an example of performing data transmission and reception with one detection device 10, but the warehouse management system 21 may be capable of performing data transmission and reception with a plurality of detection devices. Also, in FIG. 1, an example where the detection device 10 is installed in the warehouse 2 is shown, but the detection device 10 may be installed outside the warehouse 2. However, it is essential that the sensor 20 be installed inside the warehouse 2.

[0036] Based on the image acquired by the sensor 20, the detection device 10 of the detection system 1 detects the shelves installed in the warehouse 2 and measures the ratio of the volume of the packages stored on the detected shelf to the volume of the shelf. Then, based on the measurement result and the package information 25 acquired from the warehouse management system 21, the detection device 10 selects the shelf recommended for storing the packages to be stored in the warehouse 2 in the future. Hereinafter, the ratio of the volume of the packages stored on a shelf to the volume of the shelf may be referred to as the "filling rate". The unit of the filling rate is, for example, a percentage. For example, when no package is stored on a shelf, the filling rate of the shelf is 0 percent. For example, when the volume of the packages stored on a shelf occupies half of the volume of the shelf, the filling rate of the shelf is 50 percent. For the sake of simplicity of explanation, it is assumed that all the shelves installed in the warehouse 2 have the same specifications. Hereinafter, the measurement of the filling rate may be referred to as the calculation of the filling rate.

[0037] [Shelf Number Table] Next, with reference to FIG. 2, the shelf number table 16 will be described. FIG. 2 is a table diagram showing an example of the shelf number table 16 according to the first embodiment.

[0038] In Warehouse 2, Shelf a1, Shelf a2, Shelf a3, Shelf a4, Shelf a5, Shelf a6, and Shelf a7 are arranged continuously, forming Shelf Row a. Also, in Warehouse 2, Shelf b1, Shelf b2, Shelf b3, Shelf b4, Shelf b5, Shelf b6, and Shelf b7 are arranged continuously, forming Shelf Row b. Further, in Warehouse 2, Shelf c1, Shelf c2, Shelf c3, Shelf c4, Shelf c5, Shelf c6, and Shelf c7 are arranged continuously, forming Shelf Row c. Additionally, in Warehouse 2, Shelf d1, Shelf d2, Shelf d3, Shelf d4, Shelf d5, Shelf d6, and Shelf d7 are arranged continuously, forming Shelf Row d. Also, Shelf Row b and Shelf Row c are arranged adjacent to each other.

[0039] The Shelf Number Table 16 has, as information, the shelf numbers of each shelf in Warehouse 2, each shelf row, and the layout of each shelf in Warehouse 2 described above. Also, the Shelf Number Table 16 may be displayed on the Display Device 15 as a floor plan like the example in FIG. 2. When the Shelf Number Table 16 is displayed on the Display Device 15 as a floor plan, for example, the floor plan may be a floor plan focusing on a part of Warehouse 2, that is, the area where the shelves of Warehouse 2 are arranged, or it may be a floor plan of the entire Warehouse 2.

[0040] Hereinafter, for simplicity of explanation, the shelf with shelf number a1 may be simply referred to as Shelf a1. The same applies to shelves with shelf numbers other than a1.

[0041] [Processing Flow] Next, with reference to FIG. 3, the overall flow of the operation of the detection system 1 will be described. FIG. 3 is a flowchart showing an operation example of the detection device 10 according to Embodiment 1.

[0042] The processor 11 of the detection device 10 receives an input of the shelf row to be measured for the filling rate by the user (St30). For example, when the user wants to measure the filling rate of each shelf from shelf a1 to shelf a5 in shelf row a, the user inputs and designates shelf row a.

[0043] The processor 11 receives an input of the measurement range by the user for the shelf row input by the user in step St30 (St31). For example, when the user wants to measure the filling rate of each shelf from shelf a1 to shelf a5 in shelf row a, the user inputs and designates from shelf a1 to shelf a5 as the measurement range.

[0044] The processor 11 receives an operation for starting the measurement of the filling rate by the user (St32). Although not shown in FIG. 3, after the user performs the operation for starting the measurement in step St32, the user moves the sensor 20 so that the sensor 20 can acquire images of each shelf included in the measurement range designated in step St31. For example, when the user wants to measure the filling rate of each shelf from shelf a1 to shelf a5 in shelf row a, the user moves the sensor 20 so that the sensor 20 can photograph each shelf from shelf a1 to shelf a5. At this time, the user moves, for example, a cart on which the sensor 20 is fixed from in front of shelf a1 to in front of shelf a5.

[0045] Specific examples of the user's operations in each of step St30, step St31, and step St32 will be described later with reference to FIG. 9.

[0046] The processor 11 acquires the image photographed by the sensor 20 (St33). The image photographed by the sensor 20 is stored in the memory 12 as the photographed image 18.

[0047] The processor 11 detects a shelf frame based on the image acquired in step St34 (St34). The "shelf frame" refers to a frame indicating the outer shape of the shelf. Details will be described later with reference to FIGS. 4, 5, and 6.

[0048] Based on the detection result of the shelf frame by the process of step St34, the processor 11 counts the number of shelves (St35). The "number of shelves" refers to the number of detected shelves. Details will be described later with reference to FIGS. 5 and 6.

[0049] The processor 11 selects the best shot image of each shelf within the measurement range from the image acquired in step St33 (St36). The best shot image will be described later with reference to FIG. 7.

[0050] The processor 11 measures the filling rate of each shelf in the measurement range using the best shot image selected in step St36 (St37). The measurement of the filling rate will be described later with reference to FIG. 8. The measured filling rate is stored in the memory 12 as filling rate information 19.

[0051] Note that the process of step St35 and the processes of steps St36 and St37 may be performed simultaneously in parallel.

[0052] The processor 11 accepts an operation for completion of measurement by the user (St38). A specific example of the user's operation in step St38 will be described later with reference to FIG. 12. Note that in step St38, the user confirms that the selection of the best shot image and the measurement of the filling rate have been completed without problems and performs an operation for completion of measurement. Although omitted from the flowchart shown in FIG. 3 for simplicity of explanation, the user may start over from the detection of the shelves if necessary before performing the operation for completion of measurement. Details will be described later with reference to FIGS. 11 and 12.

[0053] When the processor 11 accepts an operation for completion of measurement by the user in step St38, it accesses the warehouse management system 21 (St39).

[0054] The processor 11 accesses the warehouse management system 21 in step St39 and saves the best shot image selected in step St36 and the filling rate measured in step St37 in the warehouse management system 21 (St40). The processor 11 may save the best shot image and the filling rate in the warehouse management system 21 after associating the shooting date and time of the best shot image and the measurement date and time of the filling rate with the best shot image and the filling rate respectively.

[0055] The processor 11 accesses the warehouse management system 21 in step St39 and obtains the package information 25 from the warehouse management system 21 (St41). More precisely, the processor 11 identifies the position of each shelf to be measured in the warehouse 2 based on the shelf number table 16 and the result of counting the number of shelves in the process of step St35. Then, the processor 11 inquires of and obtains from the warehouse management system 21 the information of the packages stored at the positions of the respective shelves to be measured based on the positions of the respective shelves to be measured in the warehouse 2. The warehouse management system 21 can search for the information of the packages stored on the shelves at specific positions in the warehouse 2 based on the location data 24 and the package information 25, and transmit the searched package information to the detection device 10. For example, when each shelf from shelf a1 to shelf a5 is included in the measurement range of the filling rate, the processor 11 inquires of the warehouse management system 21 about the information of the packages stored at the positions of each of the shelves from shelf a1 to shelf a5. Then, the processor 11 obtains the information of the packages stored at the positions of each of the shelves from shelf a1 to shelf a5 in the warehouse 2. When obtaining the package information 25, the processor 11 obtains not only the information of the packages stored on each shelf to be measured for the filling rate, but also the information of the packages to be carried into the warehouse 2 in the future and the information of the packages that have already been carried into the warehouse 2 and are scheduled to be stored on any shelf.

[0056] The order of the processes in step St40 and step St41 may be swapped, or they may be performed simultaneously in parallel.

[0057] Based on the filling rate measured in step St37 and the package information 25 obtained in step St41, the processor 11 selects a shelf recommended for storing the packages to be stored in the warehouse 2 hereafter (St42). The processor 11 may select a shelf recommended for storing packages as follows. As an example for explanation, assume that the filling rates of shelves a1, a2, a3, a4, and a5 are measured to be 10%, 10%, 20%, 30%, and 60% respectively. Also, for simplicity of explanation, assume that there is only one package to be stored in one of the shelves hereafter, and the volume of the package occupies 50% of the volume of the shelf.

[0058] At this time, the processor 11 may preferentially select shelves with low filling rates. In this case, the processor 11 selects shelf a1, shelf a2, or both.

[0059] Alternatively, when the processor 11 stores a new package on a shelf, the processor 11 may select a shelf so that the filling rate of the shelf approaches 100%. In other words, the processor 11 may preferentially select shelves with high filling rates. However, if the filling rate of the shelf exceeds 100% when a new package is stored on the shelf, the processor 11 does not select the shelf. In this case, the processor 11 selects shelf a4.

[0060] Alternatively, the processor 11 may select a shelf based on the type of package. For example, if the package to be stored in one of the shelves hereafter is food and the same food is stored on shelf a2, the processor 11 selects shelf a2 so that packages of the same type are stored together on one shelf.

[0061] Note that these are examples for explanation and are not intended to limit the method of selecting a shelf by the processor 11. However, from the perspective of the utilization efficiency of the warehouse 2, it is considered that the shelf on which packages are to be stored hereafter is recommended so that packages of the same type are stored together on one shelf and the filling rate of the shelf approaches 100%.

[0062] [Detection process of shelf frame] Next, with reference to FIG. 4, the detection process of the shelf frame by the detection device 10 will be described. FIG. 4 is a schematic diagram for explaining a successful detection example and a failed detection example of the shelf frame according to the first embodiment. The image 40-1 shown in FIG. 4 and the image 40-2 shown in FIG. 4 are images acquired by the sensor 20.

[0063] In this specification, a three-dimensional coordinate system composed of the X-axis, Y-axis, and Z-axis is used for explanation, and it is assumed that the directions of the three-dimensional coordinate systems in each figure correspond. In each figure, the direction of the arrow of the coordinate system shown in the figure is regarded as positive, and the opposite direction of the arrow is regarded as negative. For convenience of explanation, the positive direction of the X-axis may be referred to as "right", the negative direction of the X-axis as "left", the positive direction of the Y-axis as "front", the negative direction of the Y-axis as "back", the positive direction of the Z-axis as "up", and the negative direction of the Z-axis as "down". Note that the configuration of each axis and the setting of the origin are merely examples and are not limited thereto. Hereinafter, for convenience of explanation, it is assumed that the imaging plane of the sensor 20 and the opening plane of the shelf are parallel to the XZ plane, and the sensor 20 photographs the shelf from the positive direction of the Y-axis. Also, it is assumed that the upper frame among the four frames surrounding the opening plane of the shelf is parallel to the X-axis.

[0064] In the example of FIG. 4, the sensor 20 moves in the positive direction of the X-axis inside the warehouse 2. At this time, for example, the sensor 20 may be fixed to a cart, and the user may move the cart in the positive direction of the X-axis. Since the image acquired by the sensor 20 is displayed on the display device 15 in real time, with the movement of the sensor 20 in the positive direction of the X-axis, the shelf shown in the screen displayed on the display device 15 moves in the negative direction of the X-axis. Hereinafter, the screen displayed on the display device 15 may be referred to as the display screen.

[0065] First, with reference to Image 40-1, a successful example of detecting the shelf frame of Shelf 41 will be described. Image 40-1 is an image including Shelf 41 acquired by Sensor 20. Image 40-1 is displayed on Display Device 15. On Shelf 41, Goods 44 are stored. Processor 11 of Detection Device 10 focuses on the upper frame among the four frames surrounding the opening surface of Shelf 41 and the left and right frames that intersect perpendicularly with the upper frame for detecting the shelf frame. That is, Processor 11 focuses on Frame 41a, Frame 41b, and Frame 41c.

[0066] Processor 11 uses the RGB data, depth data, or both included in Image 40-1 to detect the line segments of the focused frames. That is, Processor 11 detects Line Segment 42a corresponding to Frame 41a, Line Segment 42b corresponding to Frame 41b, and Line Segment 42c corresponding to Frame 41c. When the size of the shelf including the length of the shelf frame is known in advance, Processor 11 may detect the line segments corresponding to each frame by comparing the line segments detected from Image 40-1 with the length of the shelf frame known in advance.

[0067] The detection of the shelf frame refers to detecting the line segments corresponding to each of the three frames, namely the upper frame and the left and right frames that intersect perpendicularly with the upper frame, among the four frames surrounding the opening surface of the shelf. In the example of Image 40-1, when Processor 11 can detect Line Segment 42a corresponding to Frame 41a, Line Segment 42b corresponding to Frame 41b, and Line Segment 42c corresponding to Frame 41c of Shelf 41, the detection of the shelf frame of Shelf 41 is successful. As a condition for successful detection of the shelf frame, the reason for excluding the detection of the line segment corresponding to the lower frame among the four frames surrounding the opening surface of the shelf is, for example, to facilitate successful detection of the shelf frame even when there is an obstacle between the shelf and Sensor 20. The case where there is an obstacle between the shelf and Sensor 20 may be, for example, when there is some object placed on the floor or when a person is standing. In such a case, depending on the height of the obstacle from the floor, even if the obstacle obstructs the photographing of the lower frame of the shelf by Sensor 20, it may not obstruct the photographing of the upper frame of the shelf by Sensor 20. Therefore, as a condition for successful detection of the shelf frame, the detection of the line segment corresponding to the lower frame among the four frames surrounding the opening surface of the shelf is excluded.

[0068] The processor 11 identifies the centroid position of the detected line segment, that is, the centroid coordinates. In the example of Image 40-1, the processor 11 identifies the respective coordinates of centroid 43a corresponding to line segment 42a, centroid 43b corresponding to line segment 42b, and centroid 43c corresponding to line segment 42c. Note that in order for the processor 11 to identify the centroid coordinates from the line segment, the origin of the coordinate system may be defined in advance. The centroid will be described later with reference to FIGS. 5, 6, and 7.

[0069] Next, with reference to Image 40-2, an example of a detection failure of the shelf frame of shelf 45 will be described. As the sensor moves in the positive X-axis direction, the image 40-1 displayed on the display device 15 transitions to image 40-2. Image 40-2 is an image including a part of shelf 45 in which luggage 48 is stored. The processor 11 focuses on frame 45a and frame 45c of shelf 45 for detecting the shelf frame. Different from the example of Image 40-1, since the left frame (the frame corresponding to frame 41b in the case of shelf 41) among the four frames surrounding the opening surface of shelf 45 is not included in Image 40-2, the processor 11 cannot focus on the left frame. Also, the frame 45a on which the processor 11 focuses is in a state of being cut off in Image 40-2.

[0070] The processor 11 detects the line segments of the focused frame using the RGB data, depth data, or both included in Image 40-2. The processor 11 detects line segment 46c corresponding to frame 45c. At this time, since frame 45a is cut off in Image 40-2, the processor 11 cannot detect the line segment corresponding to frame 45a (the line segment corresponding to line segment 42a in the case of shelf 41).

[0071] In the example of Image 40-2, the processor 11 can only detect the line segment corresponding to the right frame among the four frames surrounding the opening surface of shelf 45, that is, line segment 46c, and the detection of the shelf frame of shelf 41 fails.

[0072] [Counting Process for the Number of Shelves] Next, with reference to FIGS. 5 and 6, the count process of the number of shelves by the detection device 10 will be described. FIG. 5 is a schematic diagram for explaining an example of the count process of the number of shelves according to the first embodiment.

[0073] With reference to FIG. 5, a case where the shelf 52 and the shelf 55 adjacent to the shelf 52 are counted will be described. Note that the fact that a shelf is counted is synonymous with the fact that the shelf is detected. In addition, in order to count the number of two or more shelves, the sensor 20 is moved.

[0074] The image 50-1 shown in FIG. 5 is displayed on the display device 15. As the sensor moves in the positive direction of the X axis, the images appearing on the display screen transition in the order of the image 50-1, the image 50-2, the image 50-3, and the image 50-4. Each of the images acquired by the sensor 20 includes a region whose range in the image is defined in advance. In the example of FIG. 5, the images 50-1, 50-2, 50-3, and 50-4 each include the region 51. When the processor 11 of the detection device 10 successfully detects the shelf frame of the shelf and newly detects the centroid of the upper frame of the shelf for which the coordinates have been specified in the region 51 in the image, the number of shelves is counted. Hereinafter, it will be described with reference to the example of FIG. 5.

[0075] The processor 11 of the detection device 10 has successfully detected the shelf frame of the shelf 52 using the image 50-1. In addition, the processor 11 has detected the line segment 53a corresponding to the frame 52a of the shelf 52 and specified the coordinates of the centroid 54a corresponding to the line segment 53a. At this time, the processor 11 detects the centroid 54a in the region 51 and increments the count of the number of shelves. As a result, in the example of FIG. 5, the count of the number of shelves is one.

[0076] As the sensor moves in the positive direction of the X axis, the image 50-1 displayed on the display device 15 transitions to the image 50-2. In the image 50-2, the centroid 54a moves in the negative direction of the X axis with respect to the region 51, but remains included in the region 51. Although the processor 11 detects the centroid 54a in the region 51, since it is not newly detected, the count of the number of shelves is not incremented.

[0077] As the sensor moves in the positive X-axis direction, the image 50-2 displayed on the display device 15 transitions to the image 50-3. In the image 50-3, the center of gravity 54a has moved in the negative X-axis direction with respect to the region 51 and is outside the region 51. In this case, since the processor 11 has not detected either the center of gravity 54a or a center of gravity different from the center of gravity 54a, it determines that the center of gravity is not detected. Further, when the processor 11 determines that the center of gravity is not detected, it starts counting the number of frames in the state where the center of gravity is not detected. For example, when the center of gravity is not detected in a certain frame, the processor 11 counts the number of frames in the state where the center of gravity is not detected as 1 frame. Then, when the center of gravity is not detected in the next frame after the said frame, the processor 11 counts the number of frames in the state where the center of gravity is not detected as 2 frames. After the processor 11 has counted the number of frames in the state where the center of gravity is not detected for a specified number of frames or more, when a center of gravity is newly detected in the region 51, it increments the shelf count. If the number of frames counted from when the center of gravity became undetected until the next time the center of gravity is detected in the region 51 is less than the specified number of frames, the processor 11 does not increment the shelf count. This is to avoid the situation where, although the center of gravity is included in the region 51, the detection fails temporarily (for example, for only 1 frame), and then the center of gravity is detected, and the shelf count is incremented even though the same shelf is actually detected. For the sake of convenience of explanation, it is assumed that the processor 11 has counted the number of frames in the state where the center of gravity is not detected for a specified number of frames or more.

[0078] As the sensor moves in the positive X-axis direction, the image 50-3 displayed on the display device 15 transitions to the image 50-4. The processor 11 uses the image 50-4 to successfully detect the shelf frame of the shelf 55 adjacent to the shelf 52 and detects the line segment 57a corresponding to the frame 56a of the shelf 55. The processor 11 further specifies the coordinates of the center of gravity 58a corresponding to the line segment 57a. Then, the processor 11 detects the center of gravity 58a in the region 51 and increments the shelf count. As a result, the shelf count becomes 2.

[0079] Note that the number of frames and the range of the region 51 in the above regulations may be determined based on the moving speed of the sensor 20, the frame rate, the size of the frame on the shelf, the magnification of the lens included in the sensor 20, the positional relationship between the sensor 20 and the shelf, or a combination thereof. Further, the processor 11 may obtain the number of frames in a state where the center of gravity is not detected by estimating the moving speed of the sensor 20 from the amount of movement of the center of gravity per frame. The positional relationship between the sensor 20 and the shelf is, for example, the distance between the sensor 20 and the shelf.

[0080] In this way, the processor 11 detects the shelf frame of a certain shelf, specifies the coordinates of the center of gravity of the frame on the shelf, and counts up the number of shelves by detecting the center of gravity in the region 51 with a predefined range in the image captured by the sensor 20. Then, when the processor 11 counts up the number of shelves by one, after a state where the center of gravity is not detected for at least the specified number of frames continues, the processor 11 detects the center of gravity in the new region 51 to count up the number of the next shelf. That is, the detection of the center of gravity in the region 51 by the processor 11 and the non-detection of the center of gravity for a certain period (the specified number of frames) are repeated to count up the number of shelves.

[0081] With reference to FIG. 5, an example of counting the number of shelves using one region 51 has been described. Next, with reference to FIG. 6, an example of counting the number of shelves using two regions will be described. FIG. 6 is a schematic diagram for explaining an example of the shelf number counting process according to the first embodiment.

[0082] With reference to FIG. 6, a case where the shelf 62 and the shelf 65 adjacent to the shelf 62 are counted will be described. Similar to the example of FIG. 5, the sensor 20 moves in the positive direction of the X-axis inside the warehouse 2, and the shelf photographed by the sensor 20 moves in the negative direction of the X-axis within the display screen.

[0083] The image 60-1 shown in FIG. 6 is displayed on the display device 15. As the sensor moves in the positive X-axis direction, the images appearing on the display screen transition in the order of image 60-1, image 60-2, image 60-3, and image 60-4. Each of the images acquired by the sensor 20 includes two regions whose ranges are defined in advance within the image. In the example of FIG. 6, the images 60-1, 60-2, 60-3, and 60-4 each include a region 61R and a region 61L. Note that the region 61R and the region 61L are adjacent, the region 61R is located on the right, and the region 61L is located on the left.

[0084] First, with reference to the image 60-1, the case of counting up the number of shelves from 0 to 1 will be described. The processor 11 of the detection device 10 has successfully detected the shelf frame of the shelf 62 using the image 60-1. Also, the processor 11 has detected a line segment 63a corresponding to the frame 62a of the shelf 62 and specified the coordinates of the centroid 64a corresponding to the line segment 63a. At this time, the processor 11 detects the centroid 64a in the region 61R and counts up the number of shelves. As a result, in the example of FIG. 6, the count of the number of shelves becomes 1. When the processor 11 counts the number of shelves using the two regions 61R and 61L, the first shelf to be counted is counted using only the region 61R.

[0085] As the sensor moves in the positive X-axis direction, the image 60-1 displayed on the display device 15 transitions to the image 60-2. In the image 60-2, the centroid 64a moves in the negative X-axis direction with respect to the region 61R and is included in the region 61L. The processor 11 detects the centroid 64a in the region 61L.

[0086] As the sensor moves in the positive X-axis direction, the image 60-2 displayed on the display device 15 transitions to the image 60-3. In the image 60-3, since the centroid 64a has moved in the negative X-axis direction with respect to the region 61L, it is outside the regions 61R and 61L. In this case, since the processor 11 has not detected either the centroid 64a or a centroid different from the centroid 64a, it is determined that the centroid is not detected.

[0087] As the sensor moves in the positive X-axis direction, the image 60-3 displayed on the display device 15 transitions to the image 60-4. Using the image 60-4, the processor 11 successfully detects the shelf frame of the shelf 65 adjacent to the shelf 62 and detects the line segment 67a corresponding to the frame 66a of the shelf 65. The processor 11 further identifies the coordinates of the centroid 68a corresponding to the line segment 67a. Then, the processor 11 detects the centroid 68a in the region 61R and increments the shelf count. As a result, the shelf count becomes two units.

[0088] In this way, first, when the centroid 64a is detected in the region 61R, the shelf 62 is detected and the shelf count is incremented. After that, when the centroid 64a detected in the region 61R moves to the left region 61L and, after the processor 11 fails to detect the centroid, the centroid 68a is detected in the region 61R, the shelf count is incremented to the next unit number. That is, the detection of the centroid by the right region 61R, the detection of the centroid by the left region 61L, the non-detection of the centroid, and the detection of the centroid by the right region 61R are repeated, thereby incrementing the shelf count. In other words, based on the above-described order of detection, continuously arranged shelves are detected. In the example of FIG. 6, the shelves 62 and 65 are detected in the order of the shelf 62 and the shelf 65. If the above-described detection process has not been passed through, the processor 11 determines that it has not been able to detect the shelf normally and does not increment the shelf count.

[0089] Furthermore, the processor 11 can determine the moving direction of the sensor 20 based on the above-described order of detection. For example, when the detection of the centroid by the right region 61R, the detection of the centroid by the left region 61L, the non-detection of the centroid, and the detection of the centroid by the right region 61R are repeated, the processor 11 can determine that the moving direction of the sensor 20 is the positive X-axis direction. More simply put, the processor 11 can determine the moving direction of the sensor 20 based on the moving direction of the shelf on the display screen.

[0090] In FIG. 6, an example where the sensor moves in the positive X-axis direction has been described. However, the sensor may move in the negative X-axis direction. In this case, the moving direction of the shelf on the display screen is the positive X-axis direction. Also, in this case, by repeating the detection of the center of gravity by the left region 61L, the detection of the center of gravity by the right region 61R, the non-detection of the center of gravity, and the detection of the center of gravity by the left region 61L, the count-up of the number of shelves is performed. If the above-described detection process has not been passed, the processor 11 determines that the shelf could not be detected normally and does not count up the number of shelves.

[0091] Also, the respective ranges of the above-described region 61R and region 61L may be determined based on the moving speed of the sensor 20, the frame rate, the size of the frame on the shelf, the magnification of the lens included in the sensor 20, the positional relationship between the sensor 20 and the shelf, or a combination thereof.

[0092] In the shelf number counting process described with reference to FIG. 5, as a countermeasure against a temporary failure in detecting the center of gravity, it is a condition that the non-detection of the center of gravity continues for a specified number of frames or more. As a countermeasure against a temporary failure in detecting the center of gravity, a method of counting the number of shelves using the two regions 61R and 61L described with reference to FIG. 6 may be adopted.

[0093] Conventionally, when the colors, sizes, and shapes of a plurality of shelves are the same or similar, each shelf has been detected by character recognition of the characters described on each shelf or by reading the barcodes attached to each shelf. Also, a method of distinguishing and detecting a shelf by identifying the goods stored on the shelf has been considered. However, when the shapes and sizes of the goods stored on each shelf are the same or similar, it is difficult to distinguish and detect each shelf by identifying the goods. The detection system 1 in the present embodiment can detect and count shelves more simply than in the prior art by adopting the method of detecting the shelf frame.

[0094] [Selection Process of Best Shot Image] Next, with reference to FIG. 7, the best shot image and the selection process of the best shot image will be described. In the examples of FIGS. 5 and 6, for simplicity of explanation, four images are shown in each figure, but actually, more images are captured by the sensor 20. The best shot image is an image among a plurality of images captured by the sensor 20 for a certain shelf, in which the center-of-gravity coordinates of the upper frame of the shelf are closest to the center of the image. The processor 11 calculates the filling rate using the best shot image. Further, the processor 11 stores the best shot image and the filling rate calculated using the best shot image in the warehouse management system. That is, the processor 11 selects the best shot image for calculating the filling rate and for recording. FIG. 7 is a schematic diagram for explaining an example of the best shot image according to Embodiment 1.

[0095] The image 70 is an image including the shelf 71. The processor 11 has succeeded in detecting the shelf frame of the shelf 71. Further, a line segment 72a corresponding to the upper frame of the shelf 71, that is, the frame 71a, is detected by the processor 11, and the coordinates of the center of gravity 73a corresponding to the line segment 72a are specified. The image center line 74 is a line indicating the center in the X-axis direction of the image 70. The processor 11 selects, as the best shot image, an image in which the center of gravity 73a is closest to the image center line 74 among the images including the shelf 71 captured by the sensor 20. For example, the processor 11 selects the image 70 as the best shot image.

[0096] Note that, for example, in the image 70, if the absolute value of the inclination of the line segment 72a with respect to the X-axis is greater than a specified threshold value, the processor 11 may exclude the image 70 from the candidates for the best shot image. This is, for example, when the user moves the cart on which the sensor 20 is fixed for photographing the shelf and the cart meanders, causing the sensor 20 to tilt with respect to the shelf. In this case, the image captured with the sensor 20 tilted with respect to the shelf is excluded from the candidates for the best shot image. The specified threshold value may be preset by the user.

[0097] [Measurement Process of Filling Rate] FIG. 8 is a schematic diagram for explaining an example of calculating the filling rate according to Embodiment 1. The processor 11 of the detection device 10 calculates the filling rate of a certain shelf using the best shot image of the shelf. In the example of FIG. 8, the processor 11 calculates the filling rate of shelf 82 using the best shot image of shelf 82. Luggage 81 is stored on shelf 82. The processor 11 calculates the filling rate of shelf 82, that is, the ratio of the volume of shelf 82 occupied by the volume of luggage 81. In Embodiment 1, the luggage stored on the shelf occupies the volume of the shelf up to the innermost surface facing the opening surface. Briefly speaking, when luggage is stored on a shelf, the luggage is stored deep inside the shelf like luggage 81 shown in FIG. 8.

[0098] The processor 11 generates 3D point cloud data 83 using an image including depth information acquired by the sensor 20. The image is the best shot image of shelf 82. For convenience of explanation, FIG. 8 is represented as a ZY plane view. The processor 11 voxelizes the generated 3D point cloud data 83 to generate voxel data 84. Then, the processor 11 compares the generated voxel data 84 with the empty volume data 85. The empty volume data 85 is voxel data of the volume when no luggage is stored on shelf 82, that is, when shelf 82 is empty, and is data included in the shelf size information 17. In the example of FIG. 8, the processor 11 calculates the filling rate of shelf 82 to be 60 percent by comparing the voxel data 84 with the empty volume data 85.

[0099] In Embodiment 1, the image used for calculating the filling rate of a certain shelf only needs to be an image including the shelf. Therefore, for example, data on the overall shape of the warehouse 2 including the shelf is unnecessary. Accordingly, since the filling rate of the shelf can be calculated according to the acquisition of the image including the shelf, real-time measurement of the filling rate becomes possible.

[0100] [Display screen] Next, with reference to FIGS. 9, 10, 11, and 12, the display screen displayed on the display device 15 will be described. FIGS. 9, 10, 11, and 12 are schematic diagrams for explaining an example of the display screen according to Embodiment 1.

[0101] The display screen 90 shown in FIG. 9 is a screen at a stage before an operation for starting measurement of the filling rate by the user. The display screen 90 displayed on the display device 15 includes a real-time screen 91 and an operation screen 92.

[0102] An image acquired by the sensor 20 is displayed in real time on the real-time screen 91. In the example of FIG. 9, the shelves 94, 95, and 96 are shown on the real-time screen 91. In the real-time screen 91, a part of the shelves 94 and 96 is shown, while the entire shelf 95 is shown. This is because the sensor 20 is located in front of the shelf 95. Even at a stage before an operation for starting measurement of the filling rate by the user, the processor 11 of the detection device 10 may detect a detectable shelf based on the image acquired by the sensor 20. In the example of FIG. 9, the processor 11 may detect the shelf 95 based on the image acquired by the sensor 20, that is, the image displayed on the real-time screen 91. The processor 11 cannot detect the shelves 94 and 96 that are not entirely photographed. More precisely, in the image displayed on the real-time screen 91, since the upper frame and the left frame among the four shelf frames surrounding the opening surface of the shelf 94 are cut off, the processor 11 cannot detect the shelf frame of the shelf 94. Also, in the image displayed on the real-time screen 91, since the upper frame and the right frame among the four shelf frames surrounding the opening surface of the shelf 96 are cut off, the processor 11 cannot detect the shelf frame of the shelf 96. Therefore, the processor 11 cannot detect the shelves 94 and 96.

[0103] Even before the operation to start measuring the filling rate by the user is performed, the processor 11 may measure the filling rate of the shelf detected based on the image displayed on the real-time screen 91. In the example of FIG. 9, the processor 11 may measure the filling rate of the shelf 95. On the real-time screen 91, a filling rate display window 93 for displaying the filling rate measured by the processor 11 overlapping the image acquired by the sensor 20 is displayed. In the filling rate display window 93, the shelf number of the shelf for which the filling rate is to be measured and the filling rate of the shelf are displayed. In the example of FIG. 9, the filling rate of the shelf 95 is displayed in the filling rate display window 93. In the example of FIG. 9, the shelf number of the shelf 95 is a2, and the filling rate of the shelf 95 is 60%. Even before the operation to start measuring the filling rate by the user is performed, the filling rate measured by the processor 11 based on the image acquired by the sensor 20 may be displayed in the filling rate display window 93. In the filling rate display window 93, the filling rate is displayed, for example, in the range from 0 percent to 100 percent.

[0104] On the operation screen 92, a shelf row selection window 97 for the user to specify the shelf row for which the filling rate is to be measured is displayed. The processor 11 may display each shelf row read from, for example, the shelf number table 16 as candidates on the operation screen 92 for the user to specify the shelf row in the shelf row selection window 97. Then, the user may specify the shelf row for which the filling rate is to be measured from the displayed shelf row candidates. In the example of FIG. 9, the shelf row a is specified. The user's specification of the shelf row corresponds to step St30 of the flowchart shown in FIG. 3.

[0105] On the operation screen 92, further, a measurement range specification window 98 for the user to specify the range of the shelves included in the shelf row specified by the user for which the filling rate is to be measured is displayed. In the example of FIG. 9, the range from shelf a1 to shelf a7 is specified. The user's specification of the shelf range corresponds to step St31 of the flowchart shown in FIG. 3. When the measurement range is from shelf a1 to shelf a7, before operating the measurement start button 99 described later, the user moves the sensor 20 to the front of shelf a1, that is, to a position where the sensor 20 can capture an image including shelf a1.

[0106] On the operation screen 92, a start button 99 for the user to perform an operation to start measuring the filling rate is further displayed. The user operates the start button 99 to start the measurement. For example, when the input device 14 is a mouse, the user clicks it, and when the display device 15 is a touch panel, the user taps the screen or the like to operate the start button 99. The operation by the user to start the measurement corresponds to step St32 in the flowchart shown in FIG. 3. The user moves the sensor 20 from the front of shelf a1 to the front of shelf a7. That is, the user moves the sensor 20 so that the sensor 20 can photograph each of shelves a1, a2, a3, a4, a5, a6, and a7.

[0107] When the user operates the start button 99, the display screen 90 shown in FIG. 9 transitions to the screen shown in FIG. 10. Specifically, on the operation screen 92, a count display text 100 indicating the progress of shelf detection and an end button 101 to be operated when the user finishes moving the sensor 20 are displayed. When shelves a1 to a7 are the measurement targets for the filling rate, the user moves the sensor 20 so that the sensor 20 can photograph each of shelves a1, a2, a3, a4, a5, a6, and a7. In response to the movement of the sensor 20, each shelf from shelf a1 to shelf a7 is sequentially displayed on the real-time screen 91. Also, in response to the movement of the sensor 20, the processor 11 detects each shelf from shelf a1 to shelf a7. Based on the number of shelves counted by the processor 11, the progress of shelf detection is displayed in the count display text 100. In the example of FIG. 10, the count display text 100 indicates that the detection of 3 out of 7 shelves is completed. At this time, the number of shelves counted by the processor 11 is 3. The processor 11 acquiring the image photographed by the sensor 20, detecting the shelf, that is, the shelf frame, based on the image, and counting the number of shelves corresponds to steps St33 to St35 in the flowchart shown in FIG. 3.

[0108] According to the detection of each shelf, the processor 11 selects the best shot image of each shelf and measures the filling rate of each shelf using the best shot image. This corresponds to steps St36 and St37 of the flowchart shown in FIG. 3. In the filling rate display window 93, the filling rate of the shelf shown in the real-time screen 91 may be displayed, or the filling rate of each shelf from shelf a1 to shelf a7 may be sequentially displayed according to the measurement status of the filling rate by the processor 11.

[0109] When the user completes the movement of the sensor 20 or wants to interrupt a series of processes for measuring the filling rate, the user operates the end button 101. The end button 101 may be operable in the same manner as the start button 101. Note that when the movement of the sensor 20 by the user is completed and the user operates the end button 101, it is not limited whether the measurement of the filling rate by the processor 11 is completed. However, since the processor 11 detects the shelf in real time according to the acquisition of the image, at the time when the user operates the end button 101, that is, when the movement of the sensor 20 by the user is completed, the detection of each shelf is completed and the shelf number count is also completed.

[0110] When the user operates the end button 101, the display screen 90 shown in FIG. 10 transitions to the screen shown in FIG. 11. When the measurement range is from shelf a1 to shelf a7, as shown in FIG. 11, the display content of the count display text 100 indicates that 7 out of 7 shelves have been detected. Further, on the operation screen 92, a remeasurement button 102 and a result display button 103 are displayed. The remeasurement button 102 and the result display button 103 may be operable in the same manner as the start button 99. By operating the remeasurement button 102, the user can start over from the selection of the shelf row. That is, when the remeasurement button 102 is operated by the user, the display screen 90 shown in FIG. 11 transitions to the screen shown in FIG. 9.

[0111] In addition, for example, if one shelf is not detected due to an obstacle between the sensor 20 and the shelf, the display content of the count display sentence 100 indicates that 6 out of 7 shelves have been detected. If one or more shelves are not detected, that is, if there is an error in shelf detection, the processor 11 displays a message indicating that there is an error on the display screen 90 and prompts the user to re-measure. Also, when the processor 11 displays a message indicating that there is an error on the display screen 90, it may also display, on the display screen 90, information on the section where the error occurred and the image captured by the sensor 20 in that section. The section where the error occurred is the section including the undetected shelf. Also, the detected shelves are not included in the section where the error occurred. For example, if the measurement range is from shelf a1 to shelf a7 and shelves a5, a6, and a7 are not detected, the section where the error occurred is the section from shelf a5 to shelf a7. In this case, since the processor 11 can detect shelves a1, a2, a3, and a4, it can determine that the undetected shelves are a5, a6, and a7. Then, the processor 11 displays on the display screen 90 that shelves a5 to a7 could not be detected and the image captured by the sensor 20 in the section from shelf a5 to shelf a7. Thereby, the user can confirm the undetected shelf and the image that may contain the cause of the detection error.

[0112] When the user operates the result display button 103, the display screen 104 is displayed on the display device 15. When the measurement range is from shelf a1 to shelf a7, on the display screen 104, the best shot images of each shelf from shelf a1 to shelf a7 and the filling rate of each shelf are displayed in a list. In the example of FIG. 12, images 105, 106, 107, 108, 109, and 110 are displayed, which are the best shot images of shelves a1, a2, a3, a4, a5, and a6, respectively. The best shot image and the filling rate of shelf a7 appear on the display screen 104 when the user operates the scroll bar 111.

[0113] On the display screen 104, an OK button 112 and an NG button 113 are further displayed. The OK button 112 and the NG button 113 may be operable in the same manner as the start button 99. The user can start over from the selection of the shelf row by operating the NG button 113. That is, when the NG button 113 is operated by the user, the display screen 90 shown in FIG. 9 is displayed on the display device 15.

[0114] The user checks the best shot image and the filling rate of each shelf on the display screen 104, and if it is determined that there is no problem, the user operates the OK button 112. This corresponds to step St38 of the flowchart shown in FIG. 3. Thereby, the processor 11 accesses the warehouse management system 21 (step St39 of the flowchart shown in FIG. 3). Then, the processor 11 executes the processes from step St40 and subsequent steps of the flowchart shown in FIG. 3.

[0115] Note that the best shot image of each shelf displayed on the display screen 104 may be selectable by a user operation. For example, when the input device 14 is a mouse, the user can select the best shot image of each shelf by an operation such as clicking, and when the display device 15 is a touch panel, the user can select the best shot image of each shelf by an operation such as tapping the screen. Further, when the user operates the OK button 112 after selecting one or more best shot images, the best shot image selected by the user and the filling rate of the shelf corresponding to the best shot image may be stored in the warehouse management system 21. In this case, the best shot image not selected by the user and the filling rate of the shelf corresponding to the best shot image are not stored in the warehouse management system 21. Thereby, the user can check the best shot image and the filling rate of each shelf on the display screen 104, and store only the best shot image and the filling rate in the warehouse management system 21 for the shelves determined to have no problem.

[0116] Further, the processor 11 may determine that a shelf corresponding to a best shot image that was not stored in the warehouse management system 21, i.e., not selected by the user, is a shelf to be re-measured. For example, if the measurement range is from shelf a1 to shelf a7, and after the user selects the best shot images of shelves a3 to a7, and then the user operates the OK button 112, a display screen 90 shown in FIG. 9 may be displayed on the display device 15. At this time, the processor 11 that has determined that shelves a1 and a2 are shelves to be re-measured may be pre-input so that column a is selected in the shelf column selection window 97 and the range from shelf a1 to shelf a2 is specified in the measurement range specification window 98. Thereby, the trouble of the user specifying the shelves to be re-measured can be saved.

[0117] Note that the display contents of the display screen 90 and the display screen 104 described with reference to FIGS. 9, 10, 11, and 12 are examples. The character strings and the like described in each figure are not necessarily limited to those character strings and the like. Also, the display format of the best shot images and the filling rates of the respective shelves displayed in a list is not limited to the example shown in FIG. 12.

[0118] (Modification Example of Embodiment 1) In the above-described Embodiment 1, as described with reference to FIG. 5, an example was shown in which one center of gravity 54a or center of gravity 58a exists in one region 51 with a predefined range in the image captured by the sensor 20. That is, in Embodiment 1, an example is shown in which one center of gravity exists in one region 51 and the center of gravity is detected. However, the present invention is not limited to this. Even when the centers of gravity of the upper frames of a plurality of shelves exist in one region 51, the detection device 10 can detect the plurality of shelves. The case where the centers of gravity of the upper frames of a plurality of shelves exist in one region 51 means, for example, a case where the angle of view of the lens included in the sensor 20 is wide and the entire appearance of at least two or more shelves is included in the image captured by the sensor 20. In such a case, when a new center of gravity appears in the region 51, in other words, when a new center of gravity is detected in the region 51, the detection device 10 increments the shelf count. For example, when the shelves to be measured are shelf a1 and shelf a2, the sensor 20 is moved so that shelf a1 and shelf a2 are photographed. At this time, when the center of gravity of the upper frame of shelf a1 exists in the region 51 of the image captured by the sensor 20 and a new center of gravity of the upper frame of shelf a2 appears in the region 51 in response to the movement of the sensor 20, the shelf count may be incremented. Also, when the center of gravity of the upper frame of shelf a1 moves out of the region 51 in the image and a new center of gravity of the upper frame of shelf a2 appears in the region 51 in response to the movement of the sensor 20, the shelf count may be incremented.

[0119] Also, in Embodiment 1, as described with reference to FIG. 6, when one center of gravity 64a or center of gravity 68a exists in either of the two regions 61L and 61R, an example was shown in which no center of gravity exists in a region different from the region where the center of gravity 64a or center of gravity 68a exists. However, the present invention is not limited to this. For example, referring to FIG. 6, there may be a case where the center of gravity 64a exists in the region 61L and the center of gravity 68a exists in the region 61R. For example, when the angle of view of the lens included in the sensor 20 is wide, etc., the center of gravity 64a may exist in the region 61L and the center of gravity 68a may exist in the region 61R. Thus, when one center of gravity may exist in each of the region 61L and the region 61R, the detection device 10 may count up the number of shelves when detecting the center of gravity in either the region 61L or the region 61R based on the moving direction of the sensor 20. For example, when the moving direction of the sensor 20 is as shown in FIG. 6, the detection device 10 may count up the number of shelves as if a shelf has been detected when detecting the center of gravity in the region 61L. At this time, when the detection device 10 detects the center of gravity in the region 61R, the detection device 10 may recognize the center of gravity as a candidate for the center of gravity to be detected in the region 61L next. Thereby, for example, when the center of gravity detected in the region 61R does not appear in the region 61L according to the movement of the sensor, the detection device 10 can determine that some error has occurred and notify the user.

[0120] In addition, in Embodiment 1, the shelf row a was composed of shelves a1, a2, a3, a4, a5, a6, and a7. That is, the shelf row a was formed by arranging one shelf continuously and adjacently. However, it is not limited to this, and the shelves arranged in the warehouse 2 may be stacked vertically. For example, further shelves may be arranged on each of the shelves a1, a2, a3, a4, a5, a6, and a7. In this case, for example, the shelf numbers of the upper shelves may be a1-1, a2-1, a3-1, a4-1, a5-1, a6-1, and a7-1, and the shelf numbers of the lower shelves may be a1-2, a2-2, a3-2, a4-2, a5-2, a6-2, and a7-2. That is, the vertically stacked shelves are distinguishable from each other. Thereby, when specifying the range of the shelves for which the filling rate is to be measured, the user can distinguish and specify either the upper shelf or the lower shelf even within the same shelf row.

[0121] In addition, in Embodiment 1, the detection device 10 detected the shelf frame by detecting line segments corresponding to the upper frame, the left frame, and the right frame among the four frames surrounding the opening surface of the shelf. However, it is not limited to this. The detection device 10 may detect the shelf frame by detecting line segments corresponding to the lower frame, the left frame, and the right frame among the four frames surrounding the opening surface of the shelf. For example, in Embodiment 1, when the shelves are arranged vertically overlapping each other and there is paper pasted between the upper shelf and the lower shelf, etc., there is a possibility that the upper shelf is detected and the lower shelf is not detected. This is because the paper pasted between the upper shelf and the lower shelf may obstruct the imaging of the upper frame of the lower shelf by the sensor 20. However, when the detection device 10 can detect the shelf frame by detecting line segments corresponding to the lower frame, the left frame, and the right frame of the shelf, the detection device 10 can detect the lower shelf. The detection device 10 may further selectively use the detection of the shelf frame using the upper frame, the left frame, and the right frame and the detection of the shelf frame using the lower frame, the left frame, and the right frame. For example, when there is paper pasted between the upper shelf and the lower shelf, the detection device 10 may detect the upper shelf by detecting line segments corresponding to the upper frame, the left frame, and the right frame of the upper shelf, and detect the lower shelf by detecting line segments corresponding to the lower frame, the left frame, and the right frame of the lower shelf.

[0122] (Summary of Embodiment 1) According to the description of Embodiment 1 above, at least the following technologies are disclosed. The components corresponding in Embodiment 1 are exemplified in parentheses, but it is not limited thereto.

[0123] (Technology 1) The detection device (e.g., detection device 10) is connected to a sensor (e.g., sensor 20) that acquires an image (e.g., image 50-1) including depth information corresponding to the distance from a reference position to a shelf (e.g., shelf 52), and has a communication interface (e.g., communication interface 13). The detection device also has a processor (e.g., processor 11) that detects a shelf frame (e.g., frame 53a) indicating the outer shape of the shelf from the image received from the sensor and identifies the position of the center of gravity (e.g., center of gravity 54a) of the shelf. The processor detects the shelf when the position of the center of gravity is within the first region (e.g., region 51) of the image. After detecting the first shelf, when the detection device detects the position of the center of gravity of a second shelf adjacent to the first shelf in the first region in response to the movement of the sensor, the detection device detects the second shelf.

[0124] As a result, the detection device acquires an image including depth information corresponding to the distance from the reference position to the shelf. Also, the detection device can detect the shelf when the position of the center of gravity of the shelf exists in the first region of the acquired image. Thus, the detection device can accurately detect each shelf even when the shelves are arranged adjacent to each other.

[0125] (Technology 2) In the detection device according to Technology 1, after detecting the first shelf, when the position of the center of gravity of the first shelf is not detected in the first region and the position of the center of gravity of a second shelf adjacent to the first shelf in the first region is detected in response to the movement of the sensor, the detection device detects the second shelf.

[0126] As a result, when detecting the first shelf and the second shelf, the detection device can have a period during which the position of the center of gravity of the first shelf is not detected between the detection of the position of the center of gravity of the first shelf and the detection of the position of the center of gravity of the second shelf. Thus, the detection device can avoid double-counting the number of shelves even though it is actually detecting the same first shelf.

[0127] (Technology 3) In the detection device according to Technology 1 or 2, the range of the first region in the image is determined based on the moving speed of the sensor, the frame rate of the sensor, and the size of the shelf.

[0128] As a result, the range of the first region in the image is determined within a suitable range based on the moving speed of the sensor, the specifications of the sensor, and the specifications of the shelf.

[0129] (Technology 4) In the detection device according to any one of Technologies 1 to 3, the processor detects the shelf frame of the shelf from each image acquired by the sensor, and when the shelf frame can be detected, specifies the center-of-gravity position of the shelf.

[0130] As a result, the detection device can detect the shelf for each image acquired by the sensor.

[0131] (Technology 5) In the detection device according to any one of Technologies 1 to 4, the processor specifies the center-of-gravity position of the shelf in each of a plurality of images including the shelf repeatedly acquired by the sensor, and selects, as the best-shot image for recording, the image among the plurality of images in which the center-of-gravity position of the shelf is closest to the center of the image.

[0132] As a result, the detection device can calculate the filling rate described later based on the best-shot image. Further, the detection device can save the best-shot image in a warehouse management system described later.

[0133] (Technology 6) In the detection device according to Technology 5, the processor calculates a filling rate indicating the ratio of the volume of the goods stored in the shelf to the volume of the shelf included in the best-shot image based on the best-shot image.

[0134] As a result, the detection device can calculate the filling rate of the shelf.

[0135] (Technology 7) In the detection device according to Technique 6, the communication interface communicates with a warehouse management system (for example, warehouse management system 21) that stores package information (for example, package information 25) regarding packages stored in a warehouse (for example, warehouse 2). The processor acquires the package information from the warehouse management system via the communication interface, and based on the package information and the filling rate, causes a display device (for example, display device 15) to display the shelves recommended for storing the packages scheduled for incoming shipment.

[0136] As a result, the detection device can acquire the package information from the warehouse management system. Further, the detection device can select the shelves on which the packages to be stored in the warehouse should be stored for the efficient use of the warehouse, based on the calculated filling rate and the acquired package information. Also, for example, the user of the detection device can confirm, via the display device, the shelves recommended by the detection device.

[0137] (Technique 8) In the detection device according to Technique 6 or 7, the processor causes the display device to display the best shot image and the filling rate calculated based on the best shot image.

[0138] As a result, for example, the user of the detection device can confirm, via the display device, the filling rate of the shelves and the images.

[0139] (Technique 9) In the detection device according to any one of Techniques 6 to 8, the sensor acquires a series of images including a plurality of shelves arranged continuously, the processor selects the best shot image of each of the plurality of shelves using the series of images, calculates the filling rate of each of the plurality of shelves based on the best shot image of each of the plurality of shelves, and causes the display device to display the best shot image and the filling rate of each of the plurality of shelves.

[0140] As a result, for example, the user of the detection device can confirm, via the display device, a list of the filling rates and images of the plurality of shelves.

[0141] (Technique 10) In the detection device according to any one of Technologies 1 to 9, when the processor detects each of a plurality of shelves, the processor causes the display device to display a detection status indicating how many of the plurality of shelves have been detected.

[0142] Thereby, for example, the user of the detection device can confirm by means of the display device whether the detection of the shelves is being performed without problems.

[0143] (Technology 11) In the detection device according to any one of Technologies 6 to 9, the processor causes the display device to display the filling rate of the shelf in the range from 0 percent to 100 percent.

[0144] Thereby, for example, the user of the detection device can confirm the filling rate of the shelf numerically.

[0145] (Technology 12) In the detection device according to any one of Technologies 1 to 11, when the processor fails to detect a shelf, the processor causes the display device to display an error message.

[0146] Thereby, for example, when the shelf cannot be detected, the user of the detection device can confirm that fact.

[0147] (Technology 13) In the detection device according to any one of Technologies 1 to 12, the processor detects the first shelf and the second shelf in an order based on the detection order that is the order of detection in the first region of the center-of-gravity position, detection in the second region (for example, region 61L) adjacent to the first region of the center-of-gravity position, and non-detection in the first and second regions of the center-of-gravity position.

[0148] Thereby, by detecting the shelves using two first regions and second regions, the detection device can take measures against temporary detection errors of the center of gravity of the shelves as compared with the case of detecting the shelves using one first region.

[0149] (Technology 14) In the detection device according to Technique 13, the processor determines the moving direction of the sensor based on the detection order.

[0150] Thereby, the detection device can determine the moving direction of the sensor based on the order of detection in the first region of the center-of-gravity position of the shelf, detection in the second region of the center-of-gravity position of the shelf, and non-detection of the center-of-gravity position of the shelf, respectively.

[0151] (Technique 15) The detection method acquires an image including depth information corresponding to the distance from the reference position to the shelf, detects a shelf frame indicating the outer shape of the shelf from the image to specify the center-of-gravity position of the shelf, detects the shelf when the center-of-gravity position exists within the first region of the image, and after detecting the first shelf, when detecting the center-of-gravity position of the second shelf adjacent to the first shelf in the first region in response to the movement of the sensor, detects the second shelf.

[0152] Thereby, the detection method can obtain the same effect as Technique 1.

[0153] (Technique 16) The detection system (for example, Detection System 1) includes a communication interface connected to a sensor that acquires an image including depth information corresponding to the distance from the reference position to the shelf, a processor that detects a shelf frame indicating the outer shape of the shelf from the image acquired by the sensor to specify the center-of-gravity position of the shelf, a warehouse management system that stores luggage information regarding the luggage stored in the warehouse, and a display device. The processor specifies the center-of-gravity position of the shelf in each of a plurality of images including the shelf acquired by the sensor, selects, as the best shot image, the image in which the center-of-gravity position of the shelf among the plurality of images is closest to the center of the image, calculates a filling rate indicating the ratio of the volume of the luggage stored in the shelf occupied by the volume of the shelf included in the best shot image, acquires luggage information from the warehouse management system via the communication interface, and causes the display device to display the shelf recommended for storing the luggage scheduled to be carried in based on the luggage information and the filling rate.

[0154] Thereby, the detection system can obtain the same effect as Technique 6.

[0155] The functions of the above-described embodiments can also be realized by supplying a program and an application for realizing the functions of the above-described embodiments to a system or apparatus using a network or a storage medium or the like, and causing one or more processors in a computer of the system or apparatus to read and execute the program.

[0156] In addition, the functions of the above-described embodiments may be realized by a circuit (for example, an Application Specific Integrated Circuit (hereinafter referred to as "ASIC") or an FPGA) that realizes one or more functions.

[0157] As described above, the embodiments according to the present disclosure have been described with reference to the drawings. Needless to say, the present disclosure is not limited to such examples. It is obvious that those skilled in the art can conceive of various modification examples, correction examples, substitution examples, addition examples, deletion examples, and equivalent examples within the scope described in the claims, and it is naturally understood that those also belong to the technical scope of the present disclosure. In addition, within the scope not departing from the gist of the invention, the respective components in the above-described embodiments may be arbitrarily combined.

Industrial Applicability

[0158] The present disclosure is useful as a detection device, a detection method, and a detection system.

Explanation of Signs

[0159] 1 Detection system 2 Warehouse 10 Detection device 11, 22 Processor 12, 23 Memory 13 Communication interface 14 Input device 15 Display device 16 Shelf number table 17 Shelf size information 18 Captured image 19 Filling rate information 20 Sensor 21 Warehouse Management System 24 Location Data 25 Package Information

Claims

1. A communication interface connected to a sensor that acquires an image including depth information corresponding to the distance from a reference position to a shelf, and a processor that detects a shelf frame indicating the outer shape of the shelf from the image from the sensor and specifies the center of gravity position of the shelf, wherein the processor detects the shelf when the center of gravity position exists within a first area of the image, and after detecting a first shelf, when detecting the center of gravity position of a second shelf adjacent to the first shelf in the first area in response to movement of the sensor, detects the second shelf. A detection device.

2. After detecting the first shelf, the processor detects the second shelf when the center of gravity position of the first shelf is not detected in the first area and the center of gravity position of the second shelf is detected in the first area in response to movement of the sensor. The detection device according to claim 1.

3. The range of the first area in the image is determined based on the movement speed of the sensor, the frame rate of the sensor, and the size of the shelf. The detection device according to claim 1.

4. For each image acquired by the sensor, the processor detects a shelf frame of a shelf from the image, and when the shelf frame can be detected, specifies the center of gravity position of the shelf. The detection device according to claim 1.

5. The processor specifies the center of gravity position of the shelf in each of a plurality of images including the shelf repeatedly acquired by the sensor, and selects, as a best shot image for recording, an image among the plurality of images in which the center of gravity position of the shelf is closest to the center of the image. The detection device according to claim 1.

6. Based on the best shot image, the processor calculates a filling rate indicating a ratio of the volume of the load stored in the shelf to the volume occupied by the load stored in the shelf included in the best shot image. The detection device according to claim 5.

7. The communication interface communicates with a warehouse management system that stores luggage information regarding luggage stored in a warehouse, and the processor acquires the luggage information from the warehouse management system via the communication interface, and based on the luggage information and the filling rate, causes a display device to display a shelf recommended for storing luggage scheduled to be carried in. The detection device according to claim 6.

8. The processor causes the display device to display the best shot image and the filling rate calculated based on the best shot image. The detection device according to claim 6.

9. The sensor acquires a series of images including a plurality of shelves arranged continuously, the processor selects the best shot image of each of the plurality of shelves using the series of images, calculates the filling rate of each of the plurality of shelves based on the best shot image of each of the plurality of shelves, and causes the display device to display the best shot image and the filling rate of each of the plurality of shelves, The detection device according to claim 6.

10. When detecting each of the plurality of shelves, the processor causes the display device to display a detection status indicating how many of the plurality of shelves have been detected. The detection device according to claim 9.

11. The processor causes the display device to display the filling rate of the shelf in a range from 0 percent to 100 percent. The detection device according to claim 6.

12. When the processor cannot detect the shelf, it causes the display device to display an error message. The detection device according to claim 1.

13. The processor detects the first shelf and the second shelf in an order based on a detection order that is the order of detection in the first region of the center of gravity position, detection in a second region adjacent to the first region of the center of gravity position, and non-detection in the first region and the second region of the center of gravity position. The detection device according to claim 1.

14. The processor determines the moving direction of the sensor based on the detection order. The detection device according to claim 13.

15. A detection method executed by a detection device communicably connected to a sensor that acquires an image including depth information corresponding to the distance from a reference position to a shelf, detecting a shelf frame indicating the outer shape of the shelf from the image and specifying the center of gravity position of the shelf, detecting the shelf when the center of gravity position exists within a first region of the image, after detecting a first shelf, detecting a second shelf when detecting the center of gravity position of a second shelf adjacent to the first shelf in the first region in response to movement of the sensor. Detection method.

16. A communication interface connected to a sensor that acquires an image including depth information corresponding to the distance from a reference position to a shelf, a processor that detects a shelf frame indicating the outer shape of the shelf from the image from the sensor and specifies the center of gravity position of the shelf, a warehouse management system that stores luggage information regarding luggage stored in the warehouse, a display device, and the processor is Specify the centroid position of the shelf in each of the plurality of images including the shelf acquired by the sensor, Select, as the best shot image, the image among the plurality of images in which the centroid position of the shelf is closest to the center of the image, Calculate a filling rate indicating the ratio of the volume of the load stored in the shelf to the volume of the shelf included in the best shot image, Acquire the load information from the warehouse management system via the communication interface, Based on the load information and the filling rate, cause the display device to display the shelf recommended for storing the load scheduled to be carried in, Detection system.

Images