Measurement device, measurement method, and measurement program

Abstract

Even if the entire shelf does not fit within the field of view of the sensor measuring the shelf, the system can still measure the degree to which the shelf is filled with goods. [Solution] The measurement method involves receiving an image of the shelf and a depth image indicating the distance to the shelf from a sensor that moves along the shelf where the luggage is stored and photographs the shelf. If a predetermined marker placed on the shelf is detected from the image, the position of the detected marker is used as a reference point to set a measurement range in the depth image, and the occupancy rate of the luggage within the set measurement range is measured using the depth image.

Description

Technical Field

[0004] , , ,

[0001] The present disclosure relates to a measuring device, a measuring method, and a measurement program.

Background Art

[0002] In Patent Document 1, a computer has a processor that executes a program and a storage device that stores the program. The storage device holds shelf shape data, shelf area data representing the area occupied by the shelf, and luggage area data representing the area in the shelf where luggage can be stored. The processor receives the input of the shape data inside the warehouse measured by the measurement sensor, collates the shelf shape data with the shape data inside the warehouse, identifies the position of the shelf inside the warehouse, extracts the shelf shape data from the shape data inside the warehouse based on the identified position of the shelf and the shelf area data, and extracts the shape data of the luggage area inside the shelf from the shelf shape data based on the luggage area data. A measurement system is disclosed. Also, in Patent Document 1, a measurement sensor is mounted on a cart, and while moving the cart, a laser beam is irradiated in a specified direction to measure the distance to an object such as a shelf or luggage, and then the entire shape data of the warehouse is created by connecting the measurement data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, when the distance between the measurement sensor mounted on the cart and the shelf is narrow, the entire shelf does not fit within the angular field of view of the measurement sensor, so it is difficult to extract the shelf shape data from the shape data inside the warehouse measured by the measurement sensor.

[0005] Therefore, the purpose of this disclosure is to provide a technology that can measure the state of goods filling a shelf even when the entire shelf does not fit within the field of view of the sensor that measures the shelf. [Means for solving the problem]

[0006] This disclosure provides a measuring device comprising: a communication unit that receives an image of the shelf and a depth image indicating the distance to the shelf from a sensor that photographs the shelf while moving along the shelf on which luggage is stored; and a processor, wherein the processor, when it detects a predetermined marker placed on the shelf from the image, sets a measurement range in the depth image with the position of the detected marker as a reference point, and measures the filling rate of the luggage in the set measurement range using the depth image.

[0007] This disclosure provides a measurement method that receives an image of the shelf and a depth image indicating the distance to the shelf from a sensor that moves along the shelf on which luggage is stored and photographs the shelf, and when a predetermined marker placed on the shelf is detected from the image, a measurement range is set in the depth image with the position of the detected marker as a reference point, and the filling rate of the luggage in the set measurement range is measured using the depth image.

[0008] This disclosure provides a measurement program that causes an information processing device to receive an image of the shelf and a depth image indicating the distance to the shelf from a sensor that moves along the shelf on which luggage is stored and photographs the shelf, and when it detects a predetermined marker placed on the shelf from the image, it sets a measurement range in the depth image with the position of the detected marker as a reference point, and uses the depth image to measure the occupancy rate of the luggage within the set measurement range.

[0009] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0010] According to this disclosure, even if the shape of the shelf does not fit within the field of view of the sensor measuring the shelf, the state of the shelves being filled with goods can be measured. [Brief explanation of the drawing]

[0011] [Figure 1] A schematic diagram illustrating the operation of the measuring device according to Embodiment 1 in which it performs a three-dimensional scan of shelves in a warehouse and the goods stored on those shelves. [Figure 2] Block diagram showing an example configuration of the measurement system according to Embodiment 1. [Figure 3] A side view of the shelf, luggage, and measuring device according to Embodiment 1. [Figure 4] A front view of the shelf and luggage according to Embodiment 1. [Figure 5] A diagram showing an example of a shelf according to Embodiment 2. [Figure 6] This figure shows an example of setting a divided area 50 on a shelf according to Embodiment 2 and placing markers on the shelf. [Figure 7] Diagram illustrating the divided area information according to Embodiment 2 [Figure 8] A flowchart showing an example of the processing of the measuring device according to Embodiment 2. [Figure 9] Diagram illustrating the marker detection range according to Embodiment 2. [Figure 10] A diagram illustrating the process of generating a composite image according to Embodiment 2. [Figure 11] A diagram illustrating the method for calculating the overall filling rate of the shelf according to Embodiment 2. [Figure 12] This figure shows an example of the measuring device according to Embodiment 2 moving in a meandering manner. [Figure 13]Figure showing an example of a sensor image when the measuring device according to Embodiment 2 moves in a serpentine manner [Figure 14] Figure for explaining coordinate transformation of a sensor image when the measuring device according to Embodiment 2 moves in a serpentine manner

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, detailed descriptions that are more than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] Even if the function of one configuration shown in this embodiment is realized by two or more physical configurations, or the functions of two or more configurations are realized by, for example, one physical configuration, it does not matter.

[0014] (Embodiment 1) <Measurement System> FIG. 1 is a schematic diagram for explaining an overview of an operation in which a measuring device 20 according to Embodiment 1 three-dimensionally scans a shelf 1 in a warehouse and a package 9 stored on the shelf 1. FIG. 2 is a block diagram showing a configuration example of the measuring device 20 according to Embodiment 1. FIG. 3 is a view of the shelf 1, the package 9, and the measuring device 20 according to Embodiment 1 as seen from the side. FIG. 4 is a view of the shelf 1 and the package 9 according to Embodiment 1 as seen from the front. Next, the configuration and operation of the measuring device 20 according to Embodiment 1 will be described with reference to FIGS. 1 to 4. For convenience of explanation, as shown in FIGS. 1, 3, and 4, the width direction of the shelf 1 is taken as the X-axis, the depth direction of the shelf 1 is taken as the Y-axis, and the height direction of the shelf 1 is taken as the Z-axis.

[0015] <Measuring Device> The measuring device 20 is a device for measuring the occupancy rate of goods 9 in a shelf 1 within a warehouse. The occupancy rate is a value that indicates the ratio of the volume of goods 9 actually stored in the shelf 1 to the volume of the space in the shelf 1 that can accommodate goods 9 (hereinafter referred to as the storable space). Note that the occupancy rate does not have to be an exact value, and may be an approximate or estimated value. Also, goods 9 may be read as objects.

[0016] As shown in Figures 1 and 2, the measuring device 20 comprises at least one sensor 21 (21A, 21B) and an information processing device 22. The sensor 21 is connected to the information processing device 22 via a predetermined electrical cable. An example of the electrical cable is a USB cable. However, the sensor 21 may also be connected to the information processing device 22 by wireless communication.

[0017] Sensor 21 includes an RGB sensor that captures an object and generates an RGB image, and a depth sensor that measures the distance (depth) to the object and generates a depth image. The RGB image has color information for each pixel. The RGB image may also be read as a color image or captured image. The depth image has depth (distance) information for each pixel. Hereinafter, the RGB image and the depth image will be collectively referred to as sensor image 100. The imaging sensor may be, for example, a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD) sensor. The depth sensor may be, for example, a Time of Flight (ToF) sensor, a Light Detection and Ranging (LiDAR) sensor, or a stereo camera.

[0018] As shown in Figure 2, the information processing device 22 includes a processor 23, memory 24, communication device 25, device connection device 26, input device 27, and output device 28.

[0019] The processor 23 works in cooperation with the memory 24 to execute a computer program (measurement program), thereby realizing the functions of the measuring device 20. Details of the functions of the measuring device 20 will be explained as appropriate.

[0020] Memory 24 stores computer programs and data for realizing the functions of the measuring device 20. Memory 24 may be composed of a volatile storage medium (e.g., RAM) and / or a non-volatile storage medium (e.g., ROM, flash memory, Solid State Drive (SSD), etc.).

[0021] The communication device 25 controls the transmission and reception of information with other devices or servers, etc., via a communication network (not shown).

[0022] The device connection device 26 is connected to the sensor 21 and controls the transmission and reception of information with the sensor 21.

[0023] The input device 27 is a device that receives input from the user, and is, for example, a touch panel, keyboard, mouse, microphone, etc.

[0024] The output device 28 is a device that outputs information, such as a display, speaker, lamp, etc.

[0025] Next, the operation of the measuring device 20 will be explained.

[0026] As shown in Figure 1, the measuring device 20 moves along an aisle in the warehouse, and the sensor 21 captures images of multiple shelves 1 lined up along the aisle, generating a sensor image 100. The measuring device 20 may be moved manually by a person, or it may move automatically (autonomously).

[0027] For example, as shown in Figures 3 and 4, if shelf 1 has a two-tiered configuration with a lower tier 2A and an upper tier 2B, the measuring device 20 includes a sensor 21A capable of imaging the lower tier 2A and a sensor 21B capable of imaging the upper tier 2B. It is not essential to image each shelf tier with one sensor; for example, both the lower tier 2A and the upper tier 2B may be imaged with a single sensor 21. Furthermore, although this embodiment describes an example where shelf 1 has two overlapping tiers, there may be any number of overlapping shelves 1.

[0028] Sensor 21A transmits the sensor image 100A, which captures the lower section 2A, to the information processing device 22. Sensor 21B transmits the sensor image 100B, which captures the upper section 2B, to the information processing device 22.

[0029] The processor 23 of the information processing device 22 detects the shelf frame 3A (opening) of the lower shelf 2A from the sensor image 100A and detects the luggage 9 that is located within the area enclosed by the shelf frame 3A of the lower shelf 2A (hereinafter referred to as the measurement range 101A). The processor 23 then calculates the ratio of the volume of the detected luggage 9 to the volume of the storable space in the measurement range 101A and defines this as the filling rate of the shelf frame 3A of the lower shelf 2A (measurement range 101A). The processor 23 may calculate the filling rate by the method described in Patent Document 2. For example, by the method described in Patent Document 2, the processor 23 identifies the frontmost depth of the luggage 9 from the depth image and calculates the volume of the luggage 9 by assuming that the luggage 9 is arranged from the identified frontmost depth of the luggage 9 to the deepest depth of the storable space of the shelf frame 3A. Note that depth may be read as distance or position. Similarly, the processor 23 detects the shelf frame 3B (opening) of the upper 2B from the sensor image 100B and detects packages within the area enclosed by the shelf frame 3B of the upper 2B (hereinafter referred to as the measurement range 101B). The processor 23 then calculates the ratio of the volume of the detected packages 9 to the volume of the storable space in the measurement range 101B and uses this as the filling rate for the shelf frame 3B of the upper 2B (measurement range 101B).

[0030] (Embodiment 2) Figure 5 shows an example of shelf 1 according to Embodiment 2.

[0031] Embodiment 1 describes the case where the entire shelf 1 fits within the field of view of the sensor 21. However, there are cases where the shelf 1 does not fit within the field of view of the sensor 21, such as when the entire shelf 1 is horizontally elongated, as shown in Figure 5, or when the width of the aisle is narrow and sufficient distance cannot be secured between the sensor 21 and the shelf 1. Embodiment 2 provides a technology that enables measurement of the occupancy rate of the goods 9 on the shelf 1 even when the entire shelf 1 does not fit within the field of view of the sensor 21. Note that the measuring device 20 according to Embodiment 2 has the same configuration as in Figure 2, so its description is omitted.

[0032] <Method for calculating the occupancy rate when the entire shelf does not fit within the frame> Next, we will explain how to calculate the filling rate when the entire shelf 1 does not fit within the field of view of sensor 21.

[0033] Figure 6 shows an example in which a divided area 50 is set on shelf 1 according to Embodiment 2, and a marker 51 is placed on shelf 1.

[0034] In Embodiment 2, as shown in Figure 6, the shelf 1 is divided and multiple divided areas 50 (50A, 50B, 50C) are set. The size of one divided area 50 (for example, the width of the divided area) should be such that it fits within the field of view of the sensor 21. The size of the divided areas 50 may also be set arbitrarily by the user.

[0035] Multiple markers 51 (51A, 51B, 51C) are placed in predetermined positions on shelf 1 (e.g., shelf board 62), each corresponding to one of the multiple divided areas 50 (50A, 50B, 50C). The shelf number corresponding to the divided area 50 is printed on the marker 51. For example, as shown in Figure 6, marker 51A with shelf number "A1" is placed in divided area 50A, marker 51B with shelf number "A2" is placed in divided area 50B, and marker 51C with shelf number "A3" is placed in divided area 50C. The markers 51 may have strings of characters printed on them in this manner. However, the markers 51 only need to have information printed on them that can identify the divided area 50. For example, the markers 51 are not limited to strings of characters, but may have sequences of numbers, sequences of symbols, barcodes, or two-dimensional codes printed on them.

[0036] Figure 7 is a diagram illustrating the divided area information according to Embodiment 2.

[0037] The relationship between the shelf number and the partitioned area 50 may be predetermined as partitioned area information and stored in memory 24.

[0038] For example, as shown in Figure 7, the divided area information is pre-set with size information (X1, X2, Z1, Z2, Z3, Z4) for the divided area 50A relative to the XZ coordinates of marker 51A for shelf number "A1". In the following explanation, marker 51A for shelf number "A1" may be referred to as marker "A1", marker 51B for shelf number "A2" as marker "A2", and marker 51C for shelf number "A3" as marker "A3".

[0039] Here, X1 represents the distance from the 2D coordinate 90 (XZ coordinate) of marker "A1" to the right edge of the shelf frame within the divided area 50A (hereinafter referred to as virtual shelf frames 52A and 52B), and X2 represents the distance from the 2D coordinate 90 of marker "A1" to the left edge of the virtual shelf frames 52A and 52B within the divided area 50A.

[0040] Z1 indicates the distance from the 2D coordinate 90 of marker "A1" to the top edge of the virtual shelf frame 52B within the divided area 50A, and Z2 indicates the distance from the 2D coordinate 90 of marker "A1" to the bottom edge of the virtual shelf frame 52B within the divided area 50A.

[0041] Z3 indicates the distance from the 2D coordinate 90 of marker "A1" to the top edge of the virtual shelf frame 52A within the divided area 50A, and Z4 indicates the distance from the 2D coordinate 90 of marker "A1" to the bottom edge of the virtual shelf frame 52A within the divided area 50A.

[0042] Similarly, the divided area information includes pre-set size information for the virtual shelf frame 52 within divided area 50B relative to the 2D coordinate 90 of marker "A2", and pre-set size information for the virtual shelf frame 52 within divided area 50C relative to the 2D coordinate 90 of marker "A3".

[0043] As shown in Embodiment 1 and Figure 5, the measuring device 20 moves approximately parallel to the front of the shelf 1, while orienting the sensor 21 toward the shelf 1 so that the measuring surface of the sensor 21 is approximately parallel to the front of the shelf 1. As a result, the sensor 21 captures multiple sensor images 100 while moving.

[0044] Next, the measuring device 20 extracts from the multiple sensor images 100 (RGB images) captured the sensor image 100 of the divided area 50A containing marker "A1", the sensor image 100 of the divided area 50B containing marker "A2", and the sensor image 100 of the divided area 50C containing marker "A3".

[0045] Next, the measuring device 20 uses the sensor images 100 (depth images) of each divided area 50A, 50B, and 50C to calculate the filling rate of the virtual shelf frame 52 in each divided area 50A, 50B, and 50C (hereinafter referred to as the virtual filling rate), and then calculates the overall filling rate of the shelf 1 by averaging the calculated virtual filling rates.

[0046] Furthermore, the measuring device 20 generates and displays an overall sensor image of shelf 1 (hereinafter referred to as the composite image) by combining the sensor images 100 (RGB images) of each divided area 50A, 50B, and 50C.

[0047] The process described above will be explained in detail below.

[0048] <Flowchart> Figure 8 is a flowchart showing an example of the processing of the measuring device 20 according to Embodiment 2.

[0049] The processor 23 of the measuring device 20 determines whether the entire shelf 1 is within the field of view of the sensor 21 (S100).

[0050] If the entire shelf 1 is within the field of view of the sensor 21 (S100: YES), the processor 23 calculates the occupancy rate of shelf 1 using the method described in Embodiment 1 (S130). Then, this process is terminated.

[0051] If the entire shelf 1 does not fit within the field of view of sensor 21 (S100: NO), processor 23 proceeds to the next step S101.

[0052] The processor 23 acquires a sensor image 100, which includes an RGB image and a depth image (S101).

[0053] The processor 23 performs character recognition processing on the acquired RGB image and detects those recognized as strings as character recognition candidates (S102).

[0054] The processor 23 extracts the shelf number (marker 51) from the character recognition candidates detected in step S102 (S103).

[0055] For example, if there is a rule for labeling shelf numbers, the processor 23 extracts from the character recognition candidates those that conform to that rule as shelf numbers. An example of a rule for labeling shelf numbers is as follows: • The number of characters is limited. • The first character is specified. • The order of numbers and letters is specified (for example, if only numbers increase or decrease, such as "A1" and "A2", the first character must be a letter, and subsequent characters must be numbers).

[0056] The processor 23 identifies the 2D coordinate 90 (pixel coordinate) in the RGB image of the shelf number (marker 51) extracted in step S103 (S104).

[0057] Here, the two-dimensional coordinate 90 in the RGB image of shelf number (marker 51) may be a predetermined pixel coordinate within the rectangular area recognized as shelf number (marker 51) (for example, the coordinate of one of the vertices at the four corners of the rectangular area, or the coordinate of the center point of the rectangular area). This specified two-dimensional coordinate 90 may be interpreted as a reference point.

[0058] The processor 23 transfers the two-dimensional coordinates 90 of the shelf number (marker 51) identified in step S104 to the depth image and identifies the three-dimensional coordinates of the shelf number (marker 51) relative to the sensor 21 (S105).

[0059] The processor 23 refers to the partition area information held in memory 24 and obtains the size information of the partition area 50 corresponding to the detected shelf number (S106).

[0060] The processor 23 sets a virtual shelf frame 52 for the depth image based on the size information acquired in step S106 (S107).

[0061] The processor 23 calculates the virtual filling rate of the virtual shelf frame 52 set in step S107 (S108). The virtual shelf frame 52 may be read as the "measurement range" of the virtual filling rate. The method for calculating the virtual filling rate is the same as the method for calculating the filling rate of the shelf frame 3 described in Embodiment 1.

[0062] The processor 23 determines whether it has finished measuring the virtual fill rate for all shelf numbers on shelf 1 (S109). For example, the processor 23 determines that the measurement is complete when it receives a measurement completion instruction from the user. Alternatively, the processor 23 detects the last pre-set shelf number and determines that the measurement is complete when it has finished measuring the virtual fill rate for that shelf number.

[0063] If the processor 23 determines that the measurement is not yet complete (S109: NO), it returns to step S101 and calculates the virtual filling rate of the virtual shelf frame 52 for the remaining shelf numbers.

[0064] If the processor 23 determines that the measurement is complete (S109: YES), it calculates the average value of the virtual filling rate of all virtual shelf frames 52 of shelf 1 and sets this as the overall filling rate of shelf 1 (S110). Note that the overall filling rate of shelf 1 may be calculated using a method different from the average value of all virtual filling rates of shelf 1.

[0065] The processor 23 trims and combines the RGB images of the divided areas 50 corresponding to each shelf number to generate a composite image of the entire shelf 1 (S111). Then, this process ends.

[0066] Through the above process, even if the entire shelf 1 does not fit within the field of view of the sensor 21, the overall occupancy rate of shelf 1 can be calculated. Furthermore, even if the entire shelf 1 does not fit within the field of view of the sensor 21, a composite image capturing the entire shelf 1 can be generated.

[0067] <Setting the marker detection range> Figure 9 is a diagram illustrating the marker detection range 150 according to Embodiment 2.

[0068] Markers 51 are often arranged on shelf 1 according to a certain rule. For example, markers 51 are placed on shelf board 62 at regular intervals, as shown in Figure 9.

[0069] Therefore, the measuring device 20 may set a marker detection range 150 for the RGB image based on the placement rules of the markers 51, and limit the range in which character recognition processing is performed in step S102 of Figure 8 to the set marker detection range 150.

[0070] For example, if there is a rule that marker 51 is placed on shelf 62 above shelf 1B as shown in Figure 9, a marker detection range 150 is set for the RGB image to cover a predetermined area where shelf 62 above shelf 1B is captured, and this range is stored in memory 24 beforehand. Then, in step S102 of Figure 8, the processor 23 limits the area in which character recognition processing is performed to within the marker detection range 150 set for the RGB image. This prevents the extraction of characters other than the shelf number written on marker 51 as character recognition candidates. As a result, in step S103 of Figure 8, it is possible to prevent the extraction of an incorrect character as the shelf number from the character recognition candidates. Note that the detection range 150 may be set horizontally, or as a combination of height and horizontal.

[0071] Alternatively, since markers 51 are often placed at regular intervals on the same shelf board 62, the marker detection range 150 may be set based on the height of the first extracted marker 51. For example, when the processor 23 first detects marker "A1", it sets a predetermined range corresponding to the height of that marker "A1" as the marker detection range 150. Then, from the next time onward, the processor 23 limits the range in which it performs character recognition processing to within this set marker detection range 150. This prevents the extraction of characters other than the shelf number written on the marker 51 as character recognition candidates, as described above.

[0072] Furthermore, if the processor 23 detects multiple markers 51 in the sensor image 100 in step S103, it may preferentially select the marker 51 that was previously detected (i.e., appeared earlier in time). Alternatively, if the processor 23 detects multiple markers 51 in the sensor image 100, it may select all of those markers 51. Then, the processor 23 may perform the processing from step S104 onward for the selected markers 51.

[0073] <Processing for generating composite images> Figure 10 is a diagram illustrating the composite image generation process according to Embodiment 2.

[0074] As described above, the processor 23 of the measuring device 20 captures sensor images 100 for each of the divided areas 50 while moving, and combines the sensor images 100 captured for each divided area 50 to generate a composite image that captures the entire shelf 1. For example, based on the size information associated with the shelf number in the divided area information, the processor 23 crops the sensor image 100 captured for the divided area 50 corresponding to that shelf number so that it includes the virtual shelf frame 52. Then, the processor 23 generates a composite image by arranging the sensor images cropped for each divided area 50 horizontally and combining them.

[0075] The measuring device 20 continuously captures sensor images 100 while moving, thus obtaining multiple sensor images 100 of the same shelf number. The processor 23 may select from these multiple sensor images 100 of the same shelf number to be used for setting the virtual shelf frame 52, calculating the virtual filling rate, and creating a composite image. The measuring device 20 (processor 23) according to Embodiment 2 may make this selection by either (Method B1) or (Method B2) below.

[0076] (Method B1) The processor 23 selects from among multiple sensor images 100 in which the same shelf number is captured the one in which the position of the shelf number is closest to the center of the sensor image 100 (i.e., closest to the center of the field of view).

[0077] For example, in Figure 10, if sensor 21 has sensor images 100 taken from position P1, sensor images 100 taken from position P2, and sensor images 100 taken from position P3, all containing the same shelf number "A1", then processor 23 will select the sensor image 100 taken from position P2, where the position of shelf number "A1" is closest to the center E of the sensor image 100, and then select the image with the same shelf number " Select sensor image 100 for "A1".

[0078] For example, in Figure 10, if sensor 21 has sensor images 100 taken from position P4, sensor images 100 taken from position P5, and sensor images 100 taken from position P6, all of which include the same shelf number "A2", the processor 23 selects the sensor image 100 taken from position P4, where the position of shelf number "A2" is closest to the center of the sensor image 100, as the sensor image for shelf number "A2".

[0079] This suppresses blind spots caused by luggage and shelf frames within the virtual shelf frame 52 set in the sensor image 100, allowing the processor 23 to accurately calculate the virtual filling rate of the virtual shelf frame 52.

[0080] (Method B2) The processor 23 selects from among multiple sensor images 100 taken of the same shelf number the one in which the position of sensor 21 is closest to the overall center C of shelf 1.

[0081] For example, in Figure 10, if sensor 21 has sensor images 100 taken from position P1, sensor images 100 taken from position P2, and sensor images 100 taken from position P3, all of which include the same shelf number "A1", the processor 23 selects the sensor image 100 taken from position P3 of sensor 21, which is closest to the overall center C of shelf 1, as the sensor image for shelf number "A1".

[0082] For example, in Figure 10, if sensor 21 has sensor images 100 taken from position P4, sensor images 100 taken from position P5, and sensor images 100 taken from position P6, all of which include the same shelf number "A2", the processor 23 selects the sensor image 100 taken from position P4 of sensor 21, which is closest to the overall center C of shelf 1, as the sensor image for shelf number "A2".

[0083] As a result, when the sensor image 100 of shelf number "A1" and the sensor image 100 of shelf number "A2" are combined, a composite image with minimal overall distortion of shelf 1 can be generated.

[0084] <Infill density calculation process after image synthesis> Figure 11 is a diagram illustrating the method for calculating the overall filling rate of the shelf according to Embodiment 2.

[0085] As described above, the processor 23 calculates the virtual occupancy rate of each virtual shelf frame 52 included in the entire shelf 1. Next, the processor 23 calculates the average of the multiple virtual occupancy rates and uses this as the overall occupancy rate of shelf 1.

[0086] The processor 23 may also calculate the overall state of the contents of shelf 1 by the following process.

[0087] (S201) The processor 23 sets a first threshold and a second threshold for the virtual occupancy rate of each virtual shelf frame 52. For example, the first threshold may be set to "50%" and the second threshold to "80%". Note that this threshold setting is just an example, and the number of thresholds to be set and the values ​​of the thresholds are arbitrary.

[0088] (S202) The processor 23 calculates the virtual occupancy rate of the virtual shelf frame 52 and determines whether the virtual occupancy rate is (a) less than the first threshold, (b) greater than or equal to the first threshold and less than the second threshold, or (c) greater than or equal to the second threshold.

[0089] (S203) The processor 23 sets the status of the virtual shelf frame 52 to "empty" if the virtual filling rate is (a) less than the first threshold, (b) greater than or equal to the first threshold and less than the second threshold, to "slightly empty," and (c) greater than or equal to the second threshold, to "full."

[0090] (S204) As shown in Figure 11, the processor 23 displays information indicating the occupancy status for each virtual shelf frame 52 in the composite image. For example, the processor 23 displays "○" for virtual shelf frames 52 that are "empty", "△" for virtual shelf frames 52 that are "slightly empty", and "×" for virtual shelf frames 52 that are "not empty". This allows the user to easily recognize the empty parts in the entire shelf frame 3.

[0091] (S205) The processor 23 counts and displays the number of "empty" slots, "slightly empty" slots, and "not empty" slots for the entire shelf 1. This allows the user to easily recognize the number of virtual shelf slots 52 in the entire shelf 1 that can now accommodate new items 9.

[0092] The processor 23 may also calculate the filling rate using only the virtual shelf frames 52 that are in a "vacant" state. For example, as shown in shelf 1A of Figure 11, if the virtual filling rates of the three virtual shelf frames 52 that are in a "vacant" state are "43%", "0%", and "0%", respectively, the processor 23 calculates the average of these virtual filling rates, "14.3%", as the filling rate for the "vacant" state. The user can then use this "vacant" state filling rate and the number of "vacant" virtual shelf frames to determine whether a large item can be stored in shelf frame 3.

[0093] <How to set up virtual shelf frames when the measuring device moves in a meandering manner> Figure 12 shows an example of the case when the measuring device 20 according to Embodiment 2 moves in a meandering manner. Figure 13 shows an example of the sensor image 100 when the measuring device 20 according to Embodiment 2 moves in a meandering manner. Figure 14 is a diagram for explaining the coordinate transformation of the sensor image 100 when the measuring device 20 according to Embodiment 2 moves in a meandering manner.

[0094] As shown in Figure 12, if the measuring device 20 moves in a meandering manner rather than parallel to the front of the shelf 1, a sensor image 100 (RGB image and depth image) of the shelf 1 is captured with distortion in the perspective direction, as shown in Figure 13. If this distorted sensor image 100 is used as is, the virtual shelf frame 52 will also be distorted in the perspective direction, reducing the accuracy of the calculated virtual filling rate. Therefore, in this embodiment, if the measuring device 20 moves in a meandering manner and a sensor image 100 distorted in the perspective direction is captured, the coordinate system of the sensor 21 is corrected in the following way to set the virtual shelf frame 52.

[0095] (S301) The processor 23 detects the marker 51 "Shelf_A1" from the RGB image, as shown in Figure 13.

[0096] (S302) The processor 23 calculates the height h1 from the left end of the detected marker 51 to the virtual center line M of the shelf 62, and the height h2 from the right end of the detected marker 51 to the virtual center line M of the shelf 62.

[0097] (S303) As shown in Figure 14, the processor 23 calculates the depth (distance) d1 at the left edge of the marker 51 "Shelf_A1" detected in step S301 from the depth image.

[0098] (S304) Since h1:h2=d2:d1, the processor 23 uses the heights h1 and h2 calculated in step S302 and the depth d1 calculated in step S303 to calculate the depth d2 (=d1×h1 / h2) at the right edge of the marker 51 "Shelf_A1" detected in step S301, as shown in Figure 14. Alternatively, the processor 23 may calculate the depth (distance) d2 at the right edge of the marker 51 "Shelf_A1" detected in step S301 from the depth image, as shown in Figure 14.

[0099] (S305) The processor 23 calculates the tilt angle θ of shelf 1 relative to sensor 21 based on the calculated d1 and d2 and the known width w (not shown) of marker 51 "Shelf_A1". For example, the processor 23 calculates θ = arcsin((d1-d2) / w).

[0100] (S306) The processor 23 rotates the coordinate system of the sensor 21 by θ and sets a virtual shelf frame 52 for the depth image in the coordinate system rotated by θ.

[0101] This allows the virtual shelf frame 52 to be set appropriately even if the measuring device 20 moves in a meandering manner. Therefore, a decrease in the accuracy of the virtual filling rate of the virtual shelf frame 52 can be suppressed.

[0102] (Summary of this disclosure) The following technologies are disclosed in accordance with the above description in this disclosure.

[0103] <Technology 1> The measuring device (20) according to this disclosure includes a communication unit (e.g., a communication device 25) that receives an image of the shelf and a depth image indicating the distance to the shelf from a sensor (21) that moves along the shelf (1) in which the luggage (9) is stored and photographs the shelf, and a processor (23). When the processor detects a predetermined marker (51) placed on the shelf from the image, it sets a measurement range in the depth image with the position of the detected marker as a reference point, and uses the depth image to measure the filling rate of the luggage in the set measurement range. This allows the measuring device to measure the occupancy rate of items on a shelf even if the entire shelf does not fit within the sensor's field of view.

[0104] <Technology 2> In the measuring device described in Technical 1, the marker is a string of characters, and the processor detects the marker by detecting the string of characters from the captured image, and sets a predetermined position in the region of the detected string of characters as the reference point. This allows the measuring device to detect the string of characters and identify the reference point.

[0105] <Technology 3> In the measuring device described in Technical 2, the area of ​​the detected string is rectangular, and the reference point is one of the four corners of the rectangle. This allows the measuring device to detect the string of characters and identify the reference point.

[0106] <Technology 4> The measuring device described in Technical 1 further includes a memory that stores size information relating the marker and the measurement range, and the processor sets the measurement range, which is associated with the detected marker by the size information, in the depth image, using the position of the detected marker as the reference point. This allows the measuring device to set the measurement range associated with the marker.

[0107] <Technology 5> In the measuring device described in any one of the technologies 1 to 4, when the processor detects a plurality of markers from the captured image, it selects a previously detected marker and sets the position of the selected marker as the reference point. This allows the measuring device to appropriately identify the reference point even when multiple markers are detected.

[0108] <Technology 6> In the measuring device described in any one of the technologies 1 to 5, a plurality of markers are arranged on the shelf, and the processor synthesizes a plurality of captured images in which each of the plurality of markers is detected to generate a composite image of the shelf. This allows the measuring device to generate a composite image of the shelf based on the multiple markers it has detected.

[0109] <Technology 7> In the measuring device described in Technical 6, the processor sets the reference point using the image in which the position of the marker is closest to the center of the image, from among the plurality of images in which the same marker is detected. This allows the measuring device to obtain images that suppress blind spots caused by luggage or shelf frames.

[0110] <Technology 8> In the measuring device described in Technical 6, the processor uses the image in which the position of the marker is closest to the center of the shelf from among the plurality of images in which the same marker is detected to generate a composite image of the shelf. This allows the measuring device to generate a composite image with minimal distortion.

[0111] <Technology 9> In the measuring device described in any one of the technologies 1 to 8, the processor calculates the filling rate for each measurement range corresponding to each marker on the shelf and counts the number of measurement ranges in which the filling rate is less than a predetermined threshold. This allows the measuring device to count the number of items in a measurement range with a relatively small filling rate.

[0112] <Technology 10> The measurement method according to this disclosure receives an image of the shelf and a depth image indicating the distance to the shelf from a sensor (21) that moves along the shelf (1) in which the luggage (9) is stored and photographs the shelf. If a predetermined marker (51) placed on the shelf is detected from the image, a measurement range is set in the depth image using the position of the detected marker as a reference point, and the filling rate of the luggage in the set measurement range is measured using the depth image. This measurement method allows for the measurement of the shelf's occupancy rate even when the entire shelf does not fit within the sensor's field of view.

[0113] <Technology 11> The measurement program according to this disclosure receives an image of the shelf and a depth image indicating the distance to the shelf from a sensor (21) that moves along the shelf (1) in which the luggage (9) is stored and takes a picture of the shelf. If a predetermined marker (51) placed on the shelf is detected from the image, the program causes an information processing device (22) to set a measurement range in the depth image using the position of the detected marker as a reference point, and to measure the occupancy rate of the luggage within the set measurement range using the depth image. By running this measurement program, it is possible to measure the occupancy rate of items on a shelf even if the entire shelf does not fit within the sensor's field of view.

[0114] While embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above can be combined in any way without departing from the spirit of the invention. [Industrial applicability]

[0115] The technology disclosed herein is useful for measuring the occupancy status of items stored on shelves. [Explanation of Symbols]

[0116] 1,1A,1B shelf 3,3A,3B Shelf frame 9 Luggage 20 Measuring devices 21, 21A, 21B sensors 22 Information Processing Devices 23 processors 24 memory 25 Communication equipment 26. Equipment connection device 27 Input devices 28 Output device 50, 50A, 50B, 50C divided areas 51, 51A, 51B, 51C Markers 52, 52A, 52B, 92 Virtual shelf frames 62 shelves 90 2D coordinates 100, 100A, 100B Sensor Images 101A, 101B Measurement range 150 Marker detection range

Claims

1. A communication unit receives an image of the shelf and a depth image indicating the distance to the shelf from a sensor that moves along the shelf where the luggage is stored and photographs the shelf, Equipped with a processor, The aforementioned processor, If a predetermined marker placed on the shelf is detected from the captured image, the measurement range is set in the depth image using the position of the detected marker as a reference point. Using the depth image, the filling rate of the cargo within the set measurement range is measured. Measuring device.

2. The aforementioned marker is a string of characters, The aforementioned processor, The marker is detected by detecting the string from the captured image. The predetermined position of the detected string region is used as the reference point. The measuring device according to claim 1.

3. The region of the detected string is rectangular. The aforementioned reference point is one of the four corners of the rectangle. The measuring device according to claim 2.

4. The system further includes a memory that stores size information relating the marker and the measurement range. The processor sets the measurement range in the depth image, which is associated with the detected marker by the size information, using the position of the detected marker as the reference point. The measuring device according to claim 1.

5. When the processor detects multiple markers from the captured image, it selects a previously detected marker and sets the position of the selected marker as the reference point. The measuring device according to claim 1.

6. Multiple markers are placed on the aforementioned shelf. The processor synthesizes multiple captured images in which each of the multiple markers is detected, and generates a composite image of the shelf. The measuring device according to claim 1.

7. The processor sets the reference point using the image in which the position of the marker is closest to the center of the image, from among the plurality of images in which the same marker is detected. The measuring device according to claim 6.

8. The processor uses, among the multiple captured images in which the same marker is detected, the image in which the position of the marker is closest to the center of the shelf to generate a composite image of the shelf. The measuring device according to claim 6.

9. The aforementioned processor, The filling rate is calculated for each measurement range corresponding to each marker on the shelf, The number of measurement ranges in which the filling rate is less than a predetermined threshold is counted. The measuring device according to claim 1.

10. A sensor that moves along the shelves where luggage is stored and photographs the shelves receives an image of the shelves and a depth image indicating the distance to the shelves. If a predetermined marker placed on the shelf is detected from the captured image, the measurement range is set in the depth image using the position of the detected marker as a reference point. Using the depth image, the filling rate of the cargo within the set measurement range is measured. Measurement method.

11. A sensor that moves along the shelves where luggage is stored and photographs the shelves receives an image of the shelves and a depth image indicating the distance to the shelves. If a predetermined marker placed on the shelf is detected from the captured image, the measurement range is set in the depth image using the position of the detected marker as a reference point. Using the depth image, the filling rate of the cargo within the set measurement range is measured. A measurement program that causes an information processing device to perform a specific action.

Images