Filling rate measuring method, information processing device, and program

Abstract

To provide a filling rate measuring method and the like capable of calculating the filling rate of an object to be measured.SOLUTION: The filling rate measuring method includes acquiring a spatial three-dimensional model obtained by measuring a first storage section having a first storage space in which an object to be measured is stored from an opening and an opening / closing section having a plurality of through-holes and covering the opening, via the plurality of through-holes by a distance measurement sensor (S111), acquiring a storage three-dimensional model of the first storage section in the state in which the object to be measured is not stored (S112), and estimating a three-dimensional model of the object to be measured using the spatial three-dimensional model and the storage three-dimensional model, and calculating a first filling rate of the object to be measured for the first storage space using the storage three-dimensional model and the three-dimensional model of the object to be measured (S116), in which the shape of a hidden second portion is estimated based on the shape of a first portion of the object to be measured measured by the distance measurement sensor, and the three-dimensional model of the object to be measured is estimated using the first portion and the second portion and the storage three-dimensional model.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present disclosure relates to a filling rate measurement method, an information processing device, and a program. [Background technology]

[0002] Patent Document 1 discloses a three-dimensional shape measuring device that acquires a three-dimensional shape using a three-dimensional laser scanner. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-87319 Summary of the Invention [Problem to be solved by the invention]

[0004] There has been insufficient consideration of applications of the measured three-dimensional shape, such as the calculation of the filling rate, which indicates how much of the measurement target object is stored in a given storage space.

[0005] The present disclosure provides a filling rate measurement method and the like that can calculate the filling rate of a measurement object. [Means for solving the problem]

[0006] A filling rate measurement method according to one aspect of the present disclosure includes: acquiring a spatial three-dimensional model obtained by measuring a first storage space into which a measurement object is stored through an opening, a first storage section having a plurality of through holes and an opening / closing section arranged to cover the opening, using a ranging sensor facing the first storage section through the plurality of through holes; acquiring a stored three-dimensional model of the first storage section when the measurement object is not stored; estimating an object three-dimensional model of the measurement object in the first storage space using the acquired spatial three-dimensional model and the stored three-dimensional model; calculating a first filling rate of the measurement object for the first storage space using the stored three-dimensional model and the object three-dimensional model; and estimating a shape of a second part of the measurement object that is hidden from the ranging sensor based on the shape of a first part of the measurement object measured by the ranging sensor through the plurality of through holes in a direction from the ranging sensor toward the measurement object; and estimating the object three-dimensional model using the first part, the second part, and the stored three-dimensional model.

[0007] An information processing device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire a spatial three-dimensional model obtained by measuring a first storage section having a first storage space into which a measurement object is stored through an opening, the first storage section having a plurality of through-holes and an opening / closing section that is arranged to cover the opening in a closed state, through the plurality of through-holes by a distance measuring sensor facing the first storage section, acquire a stored three-dimensional model of the first storage section when the measurement object is not stored, and perform a preprocessing using the acquired spatial three-dimensional model and the stored three-dimensional model. A three-dimensional model of the object to be measured in the first storage space is estimated, and a first filling rate of the object to be measured for the first storage space is calculated using the stored three-dimensional model and the object three-dimensional model.In the estimation, the shape of a second part of the object to be measured that is hidden from the ranging sensor is estimated based on the shape of a first part of the object to be measured that is measured by the ranging sensor through the multiple through holes in a direction from the ranging sensor toward the object to be measured, and the object three-dimensional model is estimated using the first part, the second part, and the stored three-dimensional model.

[0008] A filling rate measurement method according to one aspect of the present disclosure includes acquiring a spatial three-dimensional model obtained by measuring a first storage section having a first storage space in which a measurement object is stored and having an opening formed therein using a ranging sensor facing the first storage section through the opening, acquiring a stored three-dimensional model that is a three-dimensional model of the first storage section in which the measurement object is not stored, using the acquired spatial three-dimensional model and the stored three-dimensional model to extract an object portion of the spatial three-dimensional model that is a portion that corresponds to the measurement object, using the extracted object portion to estimate an object three-dimensional model that is a three-dimensional model of the measurement object in the first storage space, and calculating a first filling rate of the measurement object relative to the first storage space using the stored three-dimensional model and the object three-dimensional model.

[0009] An information processing device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire a spatial three-dimensional model obtained by measuring a first storage section having a first storage space in which a measurement object is stored and having an opening formed therein from the first direction side through the opening using a ranging sensor facing the first storage section, acquire a stored three-dimensional model which is a three-dimensional model of the first storage section in which the measurement object is not stored, use the acquired spatial three-dimensional model and the stored three-dimensional model to extract an object portion of the stored three-dimensional model which is a portion corresponding to the measurement object, use the extracted object portion to estimate an object three-dimensional model which is a three-dimensional model of the measurement object in the first storage space, and calculate a first filling rate of the measurement object in the first storage space using the stored three-dimensional model and the object three-dimensional model.

[0010] The present disclosure may be realized as a program that causes a computer to execute the steps included in the filling rate measurement method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network, such as the Internet. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide a filling rate measurement method and the like that can calculate the filling rate of a measurement object. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining an outline of a filling rate measurement method according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a characteristic configuration of the three-dimensional measurement system according to the embodiment. [Figure 3]FIG. 3 is a diagram illustrating a first example of the configuration of the distance measurement sensor. [Figure 4] FIG. 4 is a diagram illustrating a second example of the configuration of the distance measuring sensor. [Figure 5] FIG. 5 is a diagram illustrating a third example of the configuration of the distance measuring sensor. [Figure 6] FIG. 6 is a block diagram showing the configuration of the coordinate system calculation unit of the first example. [Figure 7] FIG. 7 is a diagram for explaining a method for calculating a measurement coordinate system by the coordinate system calculation unit of the first example. [Figure 8] FIG. 8 is a block diagram showing the configuration of the coordinate system calculation unit of the second example. [Figure 9] FIG. 9 is a diagram for explaining a method for calculating a measurement coordinate system by a coordinate system calculation unit of the second example. [Figure 10] FIG. 10 is a block diagram showing the configuration of the coordinate system calculation unit of the third example. [Figure 11] FIG. 11 is a diagram for explaining a method for calculating a measurement coordinate system by a coordinate system calculation unit of the third example. [Figure 12] FIG. 12 is a block diagram illustrating an example of the configuration of the model generating unit. [Figure 13] FIG. 13 is a flowchart of a process for calculating the volume of the storage space by the model generation unit. [Figure 14] FIG. 14 is a block diagram illustrating an example of the configuration of the filling rate calculation unit. [Figure 15] FIG. 15 is a diagram for explaining an example of a method for calculating the filling rate by the filling rate calculation unit. [Figure 16] FIG. 16 is a diagram for explaining another example of a method for calculating the filling rate by the filling rate calculation unit. [Figure 17] FIG. 17 is a flowchart of a filling rate measurement method performed by an information processing device. [Figure 18] FIG. 18 is a flowchart of a process for calculating a measurement coordinate system by the coordinate system calculation unit of the first example. [Figure 19]FIG. 19 is a flowchart of a process for calculating a measurement coordinate system by the coordinate system calculation unit of the second example. [Figure 20] FIG. 20 is a flowchart of a process for calculating a measurement coordinate system by the coordinate system calculation unit of the third example. [Figure 21] FIG. 21 is a diagram for explaining a method for calculating the filling rate. [Figure 22] FIG. 22 is a block diagram illustrating an example of the configuration of a calculation unit of a filling rate calculation unit according to the first modification. [Figure 23] FIG. 23 is a flowchart of a filling rate calculation process of the calculation unit of the filling rate calculation unit according to the first modification. [Figure 24] FIG. 24 is a diagram showing an example in which two or more shelves are stored in a storage space such as the bed of a truck. [Figure 25] FIG. 25 is a table showing the relationship between the shelves stored in the storage space of the loading platform and their filling rates. [Figure 26] FIG. 26 is a block diagram illustrating an example of the configuration of a calculation unit of a filling rate calculation unit according to the second modification. [Figure 27] FIG. 27 is a flowchart of a filling rate calculation process of the calculation unit of the filling rate calculation unit according to the second modification. [Figure 28] FIG. 28 is a diagram for explaining the configuration of a car bogie according to the third modification. [Figure 29] FIG. 29 is a block diagram showing an example of the configuration of a filling rate calculation unit according to the third modification. [Figure 30] FIG. 30 is a flowchart of a filling rate calculation process of the filling rate calculation unit according to the third modification. [Figure 31] FIG. 31 is a diagram illustrating an example of a second method for calculating the filling rate. [Figure 32] FIG. 32 is a diagram for explaining another example of the second method for calculating the filling rate. [Figure 33] FIG. 33 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fourth modification. [Figure 34]FIG. 34 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fifth modification. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Background to this disclosure) In logistics and distribution sites, there is a demand for measuring the filling rate of measurement objects, such as luggage, relative to the storage space, thereby improving the utilization efficiency of the storage space. Also, in logistics and distribution sites, since measurement objects are stored in storage areas such as many containers, there is a demand for measuring a large number of filling rates in a short period of time. However, a method for easily measuring filling rates has not been fully explored.

[0014] Therefore, the present disclosure provides a filling rate measurement method, etc., for easily calculating the filling rate of more storage sections in a short period of time by applying a technique for generating a three-dimensional model of the storage section in which the object to be measured is stored.

[0015] A filling rate measurement method according to one aspect of the present disclosure includes acquiring a spatial three-dimensional model obtained by measuring a first storage section having a first storage space in which a measurement object is stored and having an opening formed therein using a ranging sensor facing the first storage section through the opening, acquiring a stored three-dimensional model that is a three-dimensional model of the first storage section in which the measurement object is not stored, using the acquired spatial three-dimensional model and the stored three-dimensional model to extract an object portion of the spatial three-dimensional model that is a portion that corresponds to the measurement object, using the extracted object portion to estimate an object three-dimensional model that is a three-dimensional model of the measurement object in the first storage space, and calculating a first filling rate of the measurement object relative to the first storage space using the stored three-dimensional model and the object three-dimensional model.

[0016] According to this, the object 3D model of the measurement object is estimated using a spatial 3D model measured in the first storage unit in a state in which the measurement object is stored and an object portion extracted using a stored 3D model of the first storage unit in which the measurement object is not stored. This makes it possible to easily calculate the first filling rate of the measurement object in the first storage space by simply measuring the first storage unit in a state in which the measurement object is stored.

[0017] In addition, in the estimation, the three-dimensional model of the object may be estimated based on a first three-dimensional coordinate system based on a shape of a part of the first storage section.

[0018] This reduces the amount of processing required to estimate the three-dimensional model of the object.

[0019] Furthermore, the first three-dimensional coordinate system may be calculated based on only the shape of the part of the first storage section.

[0020] Therefore, the shape of only a part of the first storage section, which is easy to extract from the image, can be used to calculate the first three-dimensional coordinate system, thereby improving the processing speed for estimating the three-dimensional model of the object and the calculation accuracy of the first three-dimensional coordinate system.

[0021] The shape of the part may be the same as the shape of the opening.

[0022] Therefore, a coordinate system based on the shape of the opening can be easily calculated, and a three-dimensional model of the object can be estimated based on the calculated coordinate system.

[0023] In addition, in the estimation, the three-dimensional model of the object may be estimated based on a first three-dimensional coordinate system based on the position of a marker set in the first storage unit.

[0024] Therefore, a coordinate system based on the marker can be easily calculated, and a three-dimensional model of the object can be estimated based on the calculated coordinate system.

[0025] In addition, the estimation may involve estimating the shape of a second part of the ranging sensor that does not face the object to be measured based on the shape of a first part of the ranging sensor that faces the object to be measured in a direction from the ranging sensor toward the object to be measured, thereby estimating the three-dimensional model of the object.

[0026] Therefore, even if there is a shape of the second portion where the distance measuring sensor does not face the measurement object, it is possible to estimate a three-dimensional model of the object.

[0027] Furthermore, the first storage section may further have an opening / closing section having a through hole and arranged to cover the opening in a closed state, the first portion being a portion facing the through hole in the opening / closing section in the closed state in the direction, and the second portion being a portion hidden in the direction by the opening / closing section in the closed state, and the filling rate measurement method may further determine whether the opening / closing section is in an open state or a closed state, and if the opening / closing section is in the open state, estimate the three-dimensional model of the object by performing the extraction and the estimation, and if the opening / closing section is in the closed state, estimate the second portion based on the first portion, and estimate the three-dimensional model of the object using the first portion, the estimated second portion, and the stored three-dimensional model.

[0028] According to this, even when the object to be measured is stored in a first storage section provided with an opening / closing section that opens and closes the opening, the method for estimating the three-dimensional model of the object is switched depending on the open / close state of the opening / closing section, so that the three-dimensional model of the object can be estimated appropriately.

[0029] The direction may also be along the horizontal direction.

[0030] Therefore, there is no need to adjust the position of the distance measuring sensor so that it can measure from a direction where there is no opening or closing part with a through hole, which gives the distance measuring sensor a high degree of freedom in installation. Therefore, even if the position of the distance measuring sensor cannot be completely adjusted, measurement results for estimating a three-dimensional model of the object can be obtained using the distance measuring sensor.

[0031] In addition, in the calculation, the ratio of the volume of the measurement object stored in the first storage space to the volume of the space in the first storage space that can store the measurement object may be calculated as the first filling rate.

[0032] Therefore, it is possible to calculate the first filling rate for appropriately determining how many measurement objects can be stored in the available space in the first storage space.

[0033] In addition, the first storage section and the additional first storage section may be stored in a second storage space of a second storage section, and the filling rate measurement method may further calculate a second filling rate of the first storage section and the additional first storage section relative to the second storage space.

[0034] Therefore, it is possible to appropriately calculate the second filling rate when one or more first storage sections are stored in the second storage space.

[0035] The stored three-dimensional model may also be a three-dimensional model measured by the distance measurement sensor and an additional distance measurement sensor.

[0036] This makes it possible to generate a stored three-dimensional model with little occlusion.

[0037] The distance measurement sensor has at least two cameras for generating the three-dimensional spatial model, and is fixed on the upper side of the first storage section.

[0038] In this way, when the ranging sensor is fixed to the top of the first storage section, the objects captured by the two cameras of the ranging sensor are limited to the ground or the base (bottom surface) of the first storage section, and there is nothing that can move other than the first storage section, making it easy to separate the object being measured from the background.

[0039] An information processing device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire a spatial three-dimensional model obtained by measuring a first storage section having a first storage space in which a measurement object is stored and having an opening formed therein from the first direction side through the opening using a ranging sensor facing the first storage section, acquire a stored three-dimensional model which is a three-dimensional model of the first storage section in which the measurement object is not stored, use the acquired spatial three-dimensional model and the stored three-dimensional model to extract an object portion of the stored three-dimensional model which is a portion corresponding to the measurement object, use the extracted object portion to estimate an object three-dimensional model which is a three-dimensional model of the measurement object in the first storage space, and calculate a first filling rate of the measurement object in the first storage space using the stored three-dimensional model and the object three-dimensional model.

[0040] According to this, the object 3D model of the measurement object is estimated using a spatial 3D model measured in the first storage unit in a state in which the measurement object is stored and an object portion extracted using a stored 3D model of the first storage unit in which the measurement object is not stored. This makes it possible to easily calculate the first filling rate of the measurement object in the first storage space by simply measuring the first storage unit in a state in which the measurement object is stored.

[0041] The present disclosure may be realized as a program that causes a computer to execute the steps included in the filling rate measurement method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network, such as the Internet.

[0042] Hereinafter, each embodiment of the filling rate measurement method and the like according to the present disclosure will be described in detail with reference to the drawings. Note that each embodiment described below represents a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, arrangement and connection of the components, steps, order of steps, and the like shown in each of the following embodiments are merely examples and are not intended to limit the present disclosure.

[0043] Furthermore, each drawing is a schematic diagram and is not necessarily an exact illustration. In addition, in each drawing, substantially the same components are denoted by the same reference numerals, and duplicated explanations may be omitted or simplified.

[0044] (Embodiment) An outline of a filling rate measurement method according to an embodiment will be described with reference to FIG.

[0045] FIG. 1 is a diagram for explaining an outline of a filling rate measurement method according to an embodiment.

[0046] In the filling rate measurement method, as shown in Fig. 1, luggage 103 stored on a shelf 102 having a storage space 101 is measured using a distance measurement sensor 210. Then, the obtained measurement results are used to calculate the filling rate of the luggage 103 in the storage space 101. An opening 102a is formed in the shelf 102 for allowing the luggage 103 to be placed in and removed from the storage space 101. The distance measurement sensor 210 is disposed in a position facing the opening 102a of the shelf 102 in an orientation such that it measures the shelf 102 including the opening 102a, and measures a measurement region R1 including the interior of the storage space 101 through the opening 102a.

[0047] The shelf 102 has a box-like shape, for example, as shown in FIG. 1 . The shelf does not have to have a box-like shape as long as it has a placement surface on which the luggage 103 is placed and a storage space 101 above the placement surface in which the luggage 103 is stored. The shelf 102 is an example of a first storage section. The storage space 101 is an example of a first storage space. While the storage space 101 has been described as an internal space of the shelf 102, it is not limited to this and may be a space within a warehouse in which measurement objects such as luggage 103 are stored. The luggage 103 is an example of a measurement object. The measurement object is not limited to luggage 103 and may be a commodity. In other words, the measurement object may be any object as long as it is portable.

[0048] Fig. 2 is a block diagram showing a characteristic configuration of a three-dimensional measurement system according to an embodiment. Fig. 3 is a diagram for explaining a first example of the configuration of a distance measurement sensor. Fig. 4 is a diagram for explaining a second example of the configuration of a distance measurement sensor. Fig. 5 is a diagram for explaining a third example of the configuration of a distance measurement sensor.

[0049] 2, the three-dimensional measurement system 200 includes a distance measurement sensor 210 and an information processing device 220. The three-dimensional measurement system 200 may include multiple distance measurement sensors 210, or may include a single distance measurement sensor 210.

[0050] The distance measurement sensor 210 measures a three-dimensional space including the storage space 101 of the shelf 102 through the opening 102a of the shelf 102 to obtain measurement results including the shelf 102 and the storage space 101 of the shelf 102. Specifically, the distance measurement sensor 210 generates a spatial three-dimensional model represented by a collection of three-dimensional points indicating the three-dimensional positions of each of multiple measurement points on the shelf 102 or the baggage 103 (hereinafter referred to as the measurement target) (the surface of the measurement target). A collection of three-dimensional points is called a three-dimensional point cloud. The three-dimensional position indicated by each three-dimensional point in the three-dimensional point cloud is represented by three-value information consisting of, for example, an X component, a Y component, and a Z component in a three-dimensional coordinate space consisting of X, Y, and Z axes. Note that the three-dimensional model may include not only three-dimensional coordinates but also color information indicating the color of each point or shape information indicating the surface shape of each point and its surroundings. The color information may be represented, for example, in an RGB color space or another color space such as HSV, HLS, or YUV.

[0051] A specific example of the distance measurement sensor 210 will be described with reference to FIGS.

[0052] As shown in FIG. 3 , the distance measurement sensor 210 of the first example generates a three-dimensional spatial model by emitting electromagnetic waves and acquiring waves reflected by the measurement target. Specifically, the distance measurement sensor 210 measures the time it takes for the emitted electromagnetic waves to be reflected by the measurement target and return to the distance measurement sensor 210, and calculates the distance between the distance measurement sensor 210 and point P1 on the surface of the measurement target using the measured time and the wavelength of the electromagnetic waves used for the measurement. The distance measurement sensor 210 emits electromagnetic waves in multiple radial directions determined in advance from a reference point of the distance measurement sensor 210. For example, the distance measurement sensor 210 may emit electromagnetic waves at first angular intervals around the horizontal direction and at second angular intervals around the vertical direction. Therefore, the distance measurement sensor 210 can calculate the three-dimensional coordinates of multiple points on the measurement target by detecting the distance between the distance measurement sensor 210 and the measurement target in each of multiple directions around the distance measurement sensor 210. Therefore, the distance measurement sensor 210 can calculate position information indicating multiple three-dimensional positions on the measurement target and generate a spatial three-dimensional model having the position information. The position information may be a three-dimensional point cloud including multiple three-dimensional points indicating multiple three-dimensional positions.

[0053] 3, the distance measuring sensor 210 of the first example is a three-dimensional laser measuring instrument having a laser emitting unit 211 that emits laser light as electromagnetic waves and a laser receiving unit 212 that receives light reflected from the measurement object of the measurement. The distance measuring sensor 210 scans the measurement object with laser light by rotating or oscillating a unit including the laser emitting unit 211 and the laser receiving unit 212 on two different axes, or by installing a movable mirror (MEMS (Micro Electro Mechanical Systems) mirror) that oscillates on two axes on the path of the emitting or receiving laser. In this way, the distance measuring sensor 210 can generate a high-precision and high-density three-dimensional model of the measurement object of the measurement.

[0054] The distance measurement sensor 210 is exemplified as a three-dimensional laser measuring instrument that measures the distance to the measurement object by irradiating laser light, but is not limited to this and may also be a millimeter wave radar measuring instrument that measures the distance to the measurement object by emitting millimeter waves.

[0055] Furthermore, the ranging sensor 210 may generate a three-dimensional model having color information. The first color information is color information generated using an image captured by the ranging sensor 210, and indicates the color of each of the first three-dimensional points included in the first three-dimensional point cloud.

[0056] Specifically, the ranging sensor 210 may have a built-in camera that captures an image of a measurement target around the ranging sensor 210. The built-in camera in the ranging sensor 210 generates an image by capturing an image of an area including the irradiation range of the laser light emitted by the ranging sensor 210. The imaging range captured by the camera is associated in advance with the irradiation range. Specifically, multiple directions in which the laser light is emitted by the ranging sensor 210 are associated in advance with each pixel in the image captured by the camera, and the ranging sensor 210 sets pixel values ​​of the image associated with the directions of the multiple 3D points as color information indicating the colors of the respective 3D points included in the 3D point cloud.

[0057] The second example of the distance measuring sensor 210A is a distance measuring sensor using a structured light method, as shown in FIG. 4 . The distance measuring sensor 210A includes an infrared pattern irradiator 211A and an infrared camera 212A. The infrared pattern irradiator 211A projects a predetermined infrared pattern 213A onto the surface of the measurement target. The infrared camera 212A captures an infrared image of the measurement target on which the infrared pattern 213A is projected. The distance measuring sensor 210A searches for the infrared pattern 213A included in the acquired infrared image and calculates the distance from the infrared pattern irradiator 211A or the infrared camera 212A to the position of point P1 on the measurement target based on a triangle formed by connecting the three positions of the position of point P1 of the infrared pattern on the measurement target in real space, the position of the infrared pattern irradiator 211A, and the position of the infrared camera 212A. This allows the distance measuring sensor 210A to obtain a three-dimensional measurement point on the measurement target.

[0058] In addition, the distance measurement sensor 210A can acquire a high-density three-dimensional model by moving the unit of the distance measurement sensor 210A, which has the infrared pattern irradiation unit 211A and the infrared camera 212A, or by making the infrared pattern irradiated by the infrared pattern irradiation unit 211A into a fine texture.

[0059] Furthermore, distance measuring sensor 210A may generate a three-dimensional model having color information by using the visible light region of color information that can be acquired by infrared camera 212A and associating the obtained visible light region with three-dimensional points in consideration of the position or orientation of infrared pattern irradiator 211A or infrared camera 212A. Furthermore, distance measuring sensor 210A may be configured to further include a visible light camera for adding color information.

[0060] Distance measurement sensor 210B of the third example is a distance measurement sensor that measures three-dimensional points using stereo camera measurement, as shown in FIG. 5. Distance measurement sensor 210B is a stereo camera having two cameras 211B and 212B. Distance measurement sensor 210B acquires stereo images with parallax by capturing images of the measurement target using the two cameras 211B and 212B at synchronized timing. Distance measurement sensor 210B uses the acquired stereo images (two images) to perform a feature point matching process between the two images and acquires alignment information between the two images with pixel accuracy or subpixel accuracy. Distance measurement sensor 210B calculates the distance from one of the two cameras 211B and 212B to the matching position (i.e., point P1) on the measurement target based on a triangle formed by connecting the three positions of point P1 on the measurement target in real space and the positions of the two cameras 211B and 212B. This allows the distance measurement sensor 210B to acquire a three-dimensional point of the measurement point on the measurement target.

[0061] In addition, distance measurement sensor 210B can obtain a highly accurate three-dimensional model by moving the unit of distance measurement sensor 210B, which has two cameras 211B and 212B, or by increasing the number of cameras mounted on distance measurement sensor 210B to three or more, photographing the same measurement object, and performing matching processing.

[0062] Furthermore, by using visible light cameras as the cameras 211B and 212B of the distance measurement sensor 210B, it is possible to easily add color information to the acquired three-dimensional model.

[0063] In this embodiment, the information processing device 220 will be described as an example having the distance measurement sensor 210 of the first example, but it may also be configured to have the distance measurement sensor 210A of the second example or the distance measurement sensor 210B of the third example instead of the distance measurement sensor 210 of the first example.

[0064] The two cameras 211B and 212B can capture monochrome images including visible light images or infrared images. In this case, the matching process between the two images in the three-dimensional measurement system 200 may be performed using, for example, SLAM (Simultaneous Localization And Mapping) or SfM (Structure from Motion). Furthermore, the point cloud density of the measurement space model may be increased by MVS (Multi View Stereo) using information indicating the positions and orientations of the cameras 211B and 212B obtained by this process.

[0065] Returning to FIG. 2, the configuration of the information processing device 220 will be described.

[0066] The information processing device 220 includes an acquisition unit 221 , a coordinate system calculation unit 222 , a model generation unit 223 , a filling rate calculation unit 224 , and a storage unit 225 .

[0067] The acquisition unit 221 acquires the spatial three-dimensional model and the image generated by the distance measurement sensor 210. Specifically, the acquisition unit 221 may acquire the spatial three-dimensional model and the image from the distance measurement sensor 210. The spatial three-dimensional model and the image acquired by the acquisition unit 221 may be stored in the storage unit 225.

[0068] The coordinate system calculation unit 222 calculates the positional relationship between the distance measuring sensor 210 and the shelf 102 using the spatial three-dimensional model and the image. As a result, the coordinate system calculation unit 222 calculates a measurement coordinate system based on the shape of a portion of the shelf 102. The coordinate system calculation unit 222 may calculate a measurement coordinate system based on only the shape of a portion of the shelf 102. Specifically, the coordinate system calculation unit 222 calculates the measurement coordinate system based on the shape of the opening 102a of the shelf 102 as the part of the shape that serves as the reference for calculating the measurement coordinate system. Note that, when the shape of the opening 102a is rectangular as shown in the embodiment, the shape of the opening 102a that serves as the reference for calculating the measurement coordinate system may be a corner of the shape of the opening 102a or a side of the shape of the opening 102a.

[0069] The measurement coordinate system is a three-dimensional orthogonal coordinate system and is an example of a first three-dimensional coordinate system. By calculating the measurement coordinate system, it is possible to identify the relative position and orientation of the distance measurement sensor 210 with respect to the shelf 102. In other words, this allows the sensor coordinate system of the distance measurement sensor 210 to be aligned with the measurement coordinate system, and calibration between the shelf 102 and the distance measurement sensor 210 can be performed. The sensor coordinate system is a three-dimensional orthogonal coordinate system.

[0070] In this embodiment, the rectangular parallelepiped shelf 102 has the opening 102a on one side of the shelf 102, but this is not limited to this. The shelf may have openings on multiple sides of the rectangular parallelepiped shape, such as openings on two sides, the front and rear, or openings on two sides, the front and top. When the shelf has multiple openings, a predetermined reference position, which will be described later, may be set for one of the multiple openings. The predetermined reference position may be set to a space where no three-dimensional points or voxels exist in a storage three-dimensional model, which is a three-dimensional model of the shelf 102.

[0071] Here, the coordinate system calculation unit 222 of the first example will be described with reference to FIGS.

[0072] Fig. 6 is a block diagram showing the configuration of a coordinate system calculation unit of the first example. Fig. 7 is a diagram for explaining a method of calculating a measurement coordinate system by the coordinate system calculation unit of the first example.

[0073] The coordinate system calculation unit 222 calculates a measurement coordinate system. The measurement coordinate system is a three-dimensional coordinate system that serves as the reference for the three-dimensional spatial model. For example, the distance measurement sensor 210 is installed at the origin of the measurement coordinate system and is installed facing directly toward the opening 102a of the shelf 102. In this case, the measurement coordinate system may be set such that the upward direction of the distance measurement sensor 210 is set as the X axis, the rightward direction is set as the Y axis, and the forward direction is set as the Z axis. The coordinate system calculation unit 222 has an auxiliary unit 301 and a calculation unit 302.

[0074] 7(a), the auxiliary unit 301 sequentially acquires in real time images 2001 that are measurement results by the distance measurement sensor 210 acquired by the acquisition unit 221, and superimposes an adjustment marker 2002 on each of the sequentially acquired images 2001. The auxiliary unit 301 sequentially outputs a superimposed image 2003 in which the adjustment marker 2002 is superimposed on the image 2001 to a display device (not shown). The display device sequentially displays the superimposed image 2003 output by the information processing device 220. Note that the auxiliary unit 301 and the display device may be provided integrally with the distance measurement sensor 210.

[0075] The adjustment marker 2002 is a marker for assisting the user in moving the distance measurement sensor 210 so that the position and orientation of the distance measurement sensor 210 relative to the shelf 102 are specific. While viewing the superimposed image 2003 displayed on the display device, the user can change the position and orientation of the distance measurement sensor 210 so that the adjustment marker 2002 overlaps a specific reference position on the shelf 102, thereby placing the distance measurement sensor 210 at a specific position and orientation relative to the shelf 102. The specific reference position on the shelf 102 is, for example, the position of each of the four corners of the rectangular opening 102a of the shelf 102.

[0076] When the distance measurement sensor 210 is placed at a specific position and orientation relative to the shelf 102, a superimposed image 2003 is generated in which four adjustment markers 2002 are superimposed at four positions corresponding to the positions of the four corners of the opening 102a of the shelf 102. For example, by moving the distance measurement sensor 210 so that the adjustment markers 2002 move in the directions of the arrows shown in (a) of Fig. 7, the user can align the four adjustment markers 2002 with the positions of the four corners of the opening 102a, as shown in (b) of Fig. 7.

[0077] Although the auxiliary unit 301 is described as superimposing the adjustment marker 2002 on the image 2001, it may also be possible to superimpose the adjustment marker on a spatial three-dimensional model and display the spatial three-dimensional model with the adjustment marker superimposed on it on a display device.

[0078] As shown in FIG. 7C, the calculation unit 302 calculates a rotation matrix 2005 and a translation vector 2006 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102 when the four adjustment markers 2002 are aligned with the four corner positions of the opening 102a. The calculation unit 302 calculates a measurement coordinate system 2000 that has an origin at an arbitrary corner (one of the four corners) of the opening 102a by transforming a sensor coordinate system 2004 of the distance measurement sensor 210 using the calculated rotation matrix 2005 and translation vector 2006. Note that when the four adjustment markers 2002 are aligned with the four corner positions of the opening 102a, the user may input to an input device (not shown). The information processing device 220 may determine when the four adjustment markers 2002 are aligned with the four corner positions of the opening 102a by acquiring the time when the input is received from the input device. Furthermore, the information processing device 220 may analyze the image 2001 to determine whether or not the four adjustment markers 2002 are aligned with the positions of the four corners of the opening 102a.

[0079] Next, the coordinate system calculation unit 222A of the second example will be described with reference to FIGS.

[0080] Fig. 8 is a block diagram showing the configuration of a coordinate system calculation unit of the second example. Fig. 9 is a diagram for explaining a method of calculating a measurement coordinate system by the coordinate system calculation unit of the second example.

[0081] The coordinate system calculation unit 222A includes a detection unit 311, an extraction unit 312, and a calculation unit 313.

[0082] The detection unit 311 detects a shelf area 2014 corresponding to the shelf 102, as shown in (c) of FIG. 9, using a spatial three-dimensional model 2011, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221, as shown in (a) of FIG. 9, and a storage three-dimensional model 2012, which is shown in (b) of FIG. 9. The storage three-dimensional model 2012 is a three-dimensional model of the shelf 102 on which no luggage 103 is stored, and is a three-dimensional model generated in advance using the measurement result by the distance measurement sensor 210 for the shelf 102 when no luggage 103 is stored. The storage three-dimensional model 2012 is generated by a model generation unit 223, which will be described later, and stored in the storage unit 225. The storage three-dimensional model 2012 may include position information 2013 indicating the positions of the four corners of the opening 102a of the shelf 102.

[0083] 9(d), the extraction unit 312 uses the position information 2013 in the storage three-dimensional model 2012 to extract four opening end points 2016, which are the positions of the four corners of the opening 2015 in the shelf area 2014. The shape of the opening 2015 defined by the four opening end points 2016 is an example of a partial shape that serves as a reference for calculating the measurement coordinate system.

[0084] 9(e), the calculation unit 313 calculates a rotation matrix 2017 and a translation vector 2018 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102, based on the shapes of the four opening corner points 2016 as seen from the distance measurement sensor 210. The calculation unit 313 calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 of the distance measurement sensor 210 using the rotation matrix 2017 and the translation vector 2018. Specifically, when the rotation matrix 2017 is R and the translation vector 2018 is T, the calculation unit 313 can convert a three-dimensional point x in the sensor coordinate system 2004 into a three-dimensional point X in the measurement coordinate system 2000 using Equation 1 shown below. As a result, the calculation unit 313 can calculate the measurement coordinate system 2000.

[0085] X=Rx+T...Formula 1

[0086] Next, the coordinate system calculation unit 222B of the third example will be described with reference to FIGS.

[0087] Fig. 10 is a block diagram showing the configuration of a coordinate system calculation unit of the third example, and Fig. 11 is a diagram for explaining a method of calculating a measurement coordinate system by the coordinate system calculation unit of the third example.

[0088] The coordinate system calculation unit 222B has a detection unit 321, an extraction unit 322, and a calculation unit 323. In the third example, a marker 104 is placed at a specific position (for example, a position on the top surface) of the shelf 102, and the coordinate system calculation unit 222B specifies the measurement coordinate system 2000 based on the position of the marker 104. That is, the measurement coordinate system 2000 in this case is a coordinate system based on the position of the marker 104 placed on the shelf 102.

[0089] The marker 104 has, for example, a check pattern, but is not limited to a check pattern as long as it is an alignment mark (positioning mark) having a predetermined shape.

[0090] The detection unit 321 detects a marker area 2024 corresponding to the marker 104 installed on the shelf 102, as shown in (c) of Figure 11, from an image 2021, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221, as shown in (a) of Figure 11.

[0091] The extraction unit 322 extracts a pattern contour 2025, which is the contour of the check pattern, from the marker region 2024 on the image 2021, as shown in FIG. 11(d).

[0092] The calculation unit 323 calculates a rotation matrix 2026 and a translation vector 2027 that indicate the positional relationship between the distance measuring sensor 210 and the marker 104, based on the shape of the extracted pattern contour 2025. The calculation unit 323 calculates the three-dimensional positional relationship between the distance measuring sensor 210 and the shelf 102 using the rotation matrix 2026 and the translation vector 2027 and the positional relationship between the stored three-dimensional model 2022 and the marker 2023 shown in FIG. 11(b), and calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 using the calculated three-dimensional positional relationship. Note that the positional relationship between the stored three-dimensional model 2022 and the marker 2023 may be measured in advance, or may be generated in advance based on design data of the shelf 102 on which the marker 104 is arranged.

[0093] Returning to FIG. 2, the model generation unit 223 will be described.

[0094] The model generation unit 223 generates a storage three-dimensional model, which is a three-dimensional model of the shelf 102 on which no luggage 103 is stored. The model generation unit 223 acquires the measurement results of the shelf 102 on which no luggage 103 is stored by the distance measurement sensor 210, and generates the storage three-dimensional model. Specific processing by the model generation unit 223 will be described later. The generated storage three-dimensional model is stored in the memory unit 225.

[0095] Here, the model generation unit 223 will be specifically described with reference to FIGS.

[0096] Fig. 12 is a block diagram showing an example of the configuration of the model generating unit Fig. 13 is a flowchart showing a process of calculating the volume of the storage space by the model generating unit.

[0097] The model generation unit 223 includes a detection unit 401 , a generation unit 402 , and a volume calculation unit 403 .

[0098] The detection unit 401 detects a shelf area corresponding to the shelf 102 from the three-dimensional spatial model measured by the distance measurement sensor 210 (S101). If the three-dimensional measurement system 200 includes multiple distance measurement sensors 210, the detection unit 401 performs the process of step S101 for each of the multiple distance measurement sensors 210. As a result, the detection unit 401 detects multiple shelf areas corresponding to the multiple distance measurement sensors 210, respectively.

[0099] When the three-dimensional measurement system 200 includes multiple ranging sensors 210, the generation unit 402 integrates the multiple shelf areas to generate a three-dimensional storage model (S102). Specifically, in order to integrate the multiple shelf areas, the generation unit 402 may align the three-dimensional point clouds using ICP (Iterative Closest Point), or may calculate the relative positional relationship between the multiple ranging sensors 210 in advance and integrate the multiple shelf areas based on the calculated relative positional relationship. The relative positional relationship may be calculated using SfM (Structure from Motion) with multiple images acquired by the multiple ranging sensors 210 as multi-viewpoint images. The multiple ranging sensors 210 may be installed based on a design drawing in which the relative positional relationship is determined.

[0100] In addition, instead of using multiple ranging sensors 210, a single ranging sensor 210 may be moved and multiple measurement results taken from multiple positions may be used to generate a three-dimensional storage model of the shelf 102 by integrating multiple shelf areas obtained from the multiple measurement results.

[0101] The three-dimensional storage model may be generated based on 3D CAD data at the time of designing the shelf 102, without using the results measured by the ranging sensor 210, or may be generated based on dimensional measurement data of the shelf 102 or equipment specification data published by the manufacturer.

[0102] Note that, when the three-dimensional measurement system 200 does not include multiple distance measurement sensors 210 but includes only one distance measurement sensor 210 and uses one measurement result measured from one position, the model generation unit 223 does not need to include the generation unit 402. In other words, the model generation unit 223 does not need to perform step S102.

[0103] The volume calculation unit 403 calculates the volume of the storage space 101 of the shelf 102 using the storage three-dimensional model (S103).

[0104] Returning to FIG. 2, the filling rate calculation unit 224 will be described.

[0105] The filling rate calculation unit 224 calculates the filling rate of the cargo 103 in the storage space 101 of the shelf 102. The filling rate calculation unit 224 may calculate the ratio of the volume of the cargo 103 to the volume of the storage space 101 as the filling rate, for example, using a three-dimensional spatial model, an image, and a measurement coordinate system 2000 acquired by the distance measurement sensor 210.

[0106] Here, the filling rate calculation unit 224 will be specifically described with reference to FIGS.

[0107] FIG. 14 is a block diagram showing an example of the configuration of a filling rate calculation unit. FIG. 15 is a diagram for explaining an example of a method for calculating a filling rate by the filling rate calculation unit. FIG. 15 shows an example in which the distance measurement sensor 210 faces the opening 102a of the shelf 102. The distance measurement sensor 210 is arranged on the negative Z-axis side where the opening 102a of the shelf 102 is formed, and measures the storage space 101 of the shelf 102 through the opening 102a of the shelf 102. In other words, the distance measurement sensor 210 is arranged above the shelf 102 in the vertical direction. This example is an example in which the measurement coordinate system 2000 is measured by the coordinate system calculation unit 222 of the first example. In other words, in this case, the sensor coordinate system 2004 and the measurement coordinate system 2000 coincide with each other.

[0108] The filling rate calculation unit 224 includes an extraction unit 501 , an estimation unit 502 , and a calculation unit 503 .

[0109] The extraction unit 501 uses the spatial three-dimensional model 2011 and the stored three-dimensional model to extract a luggage region 2033, which is a portion of the spatial three-dimensional model corresponding to the luggage 103. Specifically, the extraction unit 501 converts the data structure of the spatial three-dimensional model 2011, which is the measurement result by the distance measuring sensor 210 acquired by the acquisition unit 221, shown in (a) of Fig. 15 into voxel data, thereby generating voxel data 2031, shown in (b) of Fig. 15. The extraction unit 501 uses the generated voxel data 2031 and a stored three-dimensional model 2032, which is the voxelized stored three-dimensional model shown in (c) of Fig. 15, to subtract the stored three-dimensional model 2032 from the voxel data 2031, thereby extracting a luggage region 2033, which is a region resulting from measurement of the luggage 103, from the voxel data 2031, shown in (d) of Fig. 15. The luggage area 2033 is an example of an object portion that corresponds to the measurement object.

[0110] The estimation unit 502 uses the extracted luggage area 2033 to estimate a luggage model 2034, which is a three-dimensional model of the luggage 103 in the storage space 101. Specifically, the estimation unit 502 uses the luggage area 2033 to interpolate the area where the luggage 103 is hidden from the distance measurement sensor 210 in the Z-axis direction, which is the alignment direction of the distance measurement sensor 210 and the shelf 102, i.e., the luggage area 2033 on the positive side of the Z-axis. For example, for each of the multiple voxels that make up the luggage area 2033, the estimation unit 502 determines whether the voxel is located on the negative side of the Z-axis relative to the farthest voxel that is located furthest on the positive side of the Z-axis among the multiple voxels. When the voxel is located on the negative side of the Z-axis relative to the farthest voxel, and there is no voxel located on the positive side of the Z-axis relative to the voxel, the estimation unit 502 interpolates the voxel to the same position in the Z-axis direction as the farthest voxel. As a result, the estimation unit 502 estimates a baggage model 2034 as shown in FIG. 15(e).

[0111] The calculation unit 503 calculates a first filling rate of the cargo 103 in the storage space 101 using the storage three-dimensional model and the cargo model 2034. Specifically, the calculation unit 503 counts the number of voxels that make up the cargo model 2034 and multiplies the counted number by a predetermined voxel size to calculate the volume of the cargo 103. The calculation unit 503 calculates, as the first filling rate, the ratio of the calculated volume of the cargo 103 to the volume of the storage space 101 of the shelf 102 calculated by the model generation unit 223.

[0112] The distance measurement sensor 210 does not have to face the opening 102a of the shelf 102. FIG. 16 is a diagram for explaining another example of a method for calculating the filling rate by the filling rate calculation unit. FIG. 16 shows an example in which the distance measurement sensor 210 is disposed at an angle with respect to the opening 102a of the shelf 102. This example is an example in which the measurement coordinate system 2000 is measured by the coordinate system calculation unit 222A of the second example or the coordinate system calculation unit 222B of the third example. That is, in this case, the sensor coordinate system 2004 and the measurement coordinate system 2000 are different.

[0113] 16, the coordinate system used is the measurement coordinate system 2000. Using the luggage area 2033, the estimation unit 502 interpolates the luggage area 2033 to the area where the luggage 103 is hidden from the distance measurement sensor 210 in the Z-axis direction of the measurement coordinate system 2000, which is the direction in which the distance measurement sensor 210 and the shelf 102 are aligned, that is, the luggage area 2033 on the positive side of the Z-axis.

[0114] Other processing by the filling rate calculation unit 224 is the same as in the case of FIG. 15, and therefore description thereof will be omitted.

[0115] The spatial three-dimensional model and image pair used in calculating the measurement coordinate system by the coordinate system calculation unit 222 and in calculating the filling rate by the filling rate calculation unit 224 may be the results of measurements taken by the ranging sensor 210 at the same time, or may be the results of measurements taken at different times.

[0116] The ranging sensor 210 and the information processing device 220 may be communicably connected to each other via a communication network. The communication network may be a public communication network such as the Internet, or a private communication network. As a result, the spatial 3D model and image obtained by the ranging sensor 210 are transmitted from the ranging sensor 210 to the information processing device 220 via the communication network.

[0117] Furthermore, the information processing device 220 may acquire the spatial 3D model and images from the ranging sensor 210 without using a communication network. For example, the spatial 3D model and images may be temporarily stored in an external storage device such as a hard disk drive (HDD) or a solid state drive (SSD) from the ranging sensor 210, and the information processing device 220 may acquire the spatial 3D model and images from the external storage device. Furthermore, the external storage device may be a cloud server.

[0118] The information processing device 220 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. The functions of each processing unit of the information processing device 220 may be realized by software or hardware.

[0119] Next, the operation of the information processing device 220 will be described.

[0120] FIG. 17 is a flowchart of a filling rate measurement method performed by an information processing device.

[0121] The information processing device 220 acquires a spatial three-dimensional model from the distance measurement sensor 210 (S111). At this time, the information processing device 220 may further acquire an image of the measurement target from the distance measurement sensor 210.

[0122] The information processing device 220 acquires the stored three-dimensional model stored in the storage unit 225 (S112).

[0123] The information processing device 220 calculates a measurement coordinate system based on the shape of the opening 102a of the shelf 102 (S113). Step S113 is a process performed by the coordinate system calculation unit 222.

[0124] The information processing device 220 uses the voxel data 2031 of the spatial three-dimensional model 2011 and the stored three-dimensional model 2032 of the stored three-dimensional model to extract a baggage area 2033 corresponding to the baggage 103 from the voxel data 2031 (S114). Step S114 is a process performed by the extraction unit 501 of the filling rate calculation unit 224.

[0125] The information processing device 220 uses the extracted luggage area 2033 to estimate a luggage model 2034, which is a three-dimensional model of the luggage 103 in the storage space 101 (S115). Step S115 is a process performed by the estimation unit 502 of the filling rate calculation unit 224.

[0126] The information processing device 220 calculates a first filling rate of the luggage 103 in the storage space 101 using the storage three-dimensional model and the luggage model 2034 (S116). Step S116 is a process performed by the calculation unit 503 of the filling rate calculation unit 224.

[0127] FIG. 18 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the first example.

[0128] The coordinate system calculation unit 222 sequentially acquires in real time images 2001 that are measurement results by the distance measurement sensor 210 acquired by the acquisition unit 221, and superimposes an adjustment marker 2002 on each of the sequentially acquired images 2001 (S121). Step S121 is a process performed by the auxiliary unit 301 of the coordinate system calculation unit 222.

[0129] The coordinate system calculation unit 222 acquires the position and orientation of the distance measurement sensor 210 (S122). Step S121 is a process performed by the auxiliary unit 301 of the coordinate system calculation unit 222.

[0130] The coordinate system calculation unit 222 specifies a sensor coordinate system 2004 of the distance measurement sensor 210 using the position and orientation of the distance measurement sensor 210 when the four adjustment markers 2002 are aligned with the positions of the four corners of the opening 102a, and calculates the measurement coordinate system 2000 using the specified sensor coordinate system 2004 (S123). Step S123 is a process performed by the calculation unit 302 of the coordinate system calculation unit 222.

[0131] FIG. 19 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the second example.

[0132] The coordinate system calculation unit 222A detects a shelf area 2014 corresponding to the shelf 102 using the space three-dimensional model 2011, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221, and the storage three-dimensional model 2012 (S121A). Step S121A is a process performed by the detection unit 311 of the coordinate system calculation unit 222A.

[0133] The coordinate system calculation unit 222A uses the position information 2013 in the storage three-dimensional model 2012 to extract four opening corner points 2016, which are the positions of the four corners of the opening 2015 in the shelf area 2014 (S122A). Step S122A is a process performed by the extraction unit 312 of the coordinate system calculation unit 222A.

[0134] The coordinate system calculation unit 222A calculates a rotation matrix 2017 and a translation vector 2018 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102, based on the shapes of the four opening end points 2016 as seen from the distance measurement sensor 210. Then, the coordinate system calculation unit 222A calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 of the distance measurement sensor 210 using the rotation matrix 2017 and the translation vector 2018 (S123A). Step S123A is a process performed by the calculation unit 313 of the coordinate system calculation unit 222A.

[0135] FIG. 20 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the third example.

[0136] The coordinate system calculation unit 222B detects the marker region 2024 from the image 2021, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221 (S121B). Step S121B is a process performed by the detection unit 321 of the coordinate system calculation unit 222B.

[0137] The coordinate system calculation unit 222B extracts the pattern contour 2025 from the marker region 2024 on the image 2021 (S122B). Step S122B is a process performed by the extraction unit 322 of the coordinate system calculation unit 222B.

[0138] The coordinate system calculation unit 222B calculates a rotation matrix 2026 and a translation vector 2027 that indicate the positional relationship between the distance measurement sensor 210 and the marker 104, based on the shape of the extracted pattern contour 2025. Then, the coordinate system calculation unit 222B calculates the three-dimensional positional relationship between the distance measurement sensor 210 and the shelf 102 using the rotation matrix 2026, the translation vector 2027, and the positional relationship between the storage three-dimensional model 2022 and the marker 2023, and calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 using the calculated three-dimensional positional relationship (S123B). Step S123B is a process performed by the calculation unit 323 of the coordinate system calculation unit 222B.

[0139] The filling rate calculated by the information processing device 220 may be output from the information processing device 220. The filling rate may be displayed on a display device (not shown) provided in the information processing device 220, or may be transmitted to an external device different from the information processing device 220. For example, the calculated filling rate may be output to a luggage conveying system and used to control the luggage conveying system.

[0140] According to the filling rate measurement method of this embodiment, a baggage model 2034 of the baggage 103 is estimated using a baggage area 2033 extracted using a spatial three-dimensional model obtained by measuring the shelf 102 in a state where the baggage 103 is stored and a storage three-dimensional model of the shelf 102 in which no baggage 103 is stored. This makes it possible to easily calculate the first filling rate of the baggage 103 in the storage space 101 simply by measuring the shelf 102 in a state where the baggage 103 is stored.

[0141] Furthermore, in the filling rate measurement method, the baggage model 2034 is estimated based on a three-dimensional coordinate system based on the shape of a part of the shelf 102. This makes it possible to reduce the amount of processing required to estimate the baggage model 2034.

[0142] Furthermore, in the filling rate measurement method, the luggage model 2034 is estimated based on a three-dimensional coordinate system based on the shape of only a portion of the shelf 102. The shape of only a portion of the first storage section, which is easy to extract on an image, can be used to calculate the measurement coordinate system. This improves the processing speed for estimating the luggage model and the calculation accuracy of the measurement coordinate system.

[0143] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system is a three-dimensional Cartesian coordinate system having a Z axis, and the estimation involves interpolating the Z-axis plus direction side, opposite to the Z-axis minus direction, of the luggage area 2033, to estimate the luggage model 2034. This makes it possible to effectively reduce the amount of processing required to estimate the luggage model 2034.

[0144] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system is a coordinate system based on the shape of opening 102a of shelf 102. Therefore, the coordinate system based on the shape of opening 102a of shelf 102 can be easily calculated, and baggage model 2034 can be estimated based on the calculated coordinate system.

[0145] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system is a coordinate system based on markers 104 installed on shelves 102. Therefore, the coordinate system based on markers 104 can be easily calculated, and baggage model 2034 can be estimated based on the calculated coordinate system.

[0146] In the filling rate measurement method, the distance measurement sensor 210B has at least two cameras for generating a three-dimensional space model. The distance measurement sensor 210 including the distance measurement sensor 210B is fixed on the upper side of the first storage section.

[0147] In this way, when the distance measuring sensor 210 is fixed to the upper side of the first storage section, and the first storage section is movable like a car dolly described below, the objects present within the measurement range of the distance measuring sensor 210 are limited to the ground or the base (bottom) of the car dolly, and there is nothing that can move other than the car dolly, so it is easy to separate the measurement object from the background from the measurement results. Note that, if the distance measuring sensor 210 has a camera, the measurement range is the camera's imaging range. On the other hand, when the distance measuring sensor 210 is fixed to a side other than the upper side, moving objects other than the first storage section are more likely to enter the measurement range, making it difficult to separate the measurement object from the background.

[0148] (Variation 1) In the information processing device 220 according to the above embodiment, the ratio of the volume of the luggage 103 stored in the storage space 101 to the capacity of the storage space 101 is calculated as the filling rate, but the present invention is not limited to this.

[0149] FIG. 21 is a diagram for explaining a method for calculating the filling rate.

[0150] In (a) and (b) of Figure 21, the storage space 101 of the shelf 102 has a capacity to store exactly 16 pieces of luggage 103. As shown in (a) of Figure 21, when eight pieces of luggage 103 are arranged with no gaps, eight more pieces of luggage 103 can be stored in the available storage space 101. On the other hand, as shown in (b) of Figure 21, when luggage is arranged with gaps, in order to store eight pieces of luggage 103 in the remaining space of the storage space 101, it is necessary to move the luggage 103 that is already stored. If luggage 103 is stored in the remaining space of the storage space 101 without moving the luggage 103 that is already stored, only six pieces of luggage 103 can be stored.

[0151] 21(a) and 21(b), the amount of cargo 103 that can be stored in the remaining space of storage space 101 is different, but the filling rate is calculated to be the same 50% in both cases. For this reason, it is conceivable to calculate the filling rate in consideration of the space that can actually be stored, in accordance with the shape of the remaining space of storage space 101.

[0152] Fig. 22 is a block diagram showing an example of the configuration of a calculation unit of a filling rate calculation unit according to Modification 1. Fig. 23 is a flowchart of a filling rate calculation process of the calculation unit of a filling rate calculation unit according to Modification 1.

[0153] As shown in FIG. 22, the calculation unit 503 includes a baggage volume calculation unit 601 , an area division unit 602 , an expected baggage measurement unit 603 , an area estimation unit 604 , and a calculation unit 605 .

[0154] The luggage volume calculation unit 601 calculates the luggage volume, which is the volume of the luggage 103, from the luggage model 2034 (S131). The luggage volume calculation unit 601 calculates the volume of the luggage 103 stored in the storage space 101 in the same manner as in the embodiment.

[0155] Next, the area dividing unit 602 divides the storage space 101 of the spatial three-dimensional model 2011 into an occupied area 2041 occupied by the luggage 103 and an empty area 2042 not occupied by the luggage 103 (S132).

[0156] Next, the planned package measurement unit 603 calculates the volume of one package to be stored (S133). When there are multiple types of packages with different shapes and sizes to be stored as shown in (c) of Figure 21, the planned package measurement unit 603 calculates the volume of one package for each type. For example, the planned package measurement unit 603 calculates the volume of package 103a, package 103b, and package 103c.

[0157] Next, the area estimation unit 604 estimates how to place the luggage 103 to be stored in the empty area 2042 so that the maximum number of luggage 103 to be stored can be stored, and estimates the number of luggage 103 to be stored in that case. In other words, the area estimation unit 604 estimates the maximum number of luggage 103 to be stored that can be stored in the empty area 2042. The area estimation unit 604 calculates the storage capacity of the empty area 2042 by multiplying the volume of one piece of luggage by the number of luggage that can be stored (S134).

[0158] When there are multiple types of luggage, the area estimation unit 604 may estimate how many luggage of each type can be stored, or may estimate how many luggage of each type can be stored. When storing multiple types of luggage, the area estimation unit 604 calculates the storable volume of the empty area 2042 by integrating the volume of one piece of luggage for each type by the number of luggage of that type that can be stored. For example, if the area estimation unit 604 estimates that n1 luggage 103a, n2 luggage 103b, and n3 luggage 103c can be stored, the area estimation unit 604 calculates the storable volume of the empty area 2042 by integrating the first volume obtained by multiplying the volume of the luggage 103a by n1, the second volume obtained by multiplying the volume of the luggage 103b by n2, and the third volume obtained by multiplying the volume of the luggage 103c by n3. Note that n1, n2, and n3 are each an integer greater than or equal to 0.

[0159] The calculation unit 605 calculates the filling rate by applying the volume of the stored cargo and the storable volume to the following formula 2 (S135).

[0160] Filling rate (%) = (volume of stored cargo) / (volume of stored cargo + storage capacity) × 100... Equation 2

[0161] In this way, the filling rate calculation unit 224 may calculate the ratio of the volume of the luggage 103 stored in the storage space 101 to the volume of the space in the storage space 101 that can store the luggage 103 as the filling rate.

[0162] This makes it possible to calculate a first filling rate for appropriately determining how much baggage 103 can be stored in the available space in the storage space 101.

[0163] (Variation 2) In the information processing device 220 according to the above embodiment, the filling rate of the luggage 103 in the storage space 101 of one shelf 102 is calculated, but the filling rate of the luggage 103 in the storage space 101 of two or more shelves 102 may also be calculated.

[0164] Fig. 24 is a diagram showing an example of storing two or more shelves in a storage space such as the loading platform of a truck. Fig. 25 is a table showing the relationship between the shelves stored in the storage space in the loading platform and their filling rates.

[0165] As shown in Fig. 24, a loading platform 106 having a storage space 105 stores a plurality of car dollies 112. The loading platform 106 may be, for example, a van-type loading platform of a truck. The loading platform 106 is an example of a second storage section. The second storage section is not limited to the loading platform 106, but may also be a container or a warehouse.

[0166] The storage space 105 is an example of a second storage space. The storage space 105 has a volume large enough to store a plurality of car bogies 112. In the second modification, the storage space 105 can store six car bogies 112. Because the storage space 105 can store a plurality of car bogies 112, the storage space 105 is larger than the storage space 111.

[0167] The car truck 112 has a storage space 111 capable of storing a plurality of luggage 103. The car truck 112 is an example of a first storage section. The storage space 111 is an example of a first storage space. Note that the storage space 105 may store the shelves 102 described in the embodiment.

[0168] The plurality of luggage 103 is not stored directly on the loading platform 106, but is stored on a plurality of car trucks 112. Then, the car trucks 112 storing the plurality of luggage 103 are stored on the loading platform 106.

[0169] In this case, the configuration of the calculation unit 503 of the filling rate calculation unit 224 will be described.

[0170] Fig. 26 is a block diagram showing an example of the configuration of a calculation unit of a filling rate calculation unit according to Modification 2. Fig. 27 is a flowchart of a filling rate calculation process of the calculation unit of a filling rate calculation unit according to Modification 2.

[0171] As shown in FIG. 26, the calculation unit 503 according to the second modification includes an acquisition unit 701, a counting unit 702, and a calculation unit 703.

[0172] The acquisition unit 701 acquires the number of car trucks 112 that can be stored on the loading platform 106 (S141). In the case of the second modification, the maximum number of car trucks 112 that can be stored on the loading platform 106 is six, so six is ​​acquired.

[0173] The counting unit 702 counts the number of car trucks 112 stored on the loading platform 106 (S142). When the car truck 112 shown in FIG. 24 is stored on the loading platform 106, the counting unit 702 counts the number of car trucks 112 as three.

[0174] The calculation unit 703 calculates a second filling rate, which is a filling rate of one or more car carts 112 on the loading platform 106 (S143). Specifically, the calculation unit 703 may calculate, as the second filling rate, the ratio of the number of car carts 112 stored on the loading platform 106 to the maximum number of car carts 112 that can be stored on the loading platform 106. For example, since a maximum of six car carts 112 can be stored on the loading platform 106 and three of these car carts 112 are stored on the loading platform 106, the calculation unit 703 calculates 50% as the second filling rate.

[0175] The calculation unit 703 may calculate the filling rate of the luggage 103 for each of the one or more car carts 112 stored on the loading platform 106, and use the calculated filling rate to calculate the filling rate of the luggage 103 for the second storage space. Specifically, the calculation unit 703 may calculate the average of the filling rates of the luggage 103 for the car carts 112 as the filling rate of the luggage 103 for the second storage space. In this case, when there is surplus space in the storage space 105 of the loading platform 106 that can store the car carts 112, the calculation unit 703 may calculate the average by setting the filling rate of the number of car carts 112 that can be stored in the surplus space that can store the car carts 112 to 0%.

[0176] For example, if the filling rates of the three car carts 112 shown in Figure 25 are 70%, 30%, and 20%, respectively, and a maximum of six car carts 112 can be stored in the loading platform 106, the filling rates of the six car carts 112 may be set to 70%, 30%, 20%, 0%, 0%, and 0%, respectively, and the average obtained may be 20%, which may be calculated as the filling rate of luggage 103 in the second storage space.

[0177] Therefore, the second filling rate when one or more car trucks 112 are stored in the storage space 105 can be calculated appropriately.

[0178] (Variation 3) Next, Modification 3 will be described.

[0179] FIG. 28 is a diagram for explaining the configuration of a car bogie according to the third modification.

[0180] Figure 28(a) is a diagram showing the car carriage 112 with the opening / closing unit 113 in a closed state, and Figure 28(b) is a diagram showing the car carriage 112 with the opening / closing unit 113 in an open state.

[0181] The car bogie 112 according to the third modification has an opening / closing unit 113 that opens and closes the opening 112a. The opening / closing unit 113 is a lattice- or mesh-like cover having a plurality of through-holes 113a. Therefore, even when the opening / closing unit 113 of the car bogie 112 is in a closed state, the distance measurement sensor 210 can measure the three-dimensional shape of the interior of the storage space 111 of the car bogie 112 through the plurality of through-holes 113a and the opening 112a.

[0182] This is because the electromagnetic waves emitted by distance measurement sensor 210 pass through the multiple through holes 113a and openings 112a. Note that in the case of distance measurement sensor 210A as well, the infrared pattern emitted by distance measurement sensor 210A passes through the multiple through holes 113a and openings 112a, so even if opening / closing part 113 of car bogie 112 is in the closed state, the three-dimensional shape of the interior of storage space 111 of car bogie 112 can be measured through the multiple through holes 113a and openings 112a. Also in the case of distance measurement sensor 210B, the two cameras 211B, 212B can photograph the interior of storage space 111 through the multiple through holes 113a and openings 112a, so the three-dimensional shape of the interior of storage space 111 of car bogie 112 can be measured.

[0183] Therefore, the information processing device 220 can determine whether or not luggage 103 is stored in the storage space 111. However, when the opening / closing unit 113 is in the closed state, it is difficult to obtain the correct filling rate unless the method for calculating the filling rate is switched to a different method from when the opening / closing unit 113 is in the open state or when the opening / closing unit 113 is not present. For this reason, the filling rate calculation unit 224 according to the third modification calculates the filling rate using the first method when the opening / closing unit 113 is in the open state, and calculates the filling rate using the second method when the opening / closing unit 113 is in the closed state.

[0184] 29 is a block diagram showing an example of the configuration of a filling rate calculation unit according to Modification 3. FIG. 30 is a flowchart of a filling rate calculation process of the filling rate calculation unit according to Modification 3.

[0185] As shown in FIG. 29, the filling rate calculation unit 224 according to the third modification includes a detection unit 801, a switching unit 802, a first filling rate calculation unit 803, and a second filling rate calculation unit 804.

[0186] The detection unit 801 detects the open / close state of the opening / closing unit 113 using the spatial three-dimensional model (S151). Specifically, the detection unit 801 detects that the opening / closing unit 113 is in the closed state when three-dimensional point clouds exist at positions inside and outside the storage space 111 in the front-to-rear direction of the area of ​​the opening 112a of the car truck 112 (that is, the alignment direction of the distance measurement sensor 210 and the car truck 112) using the spatial three-dimensional model. The detection unit 801 detects that the opening / closing unit 113 is in the open state when three-dimensional point clouds exist only inside the storage space 111.

[0187] The switching unit 802 determines whether the opening / closing unit 113 is in the open state or the closed state (S152), and switches the next process depending on the determination result.

[0188] When the switching unit 802 determines that the opening / closing unit 113 is in the open state (open state in S152), the first filling rate calculation unit 803 calculates the filling rate by a first method (S153). Specifically, the first filling rate calculation unit 803 calculates the filling rate of the car bogie 112 by performing processing similar to the processing by the filling rate calculation unit 224 of the embodiment.

[0189] If the switching unit 802 determines that the open / close unit 113 is in the closed state (closed state in S152), the second filling rate calculation unit 804 calculates the filling rate by the second method (S154). Details of the second method will be described with reference to FIG.

[0190] FIG. 31 is a diagram illustrating an example of a second method for calculating the filling rate.

[0191] Consider the case where a spatial three-dimensional model 2051 is acquired as shown in FIG. 31(a).

[0192] Figure 31(b) is an enlarged view of region R2 in the spatial three-dimensional model 2051. As shown in Figure 31(b), the second filling rate calculation unit 804 divides region R2 into a second portion where the opening / closing part 113 is detected and a first portion where the baggage 103 is detected.

[0193] The first portion is an area including a three-dimensional point cloud on the far side of the area of ​​the opening 112a. The first portion is a portion of the ranging sensor 210 facing the luggage 103 in a direction from the ranging sensor 210 toward the luggage 103. In other words, the first portion is a portion facing the through-hole 113a in the opening / closing unit 113 in the closed state in a direction from the ranging sensor 210 toward the luggage 103. The opening / closing unit 113 may be configured to have one through-hole 113a. The direction from the ranging sensor 210 toward the luggage 103 may be, for example, along the horizontal direction.

[0194] The second portion is an area including a three-dimensional point cloud on the front-to-back side of the area of ​​the opening 112a of the car cart 112. The second portion is also an area where the distance measuring sensor 210 does not face the luggage 103 in the direction from the distance measuring sensor 210 toward the luggage 103. In other words, the second portion is an area hidden by the opening / closing unit 113 in the closed state in the direction from the distance measuring sensor 210 toward the luggage 103.

[0195] The second filling rate calculation unit 804 generates voxel data 2052 shown in (c) of Fig. 31 by voxelizing the first portion and the second portion, respectively. In the voxel data 2052, the white areas without hatching are areas where the second portion has been voxelized, and the dotted hatched areas are areas where the first portion has been voxelized.

[0196] Then, the second filling rate calculation unit 804 estimates whether or not luggage 103 is present behind the opening / closing part 113 for the white region corresponding to the region of the opening / closing part 113. Specifically, the second filling rate calculation unit 804 assigns scores based on the probability that luggage is present to 26 voxels adjacent to the voxel hatched with dots where the luggage 103 is present in the voxelized region. Then, as shown in (d) of FIG. 31, the second filling rate calculation unit 804 assigns an added score to voxels indicated by white regions adjacent to multiple voxels where the luggage 103 is present. The second filling rate calculation unit 804 performs this for all voxels where the luggage 103 is present, and determines that the luggage 103 is present in voxels indicated by white regions where the total score is equal to or greater than an arbitrary threshold. For example, when an arbitrary threshold value is set to 0.1, the second filling rate calculation unit 804 determines that luggage 103 is present in all areas, and can therefore calculate a luggage model 2053 in which the shape of the area hidden by the opening / closing section 113 is estimated, as shown in (e) of Figure 31.

[0197] In this way, the information processing device 220 estimates the shape of the second part where the ranging sensor 210 does not face the object to be measured based on the shape of the first part where the ranging sensor 210 faces the luggage 103, so that even if a second part is present, the information processing device 220 can appropriately estimate the three-dimensional model of the object.

[0198] If a rule is established that the luggage 103 must be placed without gaps inside the car cart 112, the second filling rate calculation unit 804 may extract a contour R3 of an area where one or more luggage 103 are placed, and determine that the area inside the extracted contour R3 is an area where luggage 103 exists, as shown in Fig. 32. Then, the second filling rate calculation unit 804 may estimate the area of ​​the opening / closing unit 113 inside the contour R3 using a three-dimensional point cloud in the area of ​​the multiple through-holes 113a of the opening / closing unit 113.

[0199] In the filling rate measurement method according to the third modification, the car truck 112 further includes an opening / closing unit 113 that has a plurality of through-holes 113a and opens and closes the opening 112a. The filling rate measurement method further determines whether the opening / closing unit 113 is in an open state or a closed state, and if the opening / closing unit 113 is in an open state, estimates a luggage model 2034 by performing extraction and estimation in the same manner as the filling rate calculation unit 224 of the embodiment. If the opening / closing unit 113 is in a closed state, the filling rate calculation unit 224 estimates a second portion hidden by the opening / closing unit 113 based on a plurality of first portions that correspond to the plurality of through-holes 113a of the opening / closing unit 113 in the voxel data 2031 based on the spatial three-dimensional model 2011, and estimates the luggage model 2034 using the plurality of first portions, the estimated second portion, and the storage three-dimensional model 2032.

[0200] According to this, even when the luggage 103 is stored in the cart 112 provided with the opening / closing section 113 that opens and closes the opening 112a, the estimation method of the luggage model 2034 is switched between the first method and the second method depending on the open / closed state of the opening / closing section 113, so that the three-dimensional model of the object can be estimated appropriately.

[0201] Furthermore, in the filling rate measurement method according to the third modification, the direction from the distance measuring sensor 210 toward the baggage 103 is, for example, along the horizontal direction. Therefore, there is no need to adjust the position of the distance measuring sensor 210 so that it can measure from a direction where there is no opening / closing part 113 having the through-hole 113a, and therefore there is a high degree of freedom in installing the distance measuring sensor 210. Therefore, even if the position of the distance measuring sensor 210 cannot be completely adjusted, it is possible to obtain measurement results for estimating a three-dimensional model of the object using the distance measuring sensor 210.

[0202] (Variation 4) FIG. 33 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fourth modification.

[0203] 33, even when generating a spatial 3D model, the 3D measurement system 200 may integrate the measurement results of the multiple ranging sensors 210, similar to the processing of the model generation unit 223. In this case, the 3D measurement system 200 identifies the positions and orientations of the multiple ranging sensors 210 by performing calibration in advance, and can generate a spatial 3D model including a 3D point cloud with little occlusion by integrating the obtained multiple measurement results based on the identified positions and orientations of the multiple ranging sensors 210.

[0204] (Variation 5) FIG. 34 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fifth modification.

[0205] 34, even when generating a spatial 3D model, the 3D measurement system 200 may move at least one of the car bogie 112 and one distance measuring sensor 210 so as to cross the measurement region R1 of one distance measuring sensor 210, and integrate multiple measurement results obtained by the distance measuring sensor 210 at multiple times during the movement. In this case, the relative position and orientation between the car bogie 112 and one distance measuring sensor 210 is calculated, and the multiple measurement results are integrated using the relative position and orientation, thereby generating a spatial 3D model including a 3D point cloud with little occlusion.

[0206] (others) Although the filling rate measurement method and the like according to the present disclosure have been described above based on the above-described embodiments, the present disclosure is not limited to the above-described embodiments.

[0207] For example, in the above-described embodiments, each processing unit included in an information processing device or the like is described as being implemented by a CPU and a control program. For example, each component of the processing unit may be composed of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may include, for example, a semiconductor device, an integrated circuit (IC), or a large-scale integration (LSI). The IC or LSI may be integrated on a single chip or on multiple chips. While the IC or LSI is referred to here as an IC or LSI, the name may vary depending on the degree of integration, and may be called a system LSI, a very large-scale integration (VLSI), or an ultra-large-scale integration (ULSI). Furthermore, a field-programmable gate array (FPGA), which is programmed after the LSI is manufactured, can also be used for the same purpose.

[0208] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, the general or specific aspects may be realized as a computer-readable non-transitory recording medium such as an optical disk, a hard disk drive (HDD), or a semiconductor memory on which the computer program is stored. Alternatively, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0209] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, and forms realized by arbitrarily combining the components and functions of the embodiments within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0210] The present disclosure is useful as a filling rate measurement method, an information processing device, a program, and the like that can calculate the filling rate of a measurement object. [Explanation of symbols]

[0211] 101, 105, 111 storage space 102 Shelf 102a, 112a opening 103, 103a~103c Luggage 104 Marker 106 Cargo bed 112 Basket Cart 113 Opening and Closing Section 113a Through hole 200 Three-dimensional measurement system 210, 210A, 210B Distance Sensor 211 Laser irradiation unit 211A Infrared pattern irradiation unit 211B, 212B cameras 212 Laser receiving unit 212A Infrared Camera 220 Information processing equipment 221, 701 Acquisition Department 222, 222A, 222B Coordinate system calculation section 223 Model Generation Unit 224 Filling rate calculation section 225 Storage section 301 Auxiliary section 302, 313, 323, 503, 605, 703 Calculation section 311, 321, 401 Detector 312, 322, 501 Extraction part 402 Generator 403 Volume calculation unit 502 Estimation section 601 Baggage volume calculation unit 602 Area division part 603 Planned Baggage Measurement Department 604 Area estimation part 702 Counting Department 801 Detection unit 802 Switching unit 803 1st filling rate calculation section 804 2nd filling rate calculation section 2000 measurement coordinate system 2001, 2021 images 2002 Adjustment Marker 2003 Superimposed Image 2004 Sensor Coordinate System 2011, 2051 spatial three-dimensional model 2012, 2022, 2032 storage 3D model 2013 Location information 2014 shelf area 2015 Aperture 2016 Opening end point 2017, 2026 Rotation matrix 2018, 2027 Translation Vector 2023 Marker 2024 Marker Area 2025 pattern contour 2031, 2052 voxel data 2033 Luggage Area 2034, 2053 luggage models 2041 Occupied area 2042 Sky area P1 one point R1 measurement area R2 area R3 Contour

Claims

1. a first storage unit having a first storage space into which a measurement object is stored through an opening, and an opening / closing unit having a plurality of through holes and arranged to cover the opening, the first storage unit being measured through the plurality of through holes by a distance measuring sensor facing the first storage unit, and a three-dimensional spatial model being obtained; acquiring a stored three-dimensional model of the first storage unit in a state where the measurement object is not stored; using the acquired spatial three-dimensional model and the storage three-dimensional model, estimating an object three-dimensional model of the measurement object in the first storage space; calculating a first filling rate of the measurement object relative to the first storage space using the storage three-dimensional model and the object three-dimensional model; In the above estimation, estimating a shape of a second portion of the measurement object that is hidden from the distance measuring sensor based on a shape of a first portion of the measurement object measured by the distance measuring sensor through the plurality of through holes in a direction from the distance measuring sensor toward the measurement object; and estimating the three-dimensional model of the object using the first and second portions and the stored three-dimensional model. Filling rate measurement method.

2. In the above estimation, Furthermore, it is determined whether the opening / closing unit is in an open state or a closed state, When the opening / closing unit is in the closed state, the shape of the second portion of the measurement object is estimated. The filling rate measurement method according to claim 1 .

3. In the above estimation, When the opening / closing unit is in the open state, extracting an object portion that corresponds to the measurement object from the spatial three-dimensional model using the acquired spatial three-dimensional model and the stored three-dimensional model; A three-dimensional model of the object is estimated using the extracted object portion. The filling rate measuring method according to claim 2 .

4. In the estimation, the three-dimensional model of the object is estimated based on a first three-dimensional coordinate system based on a shape of a part of the first storage section. The filling rate measuring method according to any one of claims 1 to 3.

5. The second portion is a portion where the distance measuring sensor does not face the measurement object in the direction. The filling rate measuring method according to any one of claims 1 to 4.

6. the first storage section and the additional first storage section are stored in a second storage space of the second storage section; The filling rate measurement method further includes: Calculating a second filling rate of the first storage section and the additional first storage section with respect to the second storage space The filling rate measuring method according to any one of claims 1 to 5.

7. The stored three-dimensional model is a three-dimensional model measured by the distance measurement sensor and an additional distance measurement sensor. The filling rate measuring method according to any one of claims 1 to 6.

8. The distance measurement sensor has at least two cameras for generating the three-dimensional spatial model and is fixed on the upper side of the first storage section. The filling rate measuring method according to any one of claims 1 to 7.

9. a processor; a memory; The processor uses the memory to: a first storage unit having a first storage space into which a measurement object is stored through an opening, and an opening / closing unit having a plurality of through holes and arranged to cover the opening, the first storage unit being measured through the plurality of through holes by a distance measuring sensor facing the first storage unit, and a three-dimensional spatial model being obtained; acquiring a stored three-dimensional model of the first storage unit in a state where the measurement object is not stored; using the acquired spatial three-dimensional model and the storage three-dimensional model, estimating an object three-dimensional model of the measurement object in the first storage space; calculating a first filling rate of the measurement object relative to the first storage space using the storage three-dimensional model and the object three-dimensional model; In the above estimation, estimating a shape of a second portion of the measurement object that is hidden from the distance measuring sensor based on a shape of a first portion of the measurement object measured by the distance measuring sensor through the plurality of through holes in a direction from the distance measuring sensor toward the measurement object; and estimating the three-dimensional model of the object using the first and second portions and the stored three-dimensional model. Information processing device.

10. A program for causing a computer to execute a filling rate measurement method, The filling rate measurement method includes: a first storage unit having a first storage space into which a measurement object is stored through an opening, and an opening / closing unit having a plurality of through holes and arranged to cover the opening, the first storage unit being measured through the plurality of through holes by a distance measuring sensor facing the first storage unit, and a three-dimensional spatial model being obtained; acquiring a stored three-dimensional model of the first storage unit in a state where the measurement object is not stored; using the acquired spatial three-dimensional model and the storage three-dimensional model, estimating an object three-dimensional model of the measurement object in the first storage space; calculating a first filling rate of the measurement object relative to the first storage space using the storage three-dimensional model and the object three-dimensional model; In the above estimation, estimating a shape of a second portion of the measurement object that is hidden from the distance measuring sensor based on a shape of a first portion of the measurement object measured by the distance measuring sensor through the plurality of through holes in a direction from the distance measuring sensor toward the measurement object; and estimating the three-dimensional model of the object using the first and second portions and the stored three-dimensional model. program.

Images