Measurement method, information processing device and program

Abstract

To provide a measurement method for calculating an appropriate filling rate of an object to be measured.SOLUTION: A measurement method includes: acquiring a space three-dimensional model obtained by measuring a storage part through an opening with a rage finding sensor (S111); acquiring a stored three-dimensional model of the storage part in an unstored state (S112); acquiring a two-dimensional image of the opening and position and posture information corresponding to the two-dimensional image, and identifying a line segment that indicates a shape of the opening from the two-dimensional image (S113); associating a position of the stored three-dimensional model with a position of the space three-dimensional model based on the position and posture information and the identified line segment, and estimating an object three-dimensional model which is a three-dimensional model of a measurement object based on the stored three-dimensional model and the space three-dimensional model after the association (S115); and estimating, using the object three-dimensional model, a volume of an area in which an object of a specific shape can be further stored in a storage space in which the measurement object is stored.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present disclosure relates to a measurement method, an information processing device, and a program. [Background technology]

[0002] Patent Document 1 discloses a three-dimensional shape measuring device that acquires a three-dimensional shape using a three-dimensional laser scanner. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-87319 Summary of the Invention [Problem to be solved by the invention]

[0004] There has been insufficient consideration of applications of the measured three-dimensional shape, such as the calculation of the filling rate, which indicates how much of the measurement target object is stored in a given storage space.

[0005] The present disclosure provides a measurement method and the like for calculating an appropriate filling rate of an object to be measured. [Means for solving the problem]

[0006] A measurement method according to one aspect of the present disclosure includes acquiring a spatial three-dimensional model obtained by measuring a storage unit having an opening and a storage space in which a measurement object is stored through the opening using a ranging sensor, acquiring a storage three-dimensional model of the storage unit in an unstored state, acquiring a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image, using the storage three-dimensional model to identify a line segment indicative of the shape of the opening in the two-dimensional image, matching the position of the storage three-dimensional model to the position of the spatial three-dimensional model based on the position and orientation information and the identified line segment, estimating a three-dimensional object model, which is a three-dimensional model of the measurement object within the storage space, based on the storage three-dimensional model and the spatial three-dimensional model after the matching, and using the object three-dimensional model to estimate the volume of an area in the storage space in which the measurement object is stored that can further store an object of a specific shape.

[0007] An information processing device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire a spatial three-dimensional model obtained by measuring a storage unit having an opening and a storage space in which a measurement object is stored through the opening with a ranging sensor, acquire a storage three-dimensional model of the storage unit in an unstored state, acquire a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image, use the storage three-dimensional model to identify a line segment indicative of the shape of the opening in the two-dimensional image, correspond the position of the storage three-dimensional model to the position of the spatial three-dimensional model based on the position and orientation information and the identified line segment, estimate a three-dimensional object model that is a three-dimensional model of the measurement object within the storage space based on the storage three-dimensional model and the spatial three-dimensional model after the correspondence, and use the object three-dimensional model to estimate the volume of an area in the storage space in which the measurement object is stored that can further store an object of a specific shape.

[0008] A filling rate measurement method according to one aspect of the present disclosure includes acquiring a spatial three-dimensional model obtained by measuring a storage section having a storage space in which a measurement object is stored and having an opening formed therein through the opening using a ranging sensor facing the storage section, acquiring a storage three-dimensional model that is a three-dimensional model of the storage section in which the measurement object is not stored, acquiring a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image, identifying a line segment indicative of the shape of the opening in the two-dimensional image using the storage three-dimensional model, calculating a position of the opening in three-dimensional space based on the position and orientation information and the identified line segment, correlating the position of the storage three-dimensional model with the position of the spatial three-dimensional model based on the calculated position of the opening, estimating a three-dimensional object model that is a three-dimensional model of the measurement object within the storage space based on the storage three-dimensional model and the spatial three-dimensional model after the correspondence, and calculating a filling rate of the measurement object in the storage space using the storage three-dimensional model and the object three-dimensional model.

[0009] An information processing device according to one aspect of the present disclosure includes a processor and a memory, wherein the processor uses the memory to acquire a spatial three-dimensional model obtained by measuring a storage section having a storage space in which a measurement object is stored and having an opening formed therein through the opening using a ranging sensor facing the storage section, acquire a storage three-dimensional model that is a three-dimensional model of the storage section in which the measurement object is not stored, acquire a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image, identify a line segment indicative of the shape of the opening in the two-dimensional image using the storage three-dimensional model, calculate a position of the opening in three-dimensional space based on the position and orientation information and the identified line segment, correspond the position of the storage three-dimensional model to the position of the spatial three-dimensional model based on the calculated position of the opening, estimate a three-dimensional object model that is a three-dimensional model of the measurement object within the storage space based on the storage three-dimensional model and the spatial three-dimensional model after the correspondence, and calculate a filling rate of the measurement object in the storage space using the storage three-dimensional model and the object three-dimensional model.

[0010] The present disclosure may be realized as a program that causes a computer to execute the steps included in the filling rate measurement method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network, such as the Internet. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide a measurement method and the like for calculating an appropriate filling rate of an object to be measured. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining an outline of a filling rate measurement method according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing a characteristic configuration of the three-dimensional measurement system according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating a first example of the configuration of the distance measurement sensor. [Figure 4] FIG. 4 is a diagram illustrating a second example of the configuration of the distance measuring sensor. [Figure 5] FIG. 5 is a diagram illustrating a third example of the configuration of the distance measuring sensor. [Figure 6] FIG. 6 is a block diagram showing the configuration of the coordinate system calculation unit of the first example. [Figure 7] FIG. 7 is a diagram for explaining a method for calculating a measurement coordinate system by the coordinate system calculation unit of the first example. [Figure 8] FIG. 8 is a block diagram showing the configuration of the coordinate system calculation unit of the second example. [Figure 9] FIG. 9 is a diagram for explaining a method for calculating a measurement coordinate system by a coordinate system calculation unit of the second example. [Figure 10] FIG. 10 is a block diagram showing the configuration of the coordinate system calculation unit of the third example. [Figure 11] FIG. 11 is a diagram for explaining a method for calculating a measurement coordinate system by a coordinate system calculation unit of the third example. [Figure 12] FIG. 12 is a block diagram illustrating an example of the configuration of the model generating unit. [Figure 13] FIG. 13 is a flowchart of a process for calculating the volume of the storage space by the model generation unit. [Figure 14] FIG. 14 is a block diagram illustrating an example of the configuration of the filling rate calculation unit. [Figure 15] FIG. 15 is a diagram for explaining an example of a method for calculating the filling rate by the filling rate calculation unit. [Figure 16] FIG. 16 is a diagram for explaining another example of a method for calculating the filling rate by the filling rate calculation unit. [Figure 17] FIG. 17 is a flowchart of a filling rate measurement method performed by an information processing device. [Figure 18] FIG. 18 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the first example. [Figure 19] FIG. 19 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the second example. [Figure 20] FIG. 20 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the third example. [Figure 21] FIG. 21 is a block diagram showing the configuration of a coordinate system calculation unit according to the second embodiment. [Figure 22] FIG. 22 is a block diagram showing the configuration of a detection unit included in a coordinate system calculation unit according to the second embodiment. [Figure 23] FIG. 23 is a diagram illustrating a method for extracting aperture end points by a detection unit according to the second embodiment. [Figure 24] FIG. 24 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit according to the second embodiment. [Figure 25] FIG. 25 is a diagram for explaining a method for calculating the filling rate. [Figure 26] FIG. 26 is a block diagram showing an example of the configuration of a calculation unit of a filling rate calculation unit according to the first modification. [Figure 27] FIG. 27 is a flowchart of a filling rate calculation process of the calculation unit of the filling rate calculation unit according to the first modification. [Figure 28] FIG. 28 is a diagram showing an example in which two or more shelves are stored in a storage space such as the bed of a truck. [Figure 29] FIG. 29 is a table showing the relationship between the shelves stored in the storage space of the loading platform and their filling rates. [Figure 30] FIG. 30 is a block diagram showing an example of the configuration of a calculation unit of a filling rate calculation unit according to the second modification. [Figure 31] FIG. 31 is a flowchart of a filling rate calculation process of the calculation unit of the filling rate calculation unit according to the second modification. [Figure 32]FIG. 32 is a diagram for explaining the configuration of a car bogie according to Modification 3. In FIG. [Figure 33] FIG. 33 is a block diagram showing an example of the configuration of a filling rate calculation unit according to the third modification. [Figure 34] FIG. 34 is a flowchart of a filling rate calculation process of the filling rate calculation unit according to the third modification. [Figure 35] FIG. 35 is a diagram illustrating an example of a second method for calculating the filling rate. [Figure 36] FIG. 36 is a diagram for explaining another example of the second method for calculating the filling rate. [Figure 37] FIG. 37 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fourth modification. [Figure 38] FIG. 38 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fifth modification. [Figure 39] FIG. 39 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fifth modification. [Figure 40] FIG. 40 is a diagram showing an example in which a single distance measuring sensor measures a plurality of car bogies. [Figure 41] FIG. 41 is a diagram showing an example in which a plurality of car bogies are measured by two distance measurement sensors. [Figure 42] FIG. 42 is a diagram showing an example in which a plurality of car bogies are measured by three distance measurement sensors. [Figure 43] FIG. 43 is a block diagram showing a characteristic configuration of a three-dimensional measurement system according to the seventh modification. [Figure 44] FIG. 44 is a flowchart of a filling rate measurement method performed by an information processing device according to the seventh modification. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Background to this disclosure) In logistics and distribution sites, there is a demand for measuring the filling rate of measurement objects, such as luggage, relative to the storage space, thereby improving the utilization efficiency of the storage space. Also, in logistics and distribution sites, since measurement objects are stored in storage areas such as many containers, there is a demand for measuring a large number of filling rates in a short period of time. However, a method for easily measuring filling rates has not been fully explored.

[0014] Therefore, the present disclosure provides a filling rate measurement method, etc., for easily calculating the filling rate of more storage sections in a short period of time by applying a technique for generating a three-dimensional model of the storage section in which the object to be measured is stored.

[0015] A filling rate measurement method according to one aspect of the present disclosure includes acquiring a spatial three-dimensional model obtained by measuring a storage section having a storage space in which a measurement object is stored and having an opening formed therein through the opening using a ranging sensor facing the storage section, acquiring a storage three-dimensional model that is a three-dimensional model of the storage section in which the measurement object is not stored, acquiring a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image, identifying a line segment indicative of the shape of the opening in the two-dimensional image using the storage three-dimensional model, calculating a position of the opening in three-dimensional space based on the position and orientation information and the identified line segment, correlating the position of the storage three-dimensional model with the position of the spatial three-dimensional model based on the calculated position of the opening, estimating a three-dimensional object model that is a three-dimensional model of the measurement object within the storage space based on the storage three-dimensional model and the spatial three-dimensional model after the correspondence, and calculating a filling rate of the measurement object in the storage space using the storage three-dimensional model and the object three-dimensional model.

[0016] According to this method, a three-dimensional object model of the measurement object is estimated using a three-dimensional coordinate system based on the position of the opening and the estimated three-dimensional object model. This makes it possible to easily calculate the filling rate of the measurement object relative to the storage space by simply measuring the storage section with the measurement object stored therein.

[0017] The two-dimensional image may include an RGB image generated by capturing an image of the opening with a camera, and the position and orientation information may indicate the position and orientation of the camera when capturing the image of the opening.

[0018] The two-dimensional image may include a depth image generated based on measurement of the opening by the ranging sensor, and the position and orientation information may indicate the position and orientation of the ranging sensor at the time of measurement of the opening.

[0019] Furthermore, the two-dimensional image may include at least one of an RGB image, a grayscale image, an infrared image, and a depth image, and the RGB image may be generated by photographing the opening with a camera, and the depth image may be generated based on the measurement results of the ranging sensor.

[0020] This allows the line segments that indicate the shape of the opening to be extracted with high precision.

[0021] Furthermore, the line segments may be identified based on both the line segments identified from the RGB image and the line segments identified from the depth image.

[0022] The distance measurement sensor may include at least one of a ToF (Time of Flight) sensor and a stereo camera.

[0023] The distance measurement sensor may include a first distance measurement sensor and a second distance measurement sensor, and a first measurement area of ​​the first distance measurement sensor and a second measurement area of ​​the second distance measurement sensor may have an overlapping area.

[0024] This allows the measurement object to be measured over a wider range.

[0025] The overlapping area may have a length equal to or greater than the length of the object to be measured in the distance measurement direction of the distance measurement sensor.

[0026] This allows the measurement object to be measured over a wider range.

[0027] The overlapping area may include the entire range in which the measurement object exists.

[0028] This makes it possible to obtain a spatial 3D model with little occlusion.

[0029] In addition, the storage unit may move relative to the ranging sensor in a direction intersecting the ranging direction of the ranging sensor, and the spatial three-dimensional model may be generated using a first measurement result measured by the ranging sensor at a first timing and a second measurement result measured at a second timing.

[0030] This makes it possible to obtain a spatial 3D model with little occlusion.

[0031] Furthermore, the position of the stored three-dimensional model and the position of the spatial three-dimensional model may be associated with each other using a rotation matrix and a translation vector.

[0032] Further, a second filling rate of the one or more storage sections may be calculated for a second storage section having a second storage space in which the one or more storage sections are stored.

[0033] Furthermore, a third filling rate of the measurement objects stored in each of the one or more storage sections may be calculated for a second storage section having a second storage space in which the one or more storage sections are stored.

[0034] a storage three-dimensional model that is a three-dimensional model of the storage section in which the measurement object is not stored; a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image; a position of the opening in three-dimensional space based on the position and orientation information and the identified line segment; a correspondence between the position of the storage three-dimensional model and the position of the spatial three-dimensional model based on the calculated position of the opening; an object three-dimensional model that is a three-dimensional model of the measurement object in the storage space based on the correspondence between the storage three-dimensional model and the spatial three-dimensional model; and a filling rate of the measurement object in the storage space based on the correspondence between the storage three-dimensional model and the spatial three-dimensional model.

[0035] According to this method, a three-dimensional object model of the measurement object is estimated using a three-dimensional coordinate system based on the position of the opening and the estimated three-dimensional object model. This makes it possible to easily calculate the filling rate of the measurement object relative to the storage space by simply measuring the storage section with the measurement object stored therein.

[0036] The present disclosure may be realized as a program that causes a computer to execute the steps included in the filling rate measurement method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals representing the program. These programs, information, data, and signals may be distributed via a communication network, such as the Internet.

[0037] Hereinafter, each embodiment of the 3D model generation method and the like according to the present disclosure will be described in detail with reference to the drawings. Note that each embodiment described below represents a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, arrangement and connection of components, steps, order of steps, and the like shown in each of the following embodiments are merely examples and are not intended to limit the present disclosure.

[0038] Furthermore, each drawing is a schematic diagram and is not necessarily an exact illustration. In addition, in each drawing, substantially the same components are denoted by the same reference numerals, and duplicated explanations may be omitted or simplified.

[0039] (Embodiment 1) An outline of the filling rate measurement method according to the first embodiment will be described with reference to FIG.

[0040] FIG. 1 is a diagram for explaining an outline of a filling rate measurement method according to the first embodiment.

[0041] In the filling rate measurement method, as shown in Fig. 1, luggage 103 stored on a shelf 102 having a storage space 101 is measured using a distance measurement sensor 210. Then, the obtained measurement results are used to calculate the filling rate of the luggage 103 in the storage space 101. An opening 102a is formed in the shelf 102 for allowing the luggage 103 to be placed in and removed from the storage space 101. The distance measurement sensor 210 is disposed in a position facing the opening 102a of the shelf 102 in an orientation such that it measures the shelf 102 including the opening 102a, and measures a measurement region R1 including the interior of the storage space 101 through the opening 102a.

[0042] Note that the shelf 102 has a box-like shape, for example, as shown in FIG. 1 . The shelf 102 does not have to have a box-like shape as long as it has a configuration including a placement surface on which the luggage 103 is placed and a storage space 101 above the placement surface in which the luggage 103 is stored. The shelf 102 is an example of a storage section. The storage space 101 is an example of a first storage space. While the storage space 101 has been described as an internal space of the shelf 102, it is not limited to this and may be a space within a warehouse in which measurement objects such as luggage 103 are stored. The luggage 103 is an example of a measurement object. The measurement object is not limited to luggage 103 and may be a commodity. In other words, the measurement object may be any object as long as it is portable.

[0043] Fig. 2 is a block diagram showing a characteristic configuration of the three-dimensional measurement system according to embodiment 1. Fig. 3 is a diagram for explaining a first example of the configuration of a distance measuring sensor. Fig. 4 is a diagram for explaining a second example of the configuration of a distance measuring sensor. Fig. 5 is a diagram for explaining a third example of the configuration of a distance measuring sensor.

[0044] 2, the three-dimensional measurement system 200 includes a distance measurement sensor 210 and an information processing device 220. The three-dimensional measurement system 200 may include multiple distance measurement sensors 210, or may include a single distance measurement sensor 210.

[0045] The distance measurement sensor 210 measures a three-dimensional space including the first storage space of the shelf 102 through the opening 102a of the shelf 102 to obtain measurement results including the shelf 102 and the storage space 101 of the shelf 102. Specifically, the distance measurement sensor 210 generates a spatial three-dimensional model represented by a collection of three-dimensional points indicating the three-dimensional positions of each of multiple measurement points on the shelf 102 or the baggage 103 (hereinafter referred to as the measurement target) (the surface of the measurement target). A collection of three-dimensional points is called a three-dimensional point cloud. The three-dimensional position indicated by each three-dimensional point in the three-dimensional point cloud is represented by three-value information consisting of, for example, an X component, a Y component, and a Z component in a three-dimensional coordinate space consisting of X, Y, and Z axes. Note that the three-dimensional model may include not only three-dimensional coordinates but also color information indicating the color of each point or shape information indicating the surface shape of each point and its surroundings. The color information may be represented, for example, in an RGB color space or another color space such as HSV, HLS, or YUV.

[0046] A specific example of the distance measurement sensor 210 will be described with reference to FIGS.

[0047] As shown in FIG. 3 , the distance measurement sensor 210 of the first example generates a three-dimensional spatial model by emitting electromagnetic waves and acquiring waves reflected by a measurement target. Specifically, the distance measurement sensor 210 measures the time it takes for the emitted electromagnetic waves to be reflected by the measurement target and return to the distance measurement sensor 210, and calculates the distance between the distance measurement sensor 210 and point P1 on the surface of the measurement target using the measured time and the wavelength of the electromagnetic waves used for the measurement. The distance measurement sensor 210 emits electromagnetic waves in multiple radial directions determined in advance from a reference point of the distance measurement sensor 210. For example, the distance measurement sensor 210 may emit electromagnetic waves at first angular intervals around the horizontal direction and at second angular intervals around the vertical direction. Therefore, the distance measurement sensor 210 can calculate the three-dimensional coordinates of multiple points on the measurement target by detecting the distance between the measurement target and the distance measurement sensor 210 in each of multiple directions around the distance measurement sensor 210. Therefore, the distance measurement sensor 210 can calculate position information indicating multiple three-dimensional positions on the measurement target and generate a spatial three-dimensional model having the position information. The position information may be a three-dimensional point cloud including multiple three-dimensional points indicating multiple three-dimensional positions.

[0048] 3, the distance measuring sensor 210 of the first example is a three-dimensional laser measuring instrument having a laser emitting unit 211 that emits laser light as electromagnetic waves and a laser receiving unit 212 that receives light reflected from the measurement object of the measurement. The distance measuring sensor 210 scans the measurement object with laser light by rotating or oscillating a unit including the laser emitting unit 211 and the laser receiving unit 212 on two different axes, or by installing a movable mirror (MEMS (Micro Electro Mechanical Systems) mirror) that oscillates on two axes on the path of the emitting or receiving laser. In this way, the distance measuring sensor 210 can generate a high-precision and high-density three-dimensional model of the measurement object of the measurement.

[0049] The distance measurement sensor 210 is exemplified as a three-dimensional laser measuring instrument that measures the distance to the measurement object by irradiating laser light, but is not limited to this and may also be a millimeter wave radar measuring instrument that measures the distance to the measurement object by emitting millimeter waves.

[0050] Furthermore, the ranging sensor 210 may generate a three-dimensional model having color information. The first color information is color information generated using an image captured by the ranging sensor 210, and indicates the color of each of the first three-dimensional points included in the first three-dimensional point cloud.

[0051] Specifically, the ranging sensor 210 may have a built-in camera that captures an image of a measurement target around the ranging sensor 210. The camera built into the ranging sensor 210 generates an image by capturing an image of an area including the irradiation range of the laser light emitted by the ranging sensor 210. The camera does not have to be built into the ranging sensor 210 and may be disposed outside the ranging sensor 210. The camera disposed outside the ranging sensor 210 may be disposed at the same position as the ranging sensor 210. The imaging range captured by the camera is associated in advance with the irradiation range. Specifically, multiple directions in which the laser light is emitted by the ranging sensor 210 are associated in advance with each pixel in the image captured by the camera, and the ranging sensor 210 sets pixel values ​​of the image associated with the directions of the multiple 3D points included in the 3D point cloud as color information indicating the colors of the respective 3D points.

[0052] The second example of the distance measuring sensor 210A is a distance measuring sensor using a structured light method, as shown in FIG. 4 . The distance measuring sensor 210A includes an infrared pattern irradiator 211A and an infrared camera 212A. The infrared pattern irradiator 211A projects a predetermined infrared pattern 213A onto the surface of the measurement target. The infrared camera 212A captures an infrared image of the measurement target on which the infrared pattern 213A is projected. The distance measuring sensor 210A searches for the infrared pattern 213A included in the acquired infrared image and calculates the distance from the infrared pattern irradiator 211A or the infrared camera 212A to the position of point P1 on the measurement target based on a triangle formed by connecting the three positions of the position of point P1 of the infrared pattern on the measurement target in real space, the position of the infrared pattern irradiator 211A, and the position of the infrared camera 212A. This allows the distance measuring sensor 210A to obtain a three-dimensional measurement point on the measurement target.

[0053] In addition, the distance measurement sensor 210A can acquire a high-density three-dimensional model by moving the unit of the distance measurement sensor 210A, which has the infrared pattern irradiation unit 211A and the infrared camera 212A, or by making the infrared pattern irradiated by the infrared pattern irradiation unit 211A into a fine texture.

[0054] Furthermore, distance measuring sensor 210A may generate a three-dimensional model having color information by using the visible light region of color information that can be acquired by infrared camera 212A and associating the obtained visible light region with three-dimensional points in consideration of the position or orientation of infrared pattern irradiator 211A or infrared camera 212A. Furthermore, distance measuring sensor 210A may be configured to further include a visible light camera for adding color information.

[0055] Distance measurement sensor 210B of the third example is a distance measurement sensor that measures three-dimensional points using stereo camera measurement, as shown in FIG. 5. Distance measurement sensor 210B is a stereo camera having two cameras 211B and 212B. Distance measurement sensor 210B acquires stereo images with parallax by capturing images of the measurement target using the two cameras 211B and 212B at synchronized timing. Distance measurement sensor 210B uses the acquired stereo images (two images) to perform a feature point matching process between the two images and acquires alignment information between the two images with pixel accuracy or subpixel accuracy. Distance measurement sensor 210B calculates the distance from one of the two cameras 211B and 212B to the matching position (i.e., point P1) on the measurement target based on a triangle formed by connecting the three positions of point P1 on the measurement target in real space and the positions of the two cameras 211B and 212B. This allows the distance measurement sensor 210B to acquire a three-dimensional point of the measurement point on the measurement target.

[0056] In addition, distance measurement sensor 210B can obtain a highly accurate three-dimensional model by moving the unit of distance measurement sensor 210B, which has two cameras 211B and 212B, or by increasing the number of cameras mounted on distance measurement sensor 210B to three or more, photographing the same measurement object, and performing matching processing.

[0057] Furthermore, by using visible light cameras as the cameras 211B and 212B of the distance measurement sensor 210B, it is possible to easily add color information to the acquired three-dimensional model.

[0058] In this embodiment, the information processing device 220 will be described as an example having the distance measurement sensor 210 of the first example, but it may also be configured to have the distance measurement sensor 210A of the second example or the distance measurement sensor 210B of the third example instead of the distance measurement sensor 210 of the first example.

[0059] The two cameras 211B and 212B can capture monochrome images including visible light images or infrared images. In this case, the matching process between the two images in the three-dimensional measurement system 200 may be performed using, for example, SLAM (Simultaneous Localization And Mapping) or SfM (Structure from Motion). Furthermore, the point cloud density of the measurement space model may be increased by MVS (Multi View Stereo) using information indicating the positions and orientations of the cameras 211B and 212B obtained by this process.

[0060] Returning to FIG. 2, the configuration of the information processing device 220 will be described.

[0061] The information processing device 220 includes an acquisition unit 221 , a coordinate system calculation unit 222 , a model generation unit 223 , a filling rate calculation unit 224 , and a storage unit 225 .

[0062] The acquisition unit 221 acquires the spatial three-dimensional model and the image generated by the distance measurement sensor 210. Specifically, the acquisition unit 221 may acquire the spatial three-dimensional model and the image from the distance measurement sensor 210. The spatial three-dimensional model and the image acquired by the acquisition unit 221 may be stored in the storage unit 225.

[0063] The coordinate system calculation unit 222 calculates the positional relationship between the distance measuring sensor 210 and the shelf 102 using the spatial three-dimensional model and the image. As a result, the coordinate system calculation unit 222 calculates a measurement coordinate system based on the shape of a portion of the shelf 102. The coordinate system calculation unit 222 may calculate a measurement coordinate system based on only the shape of a portion of the shelf 102. Specifically, the coordinate system calculation unit 222 calculates the measurement coordinate system based on the shape of the opening 102a of the shelf 102 as the shape of the portion that serves as the reference for calculating the measurement coordinate system. Note that, if the shape of the opening 102a is rectangular as shown in the first embodiment, the shape of the opening 102a that serves as the reference for calculating the measurement coordinate system may be a corner of the shape of the opening 102a or a side of the shape of the opening 102a.

[0064] The measurement coordinate system is a three-dimensional orthogonal coordinate system and is an example of a first three-dimensional coordinate system. By calculating the measurement coordinate system, it is possible to identify the relative position and orientation of the distance measurement sensor 210 with respect to the shelf 102. In other words, this allows the sensor coordinate system of the distance measurement sensor 210 to be aligned with the measurement coordinate system, and calibration between the shelf 102 and the distance measurement sensor 210 can be performed. The sensor coordinate system is a three-dimensional orthogonal coordinate system.

[0065] In this embodiment, the rectangular parallelepiped shelf 102 has the opening 102a on one side of the shelf 102, but this is not limited to this. The shelf may have openings on multiple sides of the rectangular parallelepiped shape, such as openings on two sides, the front and rear, or openings on two sides, the front and top. When the shelf has multiple openings, a predetermined reference position, which will be described later, may be set for one of the multiple openings. The predetermined reference position may be set to a space where no three-dimensional points or voxels exist in a storage three-dimensional model, which is a three-dimensional model of the shelf 102.

[0066] Here, the coordinate system calculation unit 222 of the first example will be described with reference to FIGS.

[0067] Fig. 6 is a block diagram showing the configuration of a coordinate system calculation unit of the first example. Fig. 7 is a diagram for explaining a method of calculating a measurement coordinate system by the coordinate system calculation unit of the first example.

[0068] The coordinate system calculation unit 222 calculates a measurement coordinate system. The measurement coordinate system is a three-dimensional coordinate system that serves as the reference for the three-dimensional spatial model. For example, the distance measurement sensor 210 is installed at the origin of the measurement coordinate system and is installed facing directly toward the opening 102a of the shelf 102. In this case, the measurement coordinate system may be set such that the upward direction of the distance measurement sensor 210 is set as the X axis, the rightward direction is set as the Y axis, and the forward direction is set as the Z axis. The coordinate system calculation unit 222 has an auxiliary unit 301 and a calculation unit 302.

[0069] 7(a), the auxiliary unit 301 sequentially acquires in real time images 2001 that are measurement results by the distance measurement sensor 210 acquired by the acquisition unit 221, and superimposes an adjustment marker 2002 on each of the sequentially acquired images 2001. The auxiliary unit 301 sequentially outputs a superimposed image 2003 in which the adjustment marker 2002 is superimposed on the image 2001 to a display device (not shown). The display device sequentially displays the superimposed image 2003 output by the information processing device 220. Note that the auxiliary unit 301 and the display device may be provided integrally with the distance measurement sensor 210.

[0070] The adjustment marker 2002 is a marker for assisting the user in moving the distance measurement sensor 210 so that the position and orientation of the distance measurement sensor 210 relative to the shelf 102 are specific. While viewing the superimposed image 2003 displayed on the display device, the user can change the position and orientation of the distance measurement sensor 210 so that the adjustment marker 2002 overlaps a specific reference position on the shelf 102, thereby placing the distance measurement sensor 210 at a specific position and orientation relative to the shelf 102. The specific reference position on the shelf 102 is, for example, the position of each of the four corners of the rectangular opening 102a of the shelf 102.

[0071] When the distance measurement sensor 210 is placed at a specific position and orientation relative to the shelf 102, a superimposed image 2003 is generated in which four adjustment markers 2002 are superimposed at four positions corresponding to the positions of the four corners of the opening 102a of the shelf 102. For example, by moving the distance measurement sensor 210 so that the adjustment markers 2002 move in the directions of the arrows shown in (a) of Fig. 7, the user can align the four adjustment markers 2002 with the positions of the four corners of the opening 102a, as shown in (b) of Fig. 7.

[0072] Although the auxiliary unit 301 is described as superimposing the adjustment marker 2002 on the image 2001, it may also be possible to superimpose the adjustment marker on a spatial three-dimensional model and display the spatial three-dimensional model with the adjustment marker superimposed on it on a display device.

[0073] As shown in FIG. 7C, the calculation unit 302 calculates a rotation matrix 2005 and a translation vector 2006 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102 when the four adjustment markers 2002 are aligned with the four corner positions of the opening 102a. The calculation unit 302 calculates a measurement coordinate system 2000 whose origin is an arbitrary corner (one of the four corners) of the opening 102a by transforming the sensor coordinate system 2004 of the distance measurement sensor 210 using the calculated rotation matrix 2005 and translation vector 2006. This allows the calculation unit 302 to associate the position of the storage three-dimensional model with the position of the spatial three-dimensional model. Note that when the four adjustment markers 2002 are aligned with the four corner positions of the opening 102a, the user may input to an input device (not shown). The information processing device 220 may determine when the four adjustment markers 2002 have been aligned with the four corner positions of the opening 102a by acquiring the time when the input has been received from the input device. Alternatively, the information processing device 220 may analyze the image 2001 to determine whether the four adjustment markers 2002 have been aligned with the four corner positions of the opening 102a.

[0074] Next, the coordinate system calculation unit 222A of the second example will be described with reference to FIGS.

[0075] Fig. 8 is a block diagram showing the configuration of a coordinate system calculation unit of the second example. Fig. 9 is a diagram for explaining a method of calculating a measurement coordinate system by the coordinate system calculation unit of the second example.

[0076] The coordinate system calculation unit 222A includes a detection unit 311, an extraction unit 312, and a calculation unit 313.

[0077] The detection unit 311 detects a shelf area 2014 corresponding to the shelf 102, as shown in (c) of FIG. 9, using a spatial three-dimensional model 2011, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221, as shown in (a) of FIG. 9, and a storage three-dimensional model 2012, which is shown in (b) of FIG. 9. The storage three-dimensional model 2012 is a three-dimensional model of the shelf 102 on which no luggage 103 is stored, and is a three-dimensional model generated in advance using the measurement result by the distance measurement sensor 210 for the shelf 102 when no luggage 103 is stored. The storage three-dimensional model 2012 is generated by a model generation unit 223, which will be described later, and stored in the storage unit 225. The storage three-dimensional model 2012 may include position information 2013 indicating the positions of the four corners of the opening 102a of the shelf 102.

[0078] 9(d), the extraction unit 312 uses the position information 2013 in the storage three-dimensional model 2012 to extract four opening end points 2016, which are the positions of the four corners of the opening 2015 in the shelf area 2014. The shape of the opening 2015 defined by the four opening end points 2016 is an example of a partial shape that serves as a reference for calculating the measurement coordinate system.

[0079] As shown in FIG. 9(e), the calculation unit 313 calculates a rotation matrix 2017 and a translation vector 2018 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102, based on the shapes of the four opening corner points 2016 as seen from the distance measurement sensor 210. The calculation unit 313 calculates the measurement coordinate system 2000 by transforming the sensor coordinate system 2004 of the distance measurement sensor 210 using the rotation matrix 2017 and the translation vector 2018. This allows the calculation unit 313 to associate the position of the storage three-dimensional model with the position of the spatial three-dimensional model. Specifically, when the rotation matrix 2017 is R and the translation vector 2018 is T, the calculation unit 313 can convert a three-dimensional point x in the sensor coordinate system 2004 into a three-dimensional point X in the measurement coordinate system 2000 using Equation 1 shown below. This allows the calculation unit 313 to calculate the measurement coordinate system 2000.

[0080] X=Rx+T...Formula 1

[0081] Next, the coordinate system calculation unit 222A of the third example will be described with reference to FIGS.

[0082] Fig. 10 is a block diagram showing the configuration of a coordinate system calculation unit of the third example, and Fig. 11 is a diagram for explaining a method of calculating a measurement coordinate system by the coordinate system calculation unit of the third example.

[0083] The coordinate system calculation unit 222B has a detection unit 321, an extraction unit 322, and a calculation unit 323. In the third example, a marker 104 is placed at a specific position (for example, a position on the top surface) of the shelf 102, and the coordinate system calculation unit 222B specifies the measurement coordinate system 2000 based on the position of the marker 104. That is, the measurement coordinate system 2000 in this case is a coordinate system based on the position of the marker 104 placed on the shelf 102.

[0084] The marker 104 has, for example, a check pattern, but is not limited to a check pattern as long as it is an alignment mark (positioning mark) having a predetermined shape.

[0085] The detection unit 321 detects a marker area 2024 corresponding to the marker 104 installed on the shelf 102, as shown in (c) of Figure 11, from an image 2021, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221, as shown in (a) of Figure 11.

[0086] The extraction unit 322 extracts a pattern contour 2025, which is the contour of the check pattern, from the marker region 2024 on the image 2021, as shown in FIG. 11(d).

[0087] The calculation unit 323 calculates a rotation matrix 2026 and a translation vector 2027 that indicate the positional relationship between the distance measuring sensor 210 and the marker 104 based on the shape of the extracted pattern contour 2025. The calculation unit 323 calculates the three-dimensional positional relationship between the distance measuring sensor 210 and the shelf 102 using the rotation matrix 2026 and the translation vector 2027 and the positional relationship between the stored three-dimensional model 2022 and the marker 2023 shown in FIG. 11B, and calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 using the calculated three-dimensional positional relationship. This allows the calculation unit 323 to associate the position of the stored three-dimensional model with the position of the spatial three-dimensional model. The positional relationship between the stored three-dimensional model 2022 and the marker 2023 may be measured in advance or may be generated in advance based on design data of the shelf 102 on which the marker 104 is arranged.

[0088] Returning to FIG. 2, the model generation unit 223 will be described.

[0089] The model generation unit 223 generates a storage three-dimensional model, which is a three-dimensional model of the shelf 102 on which no luggage 103 is stored. The model generation unit 223 acquires the measurement results of the shelf 102 on which no luggage 103 is stored by the distance measurement sensor 210, and generates the storage three-dimensional model. Specific processing by the model generation unit 223 will be described later. The generated storage three-dimensional model is stored in the memory unit 225.

[0090] Here, the model generation unit 223 will be specifically described with reference to FIGS.

[0091] Fig. 12 is a block diagram showing an example of the configuration of the model generating unit Fig. 13 is a flowchart showing a process of calculating the volume of the storage space by the model generating unit.

[0092] The model generation unit 223 includes a detection unit 401 , a generation unit 402 , and a volume calculation unit 403 .

[0093] The detection unit 401 detects a shelf area corresponding to the shelf 102 from the three-dimensional spatial model measured by the distance measurement sensor 210 (S101). If the three-dimensional measurement system 200 includes multiple distance measurement sensors 210, the detection unit 401 performs the process of step S101 for each of the multiple distance measurement sensors 210. As a result, the detection unit 401 detects multiple shelf areas corresponding to the multiple distance measurement sensors 210, respectively.

[0094] When the three-dimensional measurement system 200 includes multiple ranging sensors 210, the generation unit 402 integrates the multiple shelf areas to generate a three-dimensional storage model (S102). Specifically, in order to integrate the multiple shelf areas, the generation unit 402 may align the three-dimensional point clouds using ICP (Iterative Closest Point), or may calculate the relative positional relationship between the multiple ranging sensors 210 in advance and integrate the multiple shelf areas based on the calculated relative positional relationship. The relative positional relationship may be calculated using SfM (Structure from Motion) with multiple images acquired by the multiple ranging sensors 210 as multi-viewpoint images. The multiple ranging sensors 210 may be installed based on a design drawing in which the relative positional relationship is determined.

[0095] In addition, instead of using multiple ranging sensors 210, a single ranging sensor 210 may be moved and multiple measurement results taken from multiple positions may be used to generate a three-dimensional storage model of the shelf 102 by integrating multiple shelf areas obtained from the multiple measurement results.

[0096] The storage three-dimensional model may be generated based on 3D CAD data at the time of designing the shelf 102 without using the results of measurement by the distance measurement sensor 210, or may be generated based on dimensional measurement data of the shelf 102 or equipment specification data published by the manufacturer. Also, the storage three-dimensional model may be generated by inputting manually measured dimensions of the shelf 102 into the information processing device 220.

[0097] Note that, when the three-dimensional measurement system 200 does not include multiple distance measurement sensors 210 but includes only one distance measurement sensor 210 and uses one measurement result measured from one position, the model generation unit 223 does not need to include the generation unit 402. In other words, the model generation unit 223 does not need to perform step S102.

[0098] The volume calculation unit 403 calculates the volume of the storage space 101 of the shelf 102 using the storage three-dimensional model (S103).

[0099] Returning to FIG. 2, the filling rate calculation unit 224 will be described.

[0100] The filling rate calculation unit 224 calculates the filling rate of the cargo 103 in the storage space 101 of the shelf 102. The filling rate calculation unit 224 may calculate the ratio of the volume of the cargo 103 to the volume of the storage space 101 as the filling rate, for example, using a three-dimensional spatial model, an image, and a measurement coordinate system 2000 acquired by the distance measurement sensor 210.

[0101] Here, the filling rate calculation unit 224 will be specifically described with reference to FIGS.

[0102] FIG. 14 is a block diagram showing an example of the configuration of a filling rate calculation unit. FIG. 15 is a diagram for explaining an example of a method for calculating a filling rate by the filling rate calculation unit. FIG. 15 shows an example in which the distance measurement sensor 210 faces the opening 102a of the shelf 102. The distance measurement sensor 210 is arranged on the negative Z-axis side where the opening 102a of the shelf 102 is formed, and measures the storage space 101 of the shelf 102 through the opening 102a of the shelf 102. This example is an example in which the measurement coordinate system 2000 is measured by the coordinate system calculation unit 222 of the first example. That is, in this case, the sensor coordinate system 2004 and the measurement coordinate system 2000 coincide with each other.

[0103] The filling rate calculation unit 224 includes an extraction unit 501 , an estimation unit 502 , and a calculation unit 503 .

[0104] The extraction unit 501 uses the spatial three-dimensional model 2011 and the stored three-dimensional model to extract a luggage region 2033, which is a portion of the spatial three-dimensional model corresponding to the luggage 103. Specifically, the extraction unit 501 converts the data structure of the spatial three-dimensional model 2011, which is the measurement result by the distance measuring sensor 210 acquired by the acquisition unit 221, shown in (a) of Fig. 15 into voxel data, thereby generating voxel data 2031, shown in (b) of Fig. 15. The extraction unit 501 uses the generated voxel data 2031 and a stored three-dimensional model 2032, which is the voxelized stored three-dimensional model shown in (c) of Fig. 15, to subtract the stored three-dimensional model 2032 from the voxel data 2031, thereby extracting a luggage region 2033, which is a region resulting from measurement of the luggage 103, from the voxel data 2031, shown in (d) of Fig. 15. The luggage area 2033 is an example of an object portion that corresponds to the measurement object.

[0105] The estimation unit 502 uses the extracted luggage area 2033 to estimate a luggage model 2034, which is a three-dimensional model of the luggage 103 in the storage space 101. The luggage model 2034 is an example of a three-dimensional object model. Specifically, the estimation unit 502 uses the luggage area 2033 to interpolate an area where the luggage 103 is hidden from the distance measurement sensor 210 in the Z-axis direction, which is the alignment direction of the distance measurement sensor 210 and the shelf 102, that is, the luggage area 2033 on the positive side of the Z-axis. For example, for each of the multiple voxels that make up the luggage area 2033, the estimation unit 502 determines whether the voxel is a voxel that is located on the negative side of the Z-axis relative to the farthest voxel that is located on the positive side of the Z-axis among the multiple voxels. If a voxel is located on the negative Z-axis side of the farthest voxel, and if no voxels are located on the positive Z-axis side of the voxel, the estimation unit 502 interpolates the voxel up to the same position in the Z-axis direction as the farthest voxel. In this way, the estimation unit 502 estimates a baggage model 2034 as shown in (e) of FIG.

[0106] The calculation unit 503 calculates a first filling rate of the cargo 103 in the storage space 101 using the storage three-dimensional model and the cargo model 2034. Specifically, the calculation unit 503 counts the number of voxels that make up the cargo model 2034 and multiplies the counted number by a predetermined voxel size to calculate the volume of the cargo 103. The calculation unit 503 calculates, as the first filling rate, the ratio of the calculated volume of the cargo 103 to the volume of the storage space 101 of the shelf 102 calculated by the model generation unit 223.

[0107] The distance measurement sensor 210 does not have to face the opening 102a of the shelf 102. FIG. 16 is a diagram for explaining another example of a method for calculating the filling rate by the filling rate calculation unit. FIG. 16 shows an example in which the distance measurement sensor 210 is disposed at an angle with respect to the opening 102a of the shelf 102. This example is an example in which the measurement coordinate system 2000 is measured by the coordinate system calculation unit 222A of the second example or the coordinate system calculation unit 222B of the third example. That is, in this case, the sensor coordinate system 2004 and the measurement coordinate system 2000 are different.

[0108] 16, the coordinate system used is the measurement coordinate system 2000. Using the luggage area 2033, the estimation unit 502 interpolates the luggage area 2033 to the area where the luggage 103 is hidden from the distance measurement sensor 210 in the Z-axis direction of the measurement coordinate system 2000, which is the direction in which the distance measurement sensor 210 and the shelf 102 are aligned, that is, the luggage area 2033 on the positive side of the Z-axis.

[0109] Other processing by the filling rate calculation unit 224 is the same as in the case of FIG. 15, and therefore description thereof will be omitted.

[0110] The spatial three-dimensional model and image pair used in calculating the measurement coordinate system by the coordinate system calculation unit 222 and in calculating the filling rate by the filling rate calculation unit 224 may be the results of measurements taken by the ranging sensor 210 at the same time, or may be the results of measurements taken at different times.

[0111] The ranging sensor 210 and the information processing device 220 may be communicably connected to each other via a communication network. The communication network may be a public communication network such as the Internet, or a private communication network. As a result, the spatial 3D model and image obtained by the ranging sensor 210 are transmitted from the ranging sensor 210 to the information processing device 220 via the communication network.

[0112] Furthermore, the information processing device 220 may acquire the spatial 3D model and images from the ranging sensor 210 without using a communication network. For example, the spatial 3D model and images may be temporarily stored in an external storage device such as a hard disk drive (HDD) or a solid state drive (SSD) from the ranging sensor 210, and the information processing device 220 may acquire the spatial 3D model and images from the external storage device. Furthermore, the external storage device may be a cloud server.

[0113] The information processing device 220 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. The functions of each processing unit of the information processing device 220 may be realized by software or hardware.

[0114] Next, the operation of the information processing device 220 will be described.

[0115] FIG. 17 is a flowchart of a filling rate measurement method performed by an information processing device.

[0116] The information processing device 220 acquires a spatial three-dimensional model from the distance measurement sensor 210 (S111). At this time, the information processing device 220 may further acquire an image of the measurement target from the distance measurement sensor 210.

[0117] The information processing device 220 acquires the stored three-dimensional model stored in the storage unit 225 (S112).

[0118] The information processing device 220 calculates a measurement coordinate system based on the shape of the opening 102a of the shelf 102 (S113). Step S113 is a process performed by the coordinate system calculation unit 222.

[0119] The information processing device 220 uses the voxel data 2031 of the spatial three-dimensional model 2011 and the stored three-dimensional model 2032 of the stored three-dimensional model to extract a baggage area 2033 corresponding to the baggage 103 from the voxel data 2031 (S114). Step S114 is a process performed by the extraction unit 501 of the filling rate calculation unit 224.

[0120] The information processing device 220 uses the extracted luggage area 2033 to estimate a luggage model 2034, which is a three-dimensional model of the luggage 103 in the storage space 101 (S115). Step S115 is a process performed by the estimation unit 502 of the filling rate calculation unit 224.

[0121] The information processing device 220 calculates a first filling rate of the luggage 103 in the storage space 101 using the storage three-dimensional model and the luggage model 2034 (S116). Step S116 is a process performed by the calculation unit 503 of the filling rate calculation unit 224.

[0122] FIG. 18 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the first example.

[0123] The coordinate system calculation unit 222 sequentially acquires in real time images 2001 that are measurement results obtained by the distance measurement sensor 210 and acquired by the acquisition unit 221, and sequentially superimposes adjustment markers 2002 on the sequentially acquired images 2001 (S121). Step S121 is a process performed by the auxiliary unit 301 of the coordinate system calculation unit 222.

[0124] The coordinate system calculation unit 222 acquires the position and orientation of the distance measurement sensor 210 (S122). Step S121 is a process performed by the auxiliary unit 301 of the coordinate system calculation unit 222.

[0125] The coordinate system calculation unit 222 specifies a sensor coordinate system 2004 of the distance measurement sensor 210 using the position and orientation of the distance measurement sensor 210 when the four adjustment markers 2002 are aligned with the positions of the four corners of the opening 102a, and calculates the measurement coordinate system 2000 using the specified sensor coordinate system 2004 (S123). Step S123 is a process performed by the calculation unit 302 of the coordinate system calculation unit 222.

[0126] FIG. 19 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the second example.

[0127] The coordinate system calculation unit 222A detects a shelf area 2014 corresponding to the shelf 102 using the space three-dimensional model 2011, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221, and the storage three-dimensional model 2012 (S121A). Step S121A is a process performed by the detection unit 311 of the coordinate system calculation unit 222A.

[0128] The coordinate system calculation unit 222A uses the position information 2013 in the storage three-dimensional model 2012 to extract four opening corner points 2016, which are the positions of the four corners of the opening 2015 in the shelf area 2014 (S122A). Step S122A is a process performed by the extraction unit 312 of the coordinate system calculation unit 222A.

[0129] The coordinate system calculation unit 222A calculates a rotation matrix 2017 and a translation vector 2018 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102, based on the shapes of the four opening end points 2016 as seen from the distance measurement sensor 210. Then, the coordinate system calculation unit 222A calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 of the distance measurement sensor 210 using the rotation matrix 2017 and the translation vector 2018 (S123A). Step S123A is a process performed by the calculation unit 313 of the coordinate system calculation unit 222A.

[0130] FIG. 20 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit of the third example.

[0131] The coordinate system calculation unit 222B detects the marker region 2024 from the image 2021, which is the measurement result by the distance measurement sensor 210 acquired by the acquisition unit 221 (S121B). Step S121B is a process performed by the detection unit 321 of the coordinate system calculation unit 222B.

[0132] The coordinate system calculation unit 222B extracts the pattern contour 2025 from the marker region 2024 on the image 2021 (S122B). Step S122B is a process performed by the extraction unit 322 of the coordinate system calculation unit 222B.

[0133] The coordinate system calculation unit 222B calculates a rotation matrix 2026 and a translation vector 2027 that indicate the positional relationship between the distance measurement sensor 210 and the marker 104, based on the shape of the extracted pattern contour 2025. Then, the coordinate system calculation unit 222B calculates the three-dimensional positional relationship between the distance measurement sensor 210 and the shelf 102 using the rotation matrix 2026, the translation vector 2027, and the positional relationship between the storage three-dimensional model 2022 and the marker 2023, and calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 using the calculated three-dimensional positional relationship (S123B). Step S123B is a process performed by the calculation unit 323 of the coordinate system calculation unit 222B.

[0134] The filling rate calculated by the information processing device 220 may be output from the information processing device 220. The filling rate may be displayed on a display device (not shown) provided in the information processing device 220, or may be transmitted to an external device different from the information processing device 220. For example, the calculated filling rate may be output to a luggage conveying system and used to control the luggage conveying system.

[0135] According to the filling rate measurement method of this embodiment, a baggage model 2034 of the baggage 103 is estimated using a baggage area 2033 extracted using a spatial three-dimensional model obtained by measuring the shelf 102 in a state where the baggage 103 is stored and a storage three-dimensional model of the shelf 102 in which no baggage 103 is stored. This makes it possible to easily calculate the first filling rate of the baggage 103 in the storage space 101 simply by measuring the shelf 102 in a state where the baggage 103 is stored.

[0136] Furthermore, in the filling rate measurement method, the baggage model 2034 is estimated based on a three-dimensional coordinate system based on the shape of a part of the shelf 102. This makes it possible to reduce the amount of processing required to estimate the baggage model 2034.

[0137] Furthermore, in the filling rate measurement method, the luggage model 2034 is estimated based on a three-dimensional coordinate system based on the shape of only a portion of the shelf 102. The shape of only a portion of the storage section, which is easy to extract on an image, can be used to calculate the measurement coordinate system. This makes it possible to improve the processing speed for estimating the luggage model and the calculation accuracy of the measurement coordinate system.

[0138] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system is a three-dimensional Cartesian coordinate system having a Z axis, and the estimation involves interpolating the Z-axis plus direction side, opposite to the Z-axis minus direction, of the luggage area 2033, to estimate the luggage model 2034. This makes it possible to effectively reduce the amount of processing required to estimate the luggage model 2034.

[0139] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system is a coordinate system based on the shape of opening 102a of shelf 102. Therefore, the coordinate system based on the shape of opening 102a of shelf 102 can be easily calculated, and baggage model 2034 can be estimated based on the calculated coordinate system.

[0140] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system may be a coordinate system based on markers 104 installed on shelves 102. Therefore, the coordinate system based on markers 104 can be easily calculated, and baggage model 2034 can be estimated based on the calculated coordinate system.

[0141] (Embodiment 2) The information processing device according to the second embodiment has a different configuration of the coordinate system calculation unit compared to the information processing device according to the first embodiment, which will be specifically described below.

[0142] Fig. 21 is a block diagram showing the configuration of a coordinate system calculation unit according to embodiment 2. Fig. 22 is a block diagram showing the configuration of an extraction unit included in the coordinate system calculation unit according to embodiment 2. Fig. 23 is a diagram for explaining a method for extracting opening end points by the extraction unit according to embodiment 2.

[0143] The coordinate system calculation unit 222C differs from the coordinate system calculation unit 222A in that it has an extraction unit 321C instead of the detection unit 311 and extraction unit 312 that the coordinate system calculation unit 222A has.

[0144] Similar to the detection unit 311 and the extraction unit 312, the extraction unit 321C extracts four opening end points 2016, which are the positions of the four corners of the opening 2015 in the shelf region 2014, using the measurement results of the distance measurement sensor 210 acquired by the acquisition unit 221 and the storage three-dimensional model 2012. Note that the extraction unit 321C is only required to be able to identify the four opening end points 2016 of the opening 2015 in the shelf region 2014, and is not required to perform processing to extract the four opening end points 2016. The extraction unit 321C includes a line segment detection unit 1331, an opening extraction unit 1332, and an end point calculation unit 1333.

[0145] The measurement results by the ranging sensor 210 in the second embodiment include an RGB image and a depth image. The RGB image is a two-dimensional image captured by a camera built into the ranging sensor 210. The RGB image is an image in which the entire opening 2015 is reflected, that is, a two-dimensional image of the opening 2015. Thus, the RGB image is an example of a two-dimensional image of the opening 2015 generated by measurement in a specific direction from the position of the ranging sensor 210. The RGB image is an image obtained by capturing (measuring) an image using a camera placed at the position of the ranging sensor 210 while facing a specific direction. The specific direction is a direction (e.g., an imaging direction) that indicates the attitude of the camera when capturing the RGB image, such as a direction from the position of the ranging sensor 210 toward the opening 2015. Note that the specific direction does not have to coincide with the direction from the position of the ranging sensor 210 toward the opening 2015, and may be the imaging direction of the camera when the range captured by the camera includes the opening 2015. The position and specific direction of the distance measurement sensor 210 may be used as the position and orientation of the camera (external parameters of the camera), respectively. The position and orientation of the camera may also be set in advance. The external parameters of the camera are position and orientation information corresponding to the RGB image.

[0146] The depth image is an image generated by the ranging sensor 210. The depth image is a two-dimensional image having, as pixel values, distances to measurement targets including the opening 2015 in the ranging direction (depth direction) measured by the ranging sensor 210. In other words, the depth image is another example of a two-dimensional image of the opening 2015. The depth image is an image generated based on, for example, measurement results by the ranging sensor 210. The depth image may be generated from measurement results of only an area including the opening 2015 and its vicinity, or may be generated from a spatial three-dimensional model or measurement results that served as a basis for generating the spatial three-dimensional model. In this case, remeasurement of the opening 2015 by the ranging sensor 210 is omitted. The specific direction is a direction (e.g., a ranging direction) that indicates the orientation of the ranging sensor 210 when the measurement results by the ranging sensor 210 that served as the basis for generating the depth image were measured, for example, a direction from the position of the ranging sensor 210 toward the opening 2015. The specific direction does not have to coincide with the direction from the position of the ranging sensor 210 toward the opening 2015, but may be any direction in which the ranging sensor 210 measures distance when the measurement range of the ranging sensor 210 includes the opening 2015. The position and the specific direction of the ranging sensor 210 indicate the position and orientation of the ranging sensor 210, respectively, and are extrinsic parameters of the ranging sensor 210. The extrinsic parameters of the ranging sensor 210 are position and orientation information corresponding to the depth image.

[0147] The two-dimensional image is not limited to an RGB image or a depth image, but may also be a grayscale image, an infrared image, or the like.

[0148] The specific direction of the camera and the specific direction of the ranging sensor 210 may be the same or different from each other. The extrinsic parameters of the camera may be the same as the extrinsic parameters of the ranging sensor 210. Therefore, the extrinsic parameters of the ranging sensor 210 may be used as the extrinsic parameters of the camera.

[0149] The line segment detection unit 1331 executes processing to detect line segments in an RGB image 2101 shown in (a) of Fig. 23. By detecting line segments from the RGB image 2101, the line segment detection unit 1331 generates a line segment image 2102 including a line segment 2103 in the RGB image 2101, as shown in (b) of Fig. 23.

[0150] Similarly, the line segment detection unit 1331 executes processing to detect line segments in the depth image 2111 shown in (c) of Fig. 23. By detecting line segments from the depth image 2111, the line segment detection unit 1331 generates a line segment image 2112 including a line segment 2113 in the depth image 2111, as shown in (d) of Fig. 23.

[0151] In the process of detecting lines, the line detection unit 1331 detects edges based on the difference in pixel values ​​between adjacent pixels in each image, and detects lines by detecting a direction perpendicular to the direction of the detected edge.

[0152] The opening extraction unit 1332 uses the line segment images 2102 and 2112 and the stored three-dimensional model 2012 shown in (e) of FIG. 23 to extract a line segment 2122 indicating the shape of the opening 2015 from the line segment images 2102 and 2112, as shown in (f) of FIG. 23, and generates a line segment image 2121 including the line segment 2122. Specifically, the opening extraction unit 1332 extracts the line segment 2122 by performing pattern matching on the line segment image obtained by combining the line segment images 2102 and 2112, using the shape of the opening in the stored three-dimensional model 2012 as a template. The line segment images 2102 and 2112 may be combined by aligning them using external parameters of the camera and external parameters of the ranging sensor 210, and then arranging multiple line segments included in one of the line segment images 2102 and 2112 at the positions of the multiple line segments in the other line segment image. The line segment image 2121 may be an image including only line segments 2122 that indicate the shapes of the openings 2015. In the line segment image 2121 of Fig. 23(f), it is determined that there are four shapes that match the stored three-dimensional model 2012 in the line segment images 2102 and 2112, and therefore line segments 2122 that include the shapes of the four openings 2015 are extracted.

[0153] As shown in (g) of FIG. 23, the endpoint calculation unit 1333 extracts four opening endpoints 2016, which are the positions of the four corners of each opening 2015 in the shelf area 2014, based on the external parameters of the camera and the external parameters of the distance measurement sensor 210, and the line segments 2122 included in the line segment image 2121. The external parameters of the camera and the external parameters of the distance measurement sensor 210 may be stored in advance in the storage unit 225 of the information processing device 220. In the example of FIG. 23, the shapes of the four openings 2015 are detected, and therefore 16 opening endpoints 2016 are extracted. In (g) of FIG. 23, some of the opening endpoints 2016 overlap, and therefore nine opening endpoints 2016 are shown.

[0154] Note that since the position of the opening 2015 can be identified once the line segment 2122 including the shape of the opening 2015 is extracted by the opening extraction unit 1332, the processing by the endpoint calculation unit 1333 does not necessarily have to be performed. The shape of the opening 2015 may be defined by the line segment 2122, by the opening endpoints 2016, or by a combination of the line segment 2122 and the opening endpoints 2016. If the shape of the opening 2015 is a rectangle, it may be defined by four line segments, by opening endpoints indicating four vertices, or by a combination of line segments and opening endpoints. In other words, if the shape of the opening is a polygon, it may be defined by the line segments that constitute the sides of the polygon, by the vertices of the polygon, or by a combination of line segments and vertices. Furthermore, if the shape of the opening is a circle, including an ellipse and a perfect circle, it may be defined by the shape of the curved outer edge of the circle.

[0155] In this way, the extraction unit 321C identifies line segments that indicate the shape of the opening 2015 from the RGB image 2101 and the depth image 2111. Then, the extraction unit 321C calculates the position of the opening 2015 in three-dimensional space based on the position of the distance measurement sensor 210, the specific direction, and the identified shape of the opening 2015 (i.e., the line segment 2122 in the line segment image 2121).

[0156] The processing performed by the calculation unit 313 after the four opening end points 2016 are extracted is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0157] 23 has been described as an example in which the line segments of the opening 2015 are extracted based on both the RGB image 2101 and the depth image 2111 as two-dimensional images, but the present invention is not limited to this. For example, the line segments of the opening 2015 may be extracted based on either the RGB image 2101 or the depth image 2111.

[0158] The accuracy of line segment extraction from the RGB image 2101 is affected by the environment, such as the brightness (illuminance) around the opening 2015. The RGB image 2101 contains more information than the depth image 2111, and therefore many line segments are detected.

[0159] On the other hand, the accuracy of line segment extraction from the depth image 2111 is less affected by the environment (brightness) than the RGB image 2101. If distance information cannot be obtained by distance measurement, the depth image 2111 may have missing parts in areas where distance information cannot be obtained. As such, the RGB image 2101 and the depth image 2111 have different characteristics, and therefore, processing for extracting line segments of the opening 2015 may be performed according to the characteristics of each image.

[0160] For example, in the process of extracting line segments, the accuracy of line segment extraction may be improved by combining a line segment image 2102 obtained from the RGB image 2101 with a line segment image 2112 obtained from the depth image 2111 so that the RGB image 2101 is given priority over the depth image 2111 if the brightness near the opening 2015 exceeds a predetermined illuminance, and extracting the line segments of the opening 2015 from the combined result. Conversely, in the process of extracting line segments, the accuracy of line segment extraction may be improved by combining a line segment image 2102 obtained from the RGB image 2101 with a line segment image 2112 obtained from the depth image 2111 so that the depth image 2111 is given priority over the RGB image 2101 if the brightness near the opening 2015 is equal to or lower than a predetermined illuminance.

[0161] The brightness near the aperture 2015 may be estimated from the pixel values ​​of the pixels included in the RGB image 2101 .

[0162] Furthermore, the brightness near the opening 2015 may decrease as the packing rate of the packages 103 in the storage space 101 of the shelf 102 increases. In this case, the packing rate may be calculated once, and a two-dimensional image to be prioritized in the line segment extraction process may be determined based on the calculated packing rate. That is, in the line segment extraction process, if the calculated packing rate is equal to or lower than a predetermined packing rate, the line segment image 2102 obtained from the RGB image 2101 may be combined with the line segment image 2112 obtained from the depth image 2111 so that the RGB image 2101 is prioritized over the depth image 2111, and the line segments of the opening 2015 may be extracted from the combination result, thereby improving the accuracy of line segment extraction. Conversely, in the process of extracting lines, the accuracy of extracting the lines of the opening 2015 may be improved by combining a line segment image 2102 obtained from the RGB image 2101 with a line segment image 2112 obtained from the depth image 2111 so that the depth image 2111 is given priority over the RGB image 2101 if the filling rate calculated once exceeds a predetermined filling rate.

[0163] Furthermore, in the process of extracting line segments, if the spatial three-dimensional model has more missing parts than a predetermined threshold, it is difficult to measure the shape of the opening from the depth image 2111. Therefore, the line segment image 2102 obtained from the RGB image 2101 may be combined with the line segment image 2112 obtained from the depth image 2111 so that the RGB image 2101 is prioritized over the depth image 2111, and the line segments of the opening 2015 may be extracted from the combined result, thereby improving the accuracy of line segment extraction. For example, distance information of an opening 2015 located farther than a predetermined distance from the ranging sensor 210 in a direction perpendicular to the direction in which the ranging sensor 210 measures distance may not be obtained with sufficient accuracy or may be missing because the laser light from the ranging sensor 210 is unlikely to be reflected. Therefore, the line segment image 2102 obtained from the RGB image 2101 may be combined with the line segment image 2112 obtained from the depth image 2111 so that the RGB image 2101 is prioritized over the depth image 2111. Conversely, in the process of extracting lines, if the number of missing parts in the spatial three-dimensional model is below a predetermined threshold, the line segment image 2102 obtained from the RGB image 2101 may be combined with the line segment image 2112 obtained from the depth image 2111 so that the depth image 2111 is prioritized over the RGB image 2101, thereby improving the accuracy of extracting the lines of the opening 2015.

[0164] In the above, the line segment image 2102 obtained from the RGB image 2101 is combined with the line segment image 2112 obtained from the depth image 2111 so that the RGB image 2101 is given priority over the depth image 2111, and the line segments of the opening 2015 are extracted from the combined result. However, specifically, the following processing may be performed.

[0165] A first example of this process will be described. When the RGB image 2101 is given priority over the depth image 2111, the result of this combination may be only the line segment image 2102. In this case, it is not necessary to generate the line segment image 2112 from the depth image 2111.

[0166] Next, a second example will be described. In the second example, in the process of extracting line segments, an evaluation value indicating likelihood (accuracy) may be assigned to each extracted line segment. That is, in this case, an evaluation value is assigned to each of the multiple line segments included in the line segment images 2102 and 2112. In the combination of the line segment image 2102 and the line segment image 2112, the evaluation values ​​of the line segments of the line segment images 2102 and 2112 are weighted and added with a weight according to the illuminance around the opening 2015. In this way, the line segment images 2102 and 2112 are integrated. If the weighting at this time prioritizes the RGB image 2101 over the depth image 2111, the weight for the line segment image 2102 in the weighted addition is set to be greater than the weight for the line segment image 2112.

[0167] Then, line segments in the integrated image having evaluation values ​​equal to or greater than a threshold are extracted as candidates for line segments of the opening 2015, and pattern matching is performed on the extracted candidate line segments to extract the line segments of the opening 2015. The evaluation value indicating the likelihood may be higher the longer the line segment, or may be higher the greater the difference in pixel values ​​between two adjacent pixels with an edge as the boundary when detecting the line segment, or the difference in pixel values ​​between two pixels belonging to two adjacent regions with the edge as the boundary.

[0168] In addition, when combining line segment image 2102 obtained from RGB image 2101 with line segment image 2112 obtained from depth image 2111 so that depth image 2111 takes priority over RGB image 2101, this can be explained by swapping RGB image 2101 and depth image 2111, and swapping line segment image 2102 and line segment image 2112, in the explanation of when RGB image 2101 takes priority over depth image 2111.

[0169] Next, a description will be given of the operation of the information processing device according to embodiment 2. Since the information processing device according to embodiment 2 has a different configuration of the coordinate system calculation unit compared to the information processing device according to embodiment 1, the operation (S113) of the coordinate system calculation unit will be described.

[0170] FIG. 24 is a flowchart of the process (S113) of calculating the measurement coordinate system by the coordinate system calculation unit according to the second embodiment.

[0171] The coordinate system calculation unit 222C detects line segments from the two-dimensional image (S1121). Specifically, the coordinate system calculation unit 222C detects line segments from the RGB image 2101 to generate a line segment image 2102 including a line segment 2103 in the RGB image 2101. The coordinate system calculation unit 222C also detects line segments from the depth image 2111 to generate a line segment image 2112 including a line segment 2113 in the depth image 2111. Step S1121 is a process performed by the line segment detection unit 1331 of the extraction unit 321C of the coordinate system calculation unit 222C.

[0172] The coordinate system calculation unit 222C extracts the line segment of the opening 2015 from the detected line segments (S1122). Specifically, the coordinate system calculation unit 222C uses the line segment images 2102, 2112 and the stored three-dimensional model 2012 to extract a line segment 2122 indicating the shape of the opening 2015 from the line segment images 2102, 2112, and generates a line segment image 2121 including the line segment 2122. Step S1122 is a process performed by the opening extraction unit 1332 of the extraction unit 321C of the coordinate system calculation unit 222C.

[0173] The coordinate system calculation unit 222C extracts four opening end points 2016, which are the positions of the four corners of each opening 2015 in the shelf region 2014, based on the position of the distance measurement sensor 210, the direction in which the RGB image 2101 and the depth image 2111 are measured (i.e., a specific direction), and the line segment 2122 included in the line segment image 2121 (S1123). Step S1123 is a process performed by the end point calculation unit 1333 of the extraction unit 321C of the coordinate system calculation unit 222C.

[0174] The coordinate system calculation unit 222C calculates a rotation matrix 2017 and a translation vector 2018 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102, based on the shapes of the four opening end points 2016 as seen from the distance measurement sensor 210. Then, the coordinate system calculation unit 222A calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 of the distance measurement sensor 210 using the rotation matrix 2017 and the translation vector 2018 (S1124). Step S1124 is a process performed by the calculation unit 313 of the coordinate system calculation unit 222C. In other words, this process is similar to the process performed by the calculation unit 313 of the coordinate system calculation unit 222A. This allows the coordinate system calculation unit 222C to associate the position of the stored three-dimensional model with the position of the spatial three-dimensional model.

[0175] Note that the processing of step S1123 does not necessarily have to be performed. If the processing of step S1123 is not performed, in step S1124, the coordinate system calculation unit 222A calculates a rotation matrix 2017 and a translation vector 2018 that indicate the positional relationship between the distance measurement sensor 210 and the shelf 102, based on a line segment 2122 that indicates the shape of the opening 2015. Then, the coordinate system calculation unit 222A calculates the measurement coordinate system 2000 by converting the sensor coordinate system 2004 of the distance measurement sensor 210 using the rotation matrix 2017 and the translation vector 2018.

[0176] (Variation 1) In the information processing device 220 according to the above embodiment, the ratio of the volume of the cargo 103 stored in the storage space 101 to the capacity of the storage space 101 is calculated as the filling rate, but the present invention is not limited to this.

[0177] FIG. 25 is a diagram for explaining a method for calculating the filling rate.

[0178] In (a) and (b) of Figures 25, the storage space 101 of the shelf 102 has a capacity to store exactly 16 pieces of luggage 103. As shown in (a) of Figure 25, when eight pieces of luggage 103 are arranged with no gaps, eight more pieces of luggage 103 can be stored in the available storage space 101. On the other hand, as shown in (b) of Figure 25, when luggage is arranged with gaps, in order to store eight pieces of luggage 103 in the remaining space of the storage space 101, it is necessary to move the luggage 103 that is already stored. If the luggage 103 is stored in the remaining space of the storage space 101 without moving the luggage 103 that is already stored, only six pieces of luggage 103 can be stored.

[0179] 25(a) and 25(b), the amount of cargo 103 that can be stored in the remaining space of storage space 101 is different, but the filling rate is calculated to be the same 50% in both cases. For this reason, it is conceivable to calculate the filling rate in consideration of the space that can actually be stored, in accordance with the shape of the remaining space of storage space 101.

[0180] Fig. 26 is a block diagram showing an example of the configuration of a calculation unit of a filling rate calculation unit according to Modification 1. Fig. 27 is a flowchart of a filling rate calculation process of the calculation unit of a filling rate calculation unit according to Modification 1.

[0181] As shown in FIG. 26, the calculation unit 503 includes a baggage volume calculation unit 601 , an area division unit 602 , an expected baggage measurement unit 603 , an area estimation unit 604 , and a calculation unit 605 .

[0182] The luggage volume calculation unit 601 calculates the luggage volume, which is the volume of the luggage 103, from the luggage model 2034 (S131). The luggage volume calculation unit 601 calculates the volume of the luggage 103 stored in the storage space 101 in the same manner as in the first embodiment.

[0183] Next, the area dividing unit 602 divides the storage space 101 of the spatial three-dimensional model 2011 into an occupied area 2041 occupied by the luggage 103 and an empty area 2042 not occupied by the luggage 103 (S132).

[0184] Next, the planned package measurement unit 603 calculates the volume of one package to be stored (S133). When there are multiple types of packages with different shapes and sizes to be stored as shown in (c) of Figure 25, the planned package measurement unit 603 calculates the volume of one package for each type. For example, the planned package measurement unit 603 calculates the volume of package 103a, package 103b, and package 103c.

[0185] Next, the area estimation unit 604 estimates how to place the luggage 103 to be stored in the empty area 2042 so that the maximum number of luggage 103 to be stored can be stored, and estimates the number of luggage 103 to be stored in that case. In other words, the area estimation unit 604 estimates the maximum number of luggage 103 to be stored that can be stored in the empty area 2042. The area estimation unit 604 calculates the storage capacity of the empty area 2042 by multiplying the volume of one piece of luggage by the number of luggage that can be stored (S134).

[0186] When there are multiple types of luggage, the area estimation unit 604 may estimate how many luggage of each type can be stored, or may estimate how many luggage of each type can be stored. When storing multiple types of luggage, the area estimation unit 604 calculates the storable volume of the empty area 2042 by integrating the volume of one piece of luggage for each type by the number of luggage of that type that can be stored. For example, if the area estimation unit 604 estimates that n1 luggage 103a, n2 luggage 103b, and n3 luggage 103c can be stored, the area estimation unit 604 calculates the storable volume of the empty area 2042 by integrating the first volume obtained by multiplying the volume of the luggage 103a by n1, the second volume obtained by multiplying the volume of the luggage 103b by n2, and the third volume obtained by multiplying the volume of the luggage 103c by n3. Note that n1, n2, and n3 are each an integer greater than or equal to 0.

[0187] The calculation unit 605 calculates the filling rate by applying the volume of the stored cargo and the storable volume to the following formula 2 (S135).

[0188] Filling rate (%) = (volume of stored cargo) / (volume of stored cargo + storage capacity) × 100... Equation 2

[0189] In this way, the filling rate calculation unit 224 may calculate the ratio of the volume of the luggage 103 stored in the storage space 101 to the volume of the space in the storage space 101 that can store the luggage 103 as the filling rate.

[0190] This makes it possible to calculate a first filling rate for appropriately determining how much baggage 103 can be stored in the available space in the storage space 101.

[0191] In addition, if the type of cargo stored in the storage space 101 is given in advance, the quantity of the stored cargo may be calculated by dividing the volume of the previously-given cargo type by the volume of the cargo already stored. For example, the type of cargo stored in the storage space 101 may be stored in the memory unit 225 of the information processing device 220 together with an ID identifying the shelf 102 that has the storage space 101. The memory unit 225 may store storage information that associates the ID identifying the shelf 102 with the type of cargo stored in the storage space 101 of the shelf 102. Furthermore, the memory unit 225 of the information processing device 220 may store cargo information that associates the type of cargo with the volume of each type of cargo. The volume of each type of cargo in the cargo information is calculated based on the size of cargo commonly used in the distribution industry. The storage information and cargo information are, for example, tables. As a result, the information processing device 220 can identify the type of luggage 103 stored in the storage space 101 of the shelf 102 and the volume of that type of luggage based on the storage information stored in the memory unit 225, and calculate the quantity of luggage stored by dividing the identified volume of the luggage by the calculated volume of the luggage already stored.

[0192] The calculated quantity of luggage may be output together with the filling rate. For example, if the stored luggage is luggage 103a, the quantity of the stored luggage can be calculated by dividing the volume of luggage 103a by the volume of the luggage that has already been stored.

[0193] (Variation 2) In the information processing device 220 according to the above embodiment, the filling rate of the luggage 103 in the storage space 101 of one shelf 102 is calculated, but the filling rate of the luggage 103 in the storage space 101 of two or more shelves 102 may also be calculated.

[0194] Fig. 28 is a diagram showing an example of storing two or more shelves in a storage space such as the loading platform of a truck. Fig. 29 is a table showing the relationship between the shelves stored in the storage space in the loading platform and their filling rates.

[0195] As shown in Fig. 28, a loading platform 106 having a storage space 105 stores a plurality of car dollies 112. The loading platform 106 may be, for example, a van-type loading platform of a truck. The loading platform 106 is an example of a second storage section. The second storage section is not limited to the loading platform 106, and may also be a container or a warehouse.

[0196] The storage space 105 is an example of a second storage space. The storage space 105 has a volume large enough to store a plurality of car bogies 112. In the second modification, the storage space 105 can store six car bogies 112. Because the storage space 105 can store a plurality of car bogies 112, the storage space 105 is larger than the storage space 111.

[0197] The car truck 112 has a storage space 111 capable of storing a plurality of luggage 103. The car truck 112 is an example of a storage unit. The storage unit in Modification 2 is not limited to the car truck 112 or a roll box, as long as it is a movable container. The storage space 111 is an example of a first storage space. Note that the storage space 105 may store the shelf 102 described in the first embodiment.

[0198] The plurality of luggage 103 is not stored directly on the loading platform 106, but is stored on a plurality of car trucks 112. Then, the car trucks 112 storing the plurality of luggage 103 are stored on the loading platform 106.

[0199] In this case, the configuration of the calculation unit 503 of the filling rate calculation unit 224 will be described.

[0200] Fig. 30 is a block diagram showing an example of the configuration of a calculation unit of a filling rate calculation unit according to Modification 2. Fig. 31 is a flowchart of a filling rate calculation process of the calculation unit of a filling rate calculation unit according to Modification 2.

[0201] As shown in FIG. 30, the calculation unit 503 according to the second modification includes an acquisition unit 701, a counting unit 702, and a calculation unit 703.

[0202] The acquisition unit 701 acquires the number of car trucks 112 that can be stored on the loading platform 106 (S141). In the case of the second modification, the maximum number of car trucks 112 that can be stored on the loading platform 106 is six, so six is ​​acquired.

[0203] The counting unit 702 counts the number of car trucks 112 stored on the loading platform 106 (S142). When the car truck 112 shown in FIG. 29 is stored on the loading platform 106, the counting unit 702 counts the number of car trucks 112 as three.

[0204] The calculation unit 703 calculates a second filling rate, which is a filling rate of one or more car carts 112 on the loading platform 106 (S143). Specifically, the calculation unit 703 may calculate, as the second filling rate, the ratio of the number of car carts 112 stored on the loading platform 106 to the maximum number of car carts 112 that can be stored on the loading platform 106. For example, since a maximum of six car carts 112 can be stored on the loading platform 106 and three of these car carts 112 are stored on the loading platform 106, the calculation unit 703 calculates 50% as the second filling rate.

[0205] The calculation unit 703 may calculate the filling rate of the luggage 103 for each of the one or more car carts 112 stored on the loading platform 106, and use the calculated filling rate to calculate the filling rate of the luggage 103 for the second storage space. Specifically, the calculation unit 703 may calculate the average of the filling rates of the luggage 103 for the car carts 112 as the filling rate of the luggage 103 for the second storage space. In this case, when there is surplus space in the storage space 105 of the loading platform 106 that can store the car carts 112, the calculation unit 703 may calculate the average by setting the filling rate of the number of car carts 112 that can be stored in the surplus space that can store the car carts 112 to 0%.

[0206] For example, if the filling rates of the three car carts 112 shown in Figure 29 are 70%, 30%, and 20%, respectively, and a maximum of six car carts 112 can be stored in the loading platform 106, the filling rates of the six car carts 112 may be set to 70%, 30%, 20%, 0%, 0%, and 0%, respectively, and the average obtained may be 20%, which may be calculated as the filling rate of luggage 103 in the second storage space.

[0207] Therefore, the second filling rate when one or more car trucks 112 are stored in the storage space 105 can be calculated appropriately.

[0208] (Variation 3) Next, Modification 3 will be described.

[0209] FIG. 32 is a diagram for explaining the configuration of a car bogie according to Modification 3. In FIG.

[0210] Figure 32(a) is a diagram showing the car truck 112 with the opening / closing unit 113 in a closed state, and Figure 32(b) is a diagram showing the car truck 112 with the opening / closing unit 113 in an open state.

[0211] The car bogie 112 according to the third modification has an opening / closing unit 113 that opens and closes the opening 112a. The opening / closing unit 113 is a lattice- or mesh-like cover having a plurality of through-holes 113a. Therefore, even when the opening / closing unit 113 of the car bogie 112 is in a closed state, the distance measurement sensor 210 can measure the three-dimensional shape of the interior of the storage space 111 of the car bogie 112 through the plurality of through-holes 113a and the opening 112a.

[0212] This is because the electromagnetic waves emitted by distance measurement sensor 210 pass through the multiple through holes 113a and openings 112a. Note that in the case of distance measurement sensor 210A as well, the infrared pattern emitted by distance measurement sensor 210A passes through the multiple through holes 113a and openings 112a, so even if opening / closing part 113 of car bogie 112 is in the closed state, the three-dimensional shape of the interior of storage space 111 of car bogie 112 can be measured through the multiple through holes 113a and openings 112a. Also in the case of distance measurement sensor 210B, the two cameras 211B, 212B can photograph the interior of storage space 111 through the multiple through holes 113a and openings 112a, so the three-dimensional shape of the interior of storage space 111 of car bogie 112 can be measured.

[0213] Therefore, the information processing device 220 can determine whether or not luggage 103 is stored in the storage space 111. However, when the opening / closing unit 113 is in the closed state, it is difficult to obtain the correct filling rate unless the method for calculating the filling rate is switched to a different method from when the opening / closing unit 113 is in the open state or when the opening / closing unit 113 is not present. For this reason, the filling rate calculation unit 224 according to the third modification calculates the filling rate using the first method when the opening / closing unit 113 is in the open state, and calculates the filling rate using the second method when the opening / closing unit 113 is in the closed state.

[0214] 33 is a block diagram showing an example of the configuration of a filling rate calculation unit according to Modification 3. FIG. 34 is a flowchart of a filling rate calculation process of the filling rate calculation unit according to Modification 3.

[0215] As shown in FIG. 33, the filling rate calculation unit 224 according to the third modification includes a detection unit 801, a switching unit 802, a first filling rate calculation unit 803, and a second filling rate calculation unit 804.

[0216] The detection unit 801 detects the open / close state of the opening / closing unit 113 using the spatial three-dimensional model (S151). Specifically, the detection unit 801 detects that the opening / closing unit 113 is in the closed state when a three-dimensional point cloud exists at each of positions inside and outside the storage space 111 in the front-to-rear direction of the area of ​​the opening 112a of the car truck 112 (that is, the alignment direction of the distance measurement sensor 210 and the car truck 112) using the spatial three-dimensional model. The detection unit 801 detects that the opening / closing unit 113 is in the open state when a three-dimensional point cloud exists only inside the storage space 111.

[0217] The switching unit 802 determines whether the opening / closing unit 113 is in the open state or the closed state (S152), and switches the next process depending on the determination result.

[0218] When the switching unit 802 determines that the opening / closing unit 113 is in the open state (open state in S152), the first filling rate calculation unit 803 calculates the filling rate by a first method (S153). Specifically, the first filling rate calculation unit 803 calculates the filling rate of the car bogie 112 by performing processing similar to the processing by the filling rate calculation unit 224 in the first embodiment.

[0219] If the switching unit 802 determines that the open / close unit 113 is in the closed state (closed state in S152), the second filling rate calculation unit 804 calculates the filling rate by the second method (S154). Details of the second method will be described with reference to FIG.

[0220] FIG. 35 is a diagram illustrating an example of a second method for calculating the filling rate.

[0221] Consider the case where a spatial three-dimensional model 2051 is acquired as shown in FIG. 35(a).

[0222] Figure 35(b) is an enlarged view of region R2 in the spatial three-dimensional model 2051. As shown in Figure 35(b), the second filling rate calculation unit 804 divides region R2 into a second portion where the opening / closing part 113 is detected and a first portion where the baggage 103 is detected.

[0223] The first portion is an area including a three-dimensional point cloud on the back side of the area of ​​the opening 112a. The first portion is also a portion of the ranging sensor 210 facing the luggage 103 in the direction from the ranging sensor 210 toward the luggage 103. In other words, the first portion is a portion facing the through-hole 113a in the opening / closing unit 113 in the closed state in the direction from the ranging sensor 210 toward the luggage 103. The opening / closing unit 113 may be configured to have one through-hole 113a.

[0224] The second portion is an area including a three-dimensional point cloud on the front-to-back side of the area of ​​the opening 112a of the car cart 112. The second portion is also an area where the distance measuring sensor 210 does not face the luggage 103 in the direction from the distance measuring sensor 210 toward the luggage 103. In other words, the second portion is an area hidden by the opening / closing unit 113 in the closed state in the direction from the distance measuring sensor 210 toward the luggage 103.

[0225] The second filling rate calculation unit 804 generates voxel data 2052 shown in (c) of Fig. 35 by voxelizing the first portion and the second portion, respectively. In the voxel data 2052, the white areas without hatching are areas where the second portion has been voxelized, and the dotted hatched areas are areas where the first portion has been voxelized.

[0226] The second filling rate calculation unit 804 then estimates whether or not a package 103 is present behind the opening / closing section 113 for the white region corresponding to the region of the opening / closing section 113. Specifically, the second filling rate calculation unit 804 assigns scores based on the probability that a package is present to 26 voxels adjacent to the hatched voxel where the package 103 is present in the voxelized region. The second filling rate calculation unit 804 then assigns an added score to the voxels indicated by the white region adjacent to the multiple voxels where the package 103 is present. The second filling rate calculation unit 804 performs this for all voxels where the package 103 is present, and determines that the package 103 is present in the voxels indicated by the white region whose total score is equal to or greater than an arbitrary threshold. For example, if the arbitrary threshold is 0.1, the second filling rate calculation unit 804 determines that the package 103 is present in all regions, and can therefore calculate a package model 2053 in which the shape of the region hidden by the opening / closing section 113 is estimated, as shown in FIG. 35(e).

[0227] In this way, the information processing device 220 estimates the shape of the second part where the ranging sensor 210 does not face the object to be measured based on the shape of the first part where the ranging sensor 210 faces the luggage 103, so that even if a second part is present, the information processing device 220 can appropriately estimate the three-dimensional model of the object.

[0228] 36, the second filling rate calculation unit 804 may extract a contour R3 of an area where one or more pieces of luggage 103 are placed, and determine that the area inside the extracted contour R3 is an area where a piece of luggage 103 exists.The second filling rate calculation unit 804 may then estimate the area of ​​the opening / closing unit 113 inside the contour R3 using a three-dimensional point cloud in the area of ​​the multiple through-holes 113a of the opening / closing unit 113.

[0229] In the filling rate measurement method according to the third modification, the car cart 112 further includes an opening / closing unit 113 that has a plurality of through-holes 113a and opens and closes the opening 112a. The filling rate measurement method further determines whether the opening / closing unit 113 is in an open state or a closed state, and if the opening / closing unit 113 is in an open state, estimates a luggage model 2034 by performing extraction and estimation in the same manner as the filling rate calculation unit 224 of the first embodiment. If the opening / closing unit 113 is in a closed state, the filling rate calculation unit 224 estimates a second portion hidden by the opening / closing unit 113 based on a plurality of first portions that correspond to the plurality of through-holes 113a of the opening / closing unit 113 in the voxel data 2031 based on the spatial three-dimensional model 2011, and estimates the luggage model 2034 using the plurality of first portions, the estimated second portion, and the storage three-dimensional model 2032.

[0230] According to this, even when the luggage 103 is stored in the cart 112 provided with the opening / closing section 113 that opens and closes the opening 112a, the estimation method of the luggage model 2034 is switched between the first method and the second method depending on the open / closed state of the opening / closing section 113, so that the three-dimensional model of the object can be estimated appropriately.

[0231] (Variation 4) FIG. 37 is a diagram for explaining a method for generating a spatial three-dimensional model according to the fourth modification.

[0232] 37, even when generating a spatial 3D model, the 3D measurement system 200 may integrate the measurement results of the multiple ranging sensors 210, similar to the processing of the model generation unit 223. In this case, the 3D measurement system 200 identifies the positions and orientations of the multiple ranging sensors 210 by performing calibration in advance, and can generate a spatial 3D model including a 3D point cloud with little occlusion by integrating the multiple measurement results obtained based on the identified positions and orientations of the multiple ranging sensors 210.

[0233] (Variation 5) 38 and 39 are diagrams for explaining a method for generating a spatial three-dimensional model according to the fifth modification.

[0234] 38, even when generating a spatial three-dimensional model, the three-dimensional measurement system 200 may move at least one of the car cart 112 and one distance measuring sensor 210 so as to cross the measurement area R1 of one distance measuring sensor 210, and may integrate multiple measurement results obtained by the distance measuring sensor 210 at multiple times during the movement. The car cart 112 may be moved so as to cross the measurement area R1 of the distance measuring sensor 210 by being transported by an automated guided vehicle (AGV) 1101, for example.

[0235] In this case, the information processing device 220 calculates the relative position and orientation between the car cart 112 and one distance measurement sensor 210 at each timing when each measurement result is measured. For example, as shown in FIG. 39 , the information processing device 220 acquires a measurement result 2010 from the distance measurement sensor 210 and acquires position information 2061 of the automatic guided vehicle 1101 from the automatic guided vehicle 1101. The measurement result 2010 includes a first measurement result measured by the distance measurement sensor 210 at a first timing and a second measurement result measured at a second timing. The first timing and the second timing are different timings. The position information 2061 includes a first position of the automatic guided vehicle 1101 at the first timing and a second position of the automatic guided vehicle 1101 at the second timing. The first position and the second position are different positions. The position information 2061 is the self-position of the automatic guided vehicle 1101 estimated by the automatic guided vehicle 1101 at a plurality of times.

[0236] The self-location can be estimated using an existing method. For example, the automated guided vehicle 1101 may be placed at a specific location and estimate that it is at the specific location by reading specific location information from a marker or tag containing specific location information indicating the specific location, or the automated guided vehicle 1101 may estimate its own location based on the distance and direction traveled from the specific location estimated by reading the specific location information from the marker or tag. The automated guided vehicle 1101 may transmit the read specific location information and the distance and direction traveled from the specific location to the information processing device 220, and the information processing device 220 may estimate the position of the automated guided vehicle 1101 based on the specific location information and the distance and direction traveled from the specific location. The position of the automated guided vehicle 1101 may also be estimated using an image captured by a camera placed outside the automated guided vehicle 1101.

[0237] The information processing device 220 extracts four opening end points 2016, which are the positions of the four corners of each opening 2015 in the shelf area 2014, based on the measurement result 2010, which includes a plurality of measurement results obtained by measuring the car cart 112 from different viewpoints, and on the position information 2061. The information processing device 220 may identify an area in which the opening 112a of the car cart 112 in the first measurement result is likely to be present, based on a first position included in the position information 2061, and perform processing to calculate a measurement coordinate system for the identified area. The information processing device 220 may identify an area in which the opening 112a of the car cart 112 in the second measurement result is likely to be present, based on a second position included in the position information 2061, and perform processing to calculate a measurement coordinate system for the identified area.

[0238] Furthermore, the information processing device 220 may generate a spatial 3D model including a 3D point cloud with less occlusion by integrating the first measurement result and the second measurement result based on the first position and the second position included in the position information 2061. This allows for more accurate calculation of the filling rate.

[0239] The position information 2061 may include only one position at a specific time, and the measurement result 2010 may include only one measurement result at a specific time.

[0240] When multiple automated guided vehicles pass through the measurement area R1 of the distance measurement sensor 210 one by one in sequence, the information processing device 220 may calculate the filling rate of each car cart 112 based on the measurement results obtained for the car cart 112 carried by each of the multiple automated guided vehicles that have passed in sequence.

[0241] (Variation 6) In the sixth modification, the measurement area of ​​the distance measuring sensor will be described.

[0242] FIG. 40 is a diagram showing an example in which a single distance measuring sensor measures a plurality of car bogies.

[0243] 40, one distance measurement sensor 210 is placed so that all of the multiple car bogies 112 to be measured are included in the measurement area R10 of this distance measurement sensor 210. For example, the distance measurement sensor 210 may be placed at a position where the face of the car bogie 112 farther from the distance measurement sensor 210 is included in the maximum length of the measurement area R10 in the distance measurement direction.

[0244] FIG. 41 is a diagram showing an example in which a plurality of car bogies are measured by two distance measurement sensors.

[0245] As shown in FIG. 41, the two distance measurement sensors 210a, 210b are arranged so that the measurement areas R11, R12 of these distance measurement sensors 210a, 210b include the entire range in which the multiple car bogies 112 to be measured exist. Furthermore, the two distance measurement sensors 210a, 210b are arranged so that, for example, a length 902 in the distance measurement direction D1 of an area R13 where the measurement areas R11 and R12 overlap is longer than a length 901 in the distance measurement direction D1 of the car bogie 112. Note that the lengths 901 and 902 are lengths (heights) in the distance measurement direction D1 based on the plane on which the car bogies 112 are arranged. In other words, the overlapping area R13 has a length 902 in the distance measurement direction D1 that is equal to or greater than the length 901 of the car bogie 112. This makes it possible to maximize the number of multiple car bogies 112 that can be measured by the two distance measurement sensors 210a, 210b. The distance measurement direction D1 is the direction in which each of the distance measurement sensors 210a and 210b measures distance. In Figures 40 and 41, the distance measurement direction D1 is along the vertical direction, but the direction along which the distance measurement direction D1 runs is not limited to the vertical direction. The distance measurement direction D1 may also be along the horizontal direction.

[0246] Each of the distance measurement sensors 210a, 210b is the same sensor as the distance measurement sensor 210 in Fig. 40, and its measurement area is also the same size. In Fig. 40, one distance measurement sensor 210 can measure up to four car bogies 112. In Fig. 41, by arranging the two distance measurement sensors 210a, 210b so that the height 902 of the overlapping area R13 is higher than the height 901 of the car bogie 112 as described above, it is possible to arrange one more car bogie 112. As a result, nine car bogies 112 can be measured with the two distance measurement sensors 210a, 210b, and it is possible to measure a number of car bogies 112 that is greater than twice the number of car bogies 112 that can be measured with one distance measurement sensor 210.

[0247] FIG. 42 is a diagram showing an example in which a plurality of car bogies are measured by three distance measurement sensors.

[0248] As shown in Fig. 42, the three distance measurement sensors 210a, 210b, and 210c are arranged so that the measurement areas R21, R22, and R23 of these distance measurement sensors 210a, 210b, and 210c include all of the multiple car bogies 112 to be measured. Furthermore, the three distance measurement sensors 210a, 210b, and 210c are arranged so that the measurement area R24, for example, where at least two of the measurement areas R21, R22, and R23 overlap, includes all of the multiple car bogies 112 to be measured. This allows all of the multiple car bogies 112 to be measured by the multiple distance measurement sensors. This makes it possible to generate a spatial 3D model including a 3D point cloud with little occlusion.

[0249] (Variation 7) A three-dimensional measurement system 200A according to the seventh modification will be described.

[0250] FIG. 43 is a block diagram showing a characteristic configuration of a three-dimensional measurement system according to the seventh modification.

[0251] Three-dimensional measurement system 200A according to Modification 7 differs from three-dimensional measurement system 200 according to Embodiment 1 in that it includes two distance measuring sensors 210a and 210b. Information processing device 220A according to Modification 7 also differs in that it includes an integration unit 226 in addition to the components of information processing device 220 according to Embodiment 1. Here, the differences from Embodiment 1 will mainly be described.

[0252] The acquisition unit 221 acquires measurement results from each of the multiple distance measurement sensors 210a and 210b. Specifically, the acquisition unit 221 acquires a first measurement result from the distance measurement sensor 210a and a second measurement result from the distance measurement sensor 210b. The first measurement result includes a first spatial three-dimensional model generated by the distance measurement sensor 210a. The second measurement result includes a second spatial three-dimensional model generated by the distance measurement sensor 210b.

[0253] The integrating unit 226 integrates the first spatial three-dimensional model and the second spatial three-dimensional model. Specifically, the integrating unit 226 integrates the first spatial three-dimensional model and the second spatial three-dimensional model based on the position and orientation (external parameters) of the ranging sensor 210a and the position and orientation (external parameters) of the ranging sensor 210b stored in the storage unit 225. In this way, the integrating unit 226 generates an integrated spatial three-dimensional model. The position and orientation of the ranging sensor 210a and the position and orientation of the ranging sensor 210b stored in the storage unit 225 are generated by a calibration performed in advance.

[0254] The coordinate system calculation unit 222, the model generation unit 223, and the filling rate calculation unit 224 execute the processes described in the first embodiment using the integrated spatial three-dimensional model as the spatial three-dimensional model.

[0255] FIG. 44 is a flowchart of a filling rate measurement method performed by an information processing device according to the seventh modification.

[0256] The information processing device 220A acquires a plurality of spatial three-dimensional models from the distance measurement sensors 210a and 210b (S111a). The plurality of spatial three-dimensional models include a first spatial three-dimensional model and a second spatial three-dimensional model. At this time, the information processing device 220 may further acquire an image of the measurement target from the distance measurement sensors 210a and 210b.

[0257] The information processing device 220A integrates a plurality of spatial three-dimensional models to generate an integrated spatial three-dimensional model (S111b).

[0258] The information processing device 220A acquires the stored three-dimensional model stored in the storage unit 225 (S112).

[0259] Steps S113 to S116 are the same as those in the first embodiment except that an integrated spatial three-dimensional model is used instead of a spatial three-dimensional model, and therefore a description thereof will be omitted.

[0260] (Variation 8) In the information processing device 220 according to the second embodiment, line segments are detected from a two-dimensional image including an RGB image and a depth image, and the line segments of the shape of the opening of the stored three-dimensional model are identified from the detected line segments. However, the invention is not limited to identifying the line segments of the shape of the opening based on a two-dimensional image. The information processing device 220 may also identify the line segments (edges) of the shape of the opening from the measurement results of the ranging sensor 210 or a spatial three-dimensional model. For example, a cloud of three-dimensional points that is more than a certain number and arranged at regular intervals along a certain direction in the measurement results of the ranging sensor 210 or the spatial three-dimensional model may be detected as line segments, and the line segments of the shape of the opening of the stored three-dimensional model may be identified from the detected line segments.

[0261] (Other embodiments) Although the filling rate measurement method and the like according to the present disclosure have been described above based on the above-described embodiments, the present disclosure is not limited to the above-described embodiments.

[0262] For example, in the above-described embodiments, each processing unit included in an information processing device or the like is described as being implemented by a CPU and a control program. For example, each component of the processing unit may be composed of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may include, for example, a semiconductor device, an integrated circuit (IC), or a large-scale integration (LSI). The IC or LSI may be integrated on a single chip or on multiple chips. While the IC or LSI is referred to here as an IC or LSI, the name may vary depending on the degree of integration, and may be called a system LSI, a very large-scale integration (VLSI), or an ultra-large-scale integration (ULSI). Furthermore, a field-programmable gate array (FPGA), which is programmed after the LSI is manufactured, can also be used for the same purpose.

[0263] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, the general or specific aspects may be realized as a computer-readable non-transitory recording medium such as an optical disk, a hard disk drive (HDD), or a semiconductor memory on which the computer program is stored. Alternatively, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0264] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, and forms realized by arbitrarily combining the components and functions of the embodiments within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0265] The present disclosure is useful as a measurement method, an information processing device, a program, and the like for calculating an appropriate filling rate of an object to be measured. [Explanation of symbols]

[0266] 101, 105, 111 storage space 102 Shelf 102a, 112a opening 103, 103a~103c Luggage 104 Marker 106 Cargo bed 112 Basket Cart 113 Opening and Closing Section 113a Through hole 200, 200A 3D measurement system 210, 210A, 210B, 210a, 210b Distance measurement sensor 211 Laser irradiation unit 211A Infrared pattern irradiation unit 211B, 212B cameras 212 Laser receiving unit 212A Infrared Camera 213A Infrared Pattern 220, 220A Information processing equipment 221, 701 Acquisition Department 222, 222A, 222B, 222C Coordinate system calculation unit 223 Model Generation Unit 224 Filling rate calculation section 225 Storage section 226 Integration Department 301 Auxiliary section 302, 313, 323, 503, 605, 703 Calculation section 311, 321, 401 Detector 312, 322, 501, 321C extraction section 402 Generator 403 Volume calculation unit 502 Estimation section 601 Baggage volume calculation unit 602 Area division part 603 Planned Baggage Measurement Department 604 Area estimation part 702 Counting Department 801 Detection unit 802 Switching unit 803 1st filling rate calculation section 804 2nd filling rate calculation section 901 length 902 width 1101 Automated Guided Vehicle 1331 Line detector 1332 Aperture extraction part 1333 Endpoint calculation part 2000 measurement coordinate system 2001, 2021 images 2002 Adjustment Marker 2003 Superimposed Image 2004 Sensor Coordinate System 2005, 2017, 2026 Rotation Matrices 2006, 2018, 2027 Translation Vector 2010 Measurement Results 2011, 2051 spatial three-dimensional model 2012, 2022, 2032 storage 3D model 2013 Location information 2014 shelf area 2015 Aperture 2016 Opening end point 2023 Marker 2024 Marker Area 2025 pattern contour 2031, 2052 voxel data 2033 Luggage Area 2034, 2053 luggage models 2041 Occupied area 2042 empty area 2061 Location information 2101 RGB images 2102, 2112, 2121 Line images 2103, 2113, 2122 line segments 2111 Depth Image P1 one point R1, R10, R11, R12, R21, R22, R23 measurement area R2 area R3 Contour R13, R24 overlapping region

Claims

1. a storage unit having an opening and a storage space in which a measurement object is stored from the opening, and a three-dimensional spatial model is obtained by measurement through the opening by a distance measuring sensor; Obtaining a storage three-dimensional model of the storage section in an unstored state; acquiring a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image; using the stored three-dimensional model to identify line segments in the two-dimensional image that represent the shape of the opening; Correlating the position of the stored 3D model with the position of the spatial 3D model based on the position and orientation information and the identified line segment; estimating an object three-dimensional model, which is a three-dimensional model of the measurement object within the storage space, based on the storage three-dimensional model and the space three-dimensional model after the correspondence; Using the three-dimensional object model, a volume of an area capable of further storing an object of a specific shape in the storage space in which the measurement object is stored is estimated. Measurement method.

2. moreover, Calculating a filling rate of the measurement object in the storage space based on the volume of the three-dimensional object model and the volume of the storage area. The measurement method according to claim 1 .

3. The volume of the storage area is estimated based on the volume of the object to be stored and the maximum number of the objects that can be stored in an empty area in the storage space excluding the measurement target. The measurement method according to claim 1 or 2.

4. The filling rate is calculated by dividing the volume of the three-dimensional object model by the sum of the volume of the three-dimensional object model and the volume of the storage area. The measurement method according to any one of claims 1 to 3.

5. a processor; a memory; The processor uses the memory to: a storage unit having an opening and a storage space in which a measurement object is stored from the opening, and a three-dimensional spatial model is obtained by measurement through the opening by a distance measuring sensor; Obtaining a storage three-dimensional model of the storage section in an unstored state; acquiring a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image; using the stored three-dimensional model to identify line segments in the two-dimensional image that represent the shape of the opening; Correlating the position of the stored 3D model with the position of the spatial 3D model based on the position and orientation information and the identified line segment; estimating an object three-dimensional model, which is a three-dimensional model of the measurement object within the storage space, based on the storage three-dimensional model and the space three-dimensional model after the correspondence; Using the three-dimensional object model, a volume of an area capable of further storing an object of a specific shape in the storage space in which the measurement object is stored is estimated. Information processing device.

6. A program for causing a computer to execute a measurement method, The measurement method includes: a storage unit having an opening and a storage space in which a measurement object is stored from the opening, and a three-dimensional spatial model is obtained by measurement through the opening by a distance measuring sensor; Obtaining a storage three-dimensional model of the storage section in an unstored state; acquiring a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image; using the stored three-dimensional model to identify line segments in the two-dimensional image that represent the shape of the opening; Correlating the position of the stored 3D model with the position of the spatial 3D model based on the position and orientation information and the identified line segment; estimating an object three-dimensional model, which is a three-dimensional model of the measurement object within the storage space, based on the storage three-dimensional model and the space three-dimensional model after the correspondence; Using the three-dimensional object model, a volume of an area capable of further storing an object of a specific shape in the storage space in which the measurement object is stored is estimated. program.

Images