Fill rate measurement method, information processing apparatus, and program recording medium

Abstract

A space three-dimensional model (S111) is acquired by measuring from a distance measuring sensor opposite to a housing having an opening via the opening; a three-dimensional model of the housing, i.e., a housing three-dimensional model (S112) is acquired; an object part of a part of the space three-dimensional model as a measurement object is extracted (S114); a line segment representing a shape of the opening is determined (S113) from a two-dimensional image of the opening generated by measuring in a specific direction from a position of the distance measuring sensor; an object three-dimensional model of a three-dimensional model of the measurement object is estimated (S115) from a three-dimensional coordinate system based on a position of the opening on a three-dimensional space determined based on the position of the distance measuring sensor, the specific direction, and the shape of the opening, and the object part; and a filling rate of the measurement object with respect to the housing space is calculated (S116).

Description

Technical Field

[0001] This disclosure relates to a method for measuring fill rate, an information processing device, and a program recording medium. Background Technology

[0002] Patent document 1 discloses a three-dimensional shape measuring device that uses a three-dimensional laser scanner to obtain three-dimensional shapes.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-87319 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Applications of measurements to three-dimensional shapes have not been sufficiently studied. For example, calculations of the fill rate, representing the amount of a measured object contained within a given space, have not been adequately studied.

[0008] This disclosure provides a method for measuring the fill rate of an object, etc.

[0009] Methods used to solve problems

[0010] A method for measuring the fill rate of a technical solution disclosed herein includes: obtaining a spatial three-dimensional model, wherein the spatial three-dimensional model is obtained by measuring a receiving portion having an opening in a receiving space that accommodates a measuring object, and measuring the receiving portion through the opening by a ranging sensor opposite to the receiving portion; obtaining a receiving three-dimensional model, wherein the receiving three-dimensional model is a three-dimensional model of the receiving portion that does not accommodate the measuring object; obtaining a two-dimensional image of the opening and position and posture information corresponding to the two-dimensional image; using the receiving three-dimensional model, determining line segments in the two-dimensional image that represent the shape of the opening; calculating the position of the opening in three-dimensional space based on the position and posture information and the determined line segments; establishing a correspondence between the position of the receiving three-dimensional model and the position of the spatial three-dimensional model based on the calculated position of the opening; inferring a three-dimensional model of an object that serves as a three-dimensional model of the measuring object within the receiving space based on the established correspondence between the receiving three-dimensional model and the spatial three-dimensional model; and calculating the fill rate of the measuring object in the receiving space using the receiving three-dimensional model and the object three-dimensional model.

[0011] An information processing apparatus relating to the present disclosure includes a processor and a memory. The processor uses the memory to acquire a spatial three-dimensional model, which is obtained by measuring a receiving portion having an opening in a receiving space for accommodating a measurement object, using a ranging sensor opposite the receiving portion through the opening. It also acquires a receiving three-dimensional model, which is a three-dimensional model of the receiving portion that does not accommodate the measurement object. Furthermore, it acquires a two-dimensional image of the opening and position and orientation information corresponding to the two-dimensional image. Using the receiving three-dimensional model, it determines line segments in the two-dimensional image representing the shape of the opening. Based on the position and orientation information and the determined line segments, it calculates the position of the opening in three-dimensional space. Based on the calculated position of the opening, it establishes a correspondence between the position of the receiving three-dimensional model and the position of the spatial three-dimensional model. Based on the established correspondence between the receiving three-dimensional model and the spatial three-dimensional model, it infers a three-dimensional model of the object to be measured within the receiving space. Using the receiving three-dimensional model and the object three-dimensional model, it calculates the fill rate of the measurement object in the receiving space.

[0012] Furthermore, this disclosure can also be implemented as a program that causes a computer to perform the steps included in the above-described fill rate measurement method. Additionally, this disclosure can also be implemented as a non-transitory recording medium such as a CD-ROM that can be read by a computer and contains the program. Furthermore, this disclosure can also be implemented as information, data, or signals representing the program. Moreover, these programs, information, data, and signals can be distributed via communication networks such as the Internet.

[0013] Invention Effects

[0014] According to this disclosure, a method for measuring the fill rate of a measured object can be provided. Attached Figure Description

[0015] Figure 1 This is a diagram used to illustrate the outline of the filling rate measurement method of Embodiment 1.

[0016] Figure 2 This is a block diagram illustrating the characteristic structure of the three-dimensional measurement system according to Embodiment 1.

[0017] Figure 3 This is the first example diagram used to illustrate the structure of a ranging sensor.

[0018] Figure 4 This is the second example diagram used to illustrate the structure of a ranging sensor.

[0019] Figure 5 This is the third example of a diagram used to illustrate the structure of a ranging sensor.

[0020] Figure 6 This is a block diagram showing the structure of the coordinate system calculation unit in the first example.

[0021] Figure 7 This is a diagram used to illustrate the calculation method of the measurement coordinate system performed by the coordinate system calculation unit in Example 1.

[0022] Figure 8 This is a block diagram showing the structure of the coordinate system calculation unit in the second example.

[0023] Figure 9 This diagram illustrates the calculation method of the measurement coordinate system performed by the coordinate system calculation unit in Example 2.

[0024] Figure 10 This is a block diagram showing the structure of the coordinate system calculation unit in the third example.

[0025] Figure 11 This is a diagram used to illustrate the calculation method of the measurement coordinate system performed by the coordinate system calculation unit in Example 3.

[0026] Figure 12 This is a block diagram illustrating an example of the structure of the model generation unit.

[0027] Figure 13 This is a flowchart of the process performed by the model generation unit to calculate the volume of the space to be accommodated.

[0028] Figure 14 This is a block diagram illustrating an example of the structure of the fill rate calculation unit.

[0029] Figure 15 This is a diagram illustrating an example of the method used by the fill rate calculation unit to calculate the fill rate.

[0030] Figure 16 This is another example of the method used to illustrate the calculation of the fill rate by the fill rate calculation unit.

[0031] Figure 17 This is a flowchart of a method for measuring fill rate using an information processing device.

[0032] Figure 18 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Example 1.

[0033] Figure 19 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Example 2.

[0034] Figure 20 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Example 3.

[0035] Figure 21 This is a block diagram showing the structure of the coordinate system calculation unit in Embodiment 2.

[0036] Figure 22 This is a block diagram showing the structure of the detection unit in the coordinate system calculation unit of Embodiment 2.

[0037] Figure 23 This is a diagram used to illustrate the method for extracting the endpoint of the opening point performed by the detection unit in Embodiment 2.

[0038] Figure 24 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Embodiment 2.

[0039] Figure 25 This is a diagram used to illustrate how the fill rate is calculated.

[0040] Figure 26 This is a block diagram illustrating an example of the structure of the calculation unit for the fill rate calculation unit in Modified Example 1.

[0041] Figure 27 This is a flowchart of the fill rate calculation process of the calculation unit in the calculation unit of the modified example 1.

[0042] Figure 28 This diagram illustrates an example of a situation where two or more shelves are accommodated in the cargo space of a truck, etc.

[0043] Figure 29 It is a table showing the shelves housed in the cargo container and their relationship to the filling rate.

[0044] Figure 30 This is a block diagram illustrating an example of the structure of the calculation unit for the fill rate calculation unit in Modified Example 2.

[0045] Figure 31 This is a flowchart of the fill rate calculation process of the calculation unit in the calculation unit of Modified Example 2.

[0046] Figure 32 This is a diagram used to illustrate the structure of the cage-type trolley in variation 3.

[0047] Figure 33 This is a block diagram illustrating an example of the structure of the fill rate calculation unit related to Modified Example 3.

[0048] Figure 34 This is a flowchart of the fill rate calculation process in the fill rate calculation section of Modified Example 3.

[0049] Figure 35 This is a diagram illustrating an example of the second method for calculating the fill rate.

[0050] Figure 36 This is another example of the second method for calculating the fill rate.

[0051] Figure 37 This is a diagram used to illustrate the method for generating the spatial three-dimensional model of variation example 4.

[0052] Figure 38 This is a diagram used to illustrate the method for generating the spatial three-dimensional model of variation example 5.

[0053] Figure 39 This is a diagram used to illustrate the method for generating the spatial three-dimensional model of variation example 5.

[0054] Figure 40 This diagram illustrates an example of using a single ranging sensor to measure multiple cage trolleys.

[0055] Figure 41 This diagram illustrates an example of using two ranging sensors to measure multiple cage trolleys.

[0056] Figure 42 This diagram illustrates an example of using three distance sensors to measure multiple cage trolleys.

[0057] Figure 43 This is a block diagram representing the characteristic structure of the three-dimensional measurement system related to variation 7.

[0058] Figure 44 This is a flowchart of a method for measuring the fill rate using the information processing device related to Modification 7. Detailed Implementation

[0059] (How this disclosure came about)

[0060] In logistics and distribution sites, it is necessary to measure the fill rate of goods and other items in the storage space to improve the efficiency of space utilization. Furthermore, in order to accommodate more items in containers and other storage compartments, it is required to measure more fill rates in a short time. However, methods for easily measuring fill rates have not yet been sufficiently researched.

[0061] Therefore, in this disclosure, by applying a method for generating a three-dimensional model to the accommodating part containing the object to be measured, a method for measuring the filling rate of more accommodating parts that can be easily calculated in a short time is provided.

[0062] A method for measuring the fill rate of a technical solution disclosed herein includes: obtaining a spatial three-dimensional model, wherein the spatial three-dimensional model is obtained by measuring a receiving portion having an opening in a receiving space that accommodates a measuring object, and measuring the receiving portion through the opening using a ranging sensor opposite to the receiving portion; obtaining a receiving three-dimensional model, wherein the receiving three-dimensional model is a three-dimensional model of the receiving portion that does not accommodate the measuring object; obtaining a two-dimensional image of the opening and position and posture information corresponding to the two-dimensional image; using the receiving three-dimensional model, determining line segments in the two-dimensional image that represent the shape of the opening; calculating the position of the opening in three-dimensional space based on the position and posture information and the determined line segments; establishing a correspondence between the position of the receiving three-dimensional model and the position of the spatial three-dimensional model based on the calculated position of the opening; inferring a three-dimensional model of the object to be the three-dimensional model of the measuring object within the receiving space based on the established correspondence between the receiving three-dimensional model and the spatial three-dimensional model; and calculating the fill rate of the measuring object in the receiving space using the receiving three-dimensional model and the object three-dimensional model.

[0063] Therefore, using a three-dimensional coordinate system based on the opening position and a deduced three-dimensional model of the object, a three-dimensional model of the object to be measured can be deduced. Thus, by simply measuring the accommodating portion containing the object to be measured, the fill rate of the accommodating space by the object to be measured can be easily calculated.

[0064] Alternatively, the two-dimensional image may include an RGB image generated by photographing the opening with a camera; the position and pose information indicates the position and pose of the camera when photographing the opening.

[0065] Alternatively, the two-dimensional image may include a depth image generated based on the measurement of the opening by the ranging sensor; the position and pose information may represent the position and pose of the ranging sensor when measuring the opening.

[0066] Alternatively, the aforementioned two-dimensional image may include at least one of an RGB image, a grayscale image, an infrared image, and a depth image; the aforementioned RGB image is generated by capturing the aforementioned opening with a camera; the aforementioned depth image is an image generated based on the measurement results of the aforementioned ranging sensor.

[0067] Therefore, it is possible to extract line segments that represent the shape of the opening with good accuracy.

[0068] Alternatively, the line segment can be determined based on both the line segment determined from the RGB image and the line segment determined from the depth image.

[0069] Alternatively, the aforementioned ranging sensor may include at least one of a ToF (Time of Flight) sensor and a stereo camera.

[0070] Alternatively, the aforementioned ranging sensor may include a first ranging sensor and a second ranging sensor; the first measurement area of ​​the first ranging sensor and the second measurement area of ​​the second ranging sensor may have an overlapping region.

[0071] Therefore, it is possible to measure the object over a wider range.

[0072] Alternatively, the overlapping area may have a length greater than or equal to the length of the object being measured in the direction of the ranging sensor.

[0073] Therefore, it is possible to measure the object over a wider range.

[0074] Alternatively, the overlapping area may include the entire range of the object being measured.

[0075] This allows us to obtain spatial 3D models with minimal occlusion.

[0076] Alternatively, the aforementioned receiving portion may move relative to the aforementioned ranging sensor in a direction that intersects with the ranging direction measured by the aforementioned ranging sensor; the aforementioned three-dimensional spatial model is generated using a first measurement result measured by the aforementioned ranging sensor at a first timing and a second measurement result measured at a second timing.

[0077] This allows us to obtain a spatial 3D model with minimal occlusion.

[0078] Alternatively, the positions of the aforementioned three-dimensional models and the aforementioned spatial three-dimensional models can be established by using rotation matrices and translation vectors.

[0079] Alternatively, a second filling rate of one or more of the multiple accommodating portions for the second accommodating portion can be calculated, wherein the second accommodating portion has a second accommodating space for accommodating one or more of the aforementioned accommodating portions.

[0080] Alternatively, the third filling rate of the measured object contained in one or more accommodating portions with respect to the second accommodating portion can be calculated, wherein the second accommodating portion has a second accommodating space for accommodating one or more of the aforementioned accommodating portions.

[0081] An information processing apparatus relating to the present disclosure includes a processor and a memory. The processor uses the memory to acquire a spatial three-dimensional model, which is obtained by measuring a receiving portion having an opening in a receiving space for accommodating a measurement object, using a ranging sensor opposite the receiving portion through the opening. It also acquires a receiving three-dimensional model, which is a three-dimensional model of the receiving portion that does not accommodate the measurement object. Furthermore, it acquires a two-dimensional image of the opening and positional information corresponding to the two-dimensional image. Using the receiving three-dimensional model, it determines line segments in the two-dimensional image representing the shape of the opening. Based on the positional information and the determined line segments, it calculates the position of the opening in three-dimensional space. Based on the calculated position of the opening, it establishes a correspondence between the position of the receiving three-dimensional model and the position of the spatial three-dimensional model. Based on the established correspondence between the receiving three-dimensional model and the spatial three-dimensional model, it infers a three-dimensional model of the object to be measured within the receiving space. Using the receiving three-dimensional model and the object three-dimensional model, it calculates the fill rate of the measurement object in the receiving space.

[0082] Therefore, using a three-dimensional coordinate system based on the opening position and a deduced three-dimensional model of the object, a three-dimensional model of the object to be measured can be deduced. Thus, by simply measuring the accommodating portion containing the object to be measured, the fill rate of the accommodating space by the object to be measured can be easily calculated.

[0083] Furthermore, this disclosure can also be implemented as a program that causes a computer to perform the steps included in the above-described fill rate measurement method. Additionally, this disclosure can also be implemented as a non-transitory recording medium such as a CD-ROM that can be read by a computer and contains the program. Furthermore, this disclosure can also be implemented as information, data, or signals representing the program. Moreover, these programs, information, data, and signals can be distributed via communication networks such as the Internet.

[0084] Hereinafter, various embodiments of the three-dimensional model generation method of this disclosure will be described in detail with the aid of accompanying drawings. Furthermore, each embodiment described below represents a specific example of this disclosure. Therefore, the numerical values, shapes, materials, constituent elements, arrangements and connection forms of constituent elements, steps, and order of steps shown in the following embodiments are examples, and their purpose is not to limit this disclosure.

[0085] Furthermore, the figures are schematic diagrams and not necessarily rigorous illustrations. Also, in the figures, substantially identical structures are given the same labels, and there are instances of omitted or simplified repetitions of explanation.

[0086] (Implementation Method 1)

[0087] While referring to Figure 1 The outline of the filling rate measurement method of Embodiment 1 is explained.

[0088] Figure 1 This is a diagram used to illustrate the outline of the filling rate measurement method of Embodiment 1.

[0089] In methods for measuring fill rate, such as Figure 1 As shown, a ranging sensor 210 is used to measure the goods 103 housed in a shelf 102 having a housing space 101. The filling rate of the goods 103 in the housing space 101 is calculated using the obtained measurement results. An opening 102a is formed in the shelf 102 for placing and removing the goods 103 relative to the housing space 101. The ranging sensor 210 is positioned opposite the opening 102a of the shelf 102, with the orientation for measuring the shelf 102 including the opening 102a, and measures the measurement area R1, including the interior of the housing space 101, through the opening 102a.

[0090] In addition, for example, Figure 1 As shown, the shelf 102 has a box-like shape. The shelf 102 may not have a box-like shape, as long as it is configured to have a mounting surface for holding goods 103 and a receiving space 101 above the mounting surface for accommodating the goods 103. The shelf 102 is an example of a receiving section. The receiving space 101 is an example of a first receiving space. The receiving space 101 is the internal space of the shelf 102, but it is not limited to this; it could also be a space within a warehouse that houses the objects to be measured, such as goods 103. Goods 103 is an example of an object to be measured. The object to be measured is not limited to goods 103; it could also be merchandise. That is, the object to be measured can be any object as long as it is a movable object.

[0091] Figure 2 This is a block diagram illustrating the characteristic structure of the three-dimensional measurement system according to Embodiment 1. Figure 3 This is the first example diagram used to illustrate the structure of a ranging sensor. Figure 4 This is the second example diagram used to illustrate the structure of a ranging sensor. Figure 5 This is the third example of a diagram used to illustrate the structure of a ranging sensor.

[0092] like Figure 2 As shown, the three-dimensional measurement system 200 includes a distance sensor 210 and an information processing device 220. The three-dimensional measurement system 200 can have multiple distance sensors 210 or only one distance sensor 210.

[0093] The ranging sensor 210 measures the three-dimensional space including the first receiving space of the shelf 102 through the opening 102a of the shelf 102, thereby obtaining the measurement results including the shelf 102 and the receiving space 101 of the shelf 102. Specifically, the ranging sensor 210 generates a spatial three-dimensional model represented by a set of three-dimensional points representing the three-dimensional positions of multiple measurement points on the shelf 102 or the goods 103 (hereinafter referred to as the measurement object). The set of three-dimensional points is called a three-dimensional point cloud. The three-dimensional position represented by each three-dimensional point in the three-dimensional point cloud is represented, for example, by three-dimensional coordinates composed of three-valued information of the X, Y, and Z components of a three-dimensional coordinate space, which is composed of the XYZ axes. In addition, the three-dimensional model may not only contain three-dimensional coordinates, but may also include color information representing the color of each point, or shape information representing the shape of each point and its surrounding surface. The color information may be represented by the RGB color space, or by other color spaces such as HSV, HLS, YUV, etc.

[0094] use Figures 3-5 A specific example of the ranging sensor 210 will be described.

[0095] like Figure 3 As shown, the first example of the ranging sensor 210 generates a spatial three-dimensional model by emitting electromagnetic waves and acquiring the reflected waves from the measured object. Specifically, the ranging sensor 210 measures the time taken from the emission of the electromagnetic wave to its reflection back to the ranging sensor 210 by the measured object, and uses the measured time and the wavelength of the electromagnetic wave used in the measurement to calculate the distance between the ranging sensor 210 and a point P1 on the surface of the measured object. The ranging sensor 210 emits electromagnetic waves from its reference point in multiple predetermined radial directions. For example, the ranging sensor 210 may emit electromagnetic waves at a first angular interval around the horizontal direction and at a second angular interval around the vertical direction. Therefore, by detecting the distance to the measured object in each of the multiple directions around the ranging sensor 210, the ranging sensor 210 can calculate the three-dimensional coordinates of multiple points on the measured object. Thus, the ranging sensor 210 can calculate positional information representing multiple three-dimensional positions on the measured object and generate a spatial three-dimensional model with positional information. Location information can be a 3D point cloud containing multiple 3D points representing multiple 3D locations.

[0096] like Figure 3As shown, the ranging sensor 210 in the first example is a three-dimensional laser measuring device having a laser irradiation unit 211 that irradiates a laser as an electromagnetic wave, and a laser light receiving unit 212 that receives the reflected light reflected from the irradiated laser by the object being measured. The ranging sensor 210 scans the object being measured with a laser by rotating or oscillating the unit equipped with the laser irradiation unit 211 and the laser light receiving unit 212 around two different axes, or by placing a movable mirror (MEMS (Micro Electro Mechanical Systems) mirror) that oscillates along the path of the irradiated or received laser. Thus, the ranging sensor 210 can generate a high-precision and high-density three-dimensional model of the object being measured.

[0097] The distance sensor 210 is illustrated as a three-dimensional laser measuring device that measures the distance between itself and the object by irradiating a laser, but it is not limited to this. It can also be a millimeter-wave radar measuring device that measures the distance between itself and the object by emitting millimeter waves.

[0098] Furthermore, the ranging sensor 210 can generate a three-dimensional model with color information. The first color information is generated using an image captured by the ranging sensor 210, and this color information represents the color of each of the multiple first three-dimensional points contained in the first three-dimensional point cloud.

[0099] Specifically, the range sensor 210 may have a built-in camera for capturing images of the measurement object surrounding it. The camera built into the range sensor 210 generates an image by capturing images of the area within the illumination range of the laser beam irradiated by the range sensor 210. Alternatively, the camera may not be built into the range sensor 210 and may be disposed externally. A camera disposed externally to the range sensor 210 may also be disposed at the same position as the range sensor 210. Furthermore, the capture range and illumination range are pre-established to correspond. Specifically, multiple directions of the laser beam irradiated by the range sensor 210 and each pixel in the image captured by the camera are pre-established to correspond, and the range sensor 210 sets pixel values ​​in the image corresponding to the directions of the three-dimensional points, as color information representing the colors of the multiple three-dimensional points contained in the three-dimensional point cloud.

[0100] like Figure 4As shown, the second example of the ranging sensor 210A is a ranging sensor using structured light. The ranging sensor 210A has an infrared pattern illumination unit 211A and an infrared camera 212A. The infrared pattern illumination unit 211A projects a pre-set infrared pattern 213A onto the surface of the object being measured. The infrared camera 212A captures an infrared image of the object on which the infrared pattern 213A has been projected. The ranging sensor 210A explores the infrared pattern 213A contained in the obtained infrared image and calculates the distance from either the infrared pattern illumination unit 211A or the infrared camera 212A to the aforementioned point P1 on the object being measured, based on a triangle formed by connecting the positions of a point P1 in the infrared pattern on the object in real space, the position of the infrared pattern illumination unit 211A, and the position of the infrared camera 212A. Thus, the ranging sensor 210A can obtain a three-dimensional point of the measurement point on the object being measured.

[0101] In addition, the ranging sensor 210A moves the unit of the ranging sensor 210A, which has an infrared pattern irradiation section 211A and an infrared camera 212A, or uses the infrared pattern irradiated by the infrared pattern irradiation section 211A as a fine texture, thereby enabling the acquisition of a high-density three-dimensional model.

[0102] Alternatively, the ranging sensor 210A can use a visible light region that can acquire color information from the infrared camera 212A, taking into account the position or orientation of the infrared pattern irradiation unit 211A or the infrared camera 212A, to establish a correlation between the obtained visible light region and three-dimensional points, thereby generating a three-dimensional model with color information. Alternatively, the ranging sensor 210A can also have a structure that includes a visible light camera for adding color information.

[0103] like Figure 5As shown, the range sensor 210B in the third example is a range sensor that measures three-dimensional points using a stereo camera. The range sensor 210B is a stereo camera with two cameras 211B and 212B. The range sensor 210B acquires a stereo image with parallax by synchronously capturing images of the object being measured using the two cameras 211B and 212B at a set time. The range sensor 210B uses the obtained stereo images (two images) to perform feature point matching processing between the two images, obtaining alignment information between the two images with pixel-level or fractional-pixel precision. Based on a triangle formed by connecting the matching position of a point P1 on the object in real space, the positions of the two cameras 211B and 212B, the range sensor 210B calculates the distance from one of the two cameras 211B and 212B to the matching position (i.e., point P1) on the object. Thus, the range sensor 210B can acquire the three-dimensional point of the measurement point on the object.

[0104] In addition, by moving the unit of the range sensor 210B, which has two cameras 211B and 212B, or by increasing the number of cameras mounted on the range sensor 210B to three or more, the same measurement object can be photographed and matched, thereby obtaining a high-precision three-dimensional model.

[0105] Furthermore, by making the cameras 211B and 212B of the ranging sensor 210B visible light cameras, color information can be easily added to the acquired 3D model.

[0106] In addition, in this embodiment, the example of the information processing device 220 having the first example of the ranging sensor 210 is described, but it is also possible to have a structure that replaces the first example of the ranging sensor 210 with the second example of the ranging sensor 210A or the third example of the ranging sensor 210B.

[0107] Furthermore, the two cameras 211B and 212B are capable of capturing monochrome images, including visible light or infrared images. In this case, the matching process between the two images in the 3D measurement system 200 can be performed, for example, using SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion). Additionally, the information representing the position and orientation of the cameras 211B and 212B obtained through this processing can be used to increase the point cloud density of the measurement space model via MVS (Multi-View Stereo).

[0108] Back Figure 2 The structure of the information processing device 220 will be described.

[0109] The information processing device 220 includes an acquisition unit 221, a coordinate system calculation unit 222, a model generation unit 223, a fill rate calculation unit 224, and a storage unit 225.

[0110] The acquisition unit 221 acquires the spatial three-dimensional model and image generated by the ranging sensor 210. Specifically, the acquisition unit 221 may also acquire the spatial three-dimensional model and image from the ranging sensor 210. Alternatively, the spatial three-dimensional model and image acquired by the acquisition unit 221 may be stored in the storage unit 225.

[0111] The coordinate system calculation unit 222 uses a spatial three-dimensional model and image to calculate the positional relationship between the ranging sensor 210 and the shelf 102. Thus, the coordinate system calculation unit 222 calculates a measurement coordinate system based on the shape of a portion of the shelf 102. The coordinate system calculation unit 222 can also calculate a measurement coordinate system based solely on the shape of a portion of the shelf 102. Specifically, the coordinate system calculation unit 222 may use the shape of the opening 102a of the shelf 102 as a reference for calculating the measurement coordinate system. Furthermore, as shown in Embodiment 1, the shape of the opening 102a, which serves as the reference for calculating the measurement coordinate system, can be either a corner or a side of the opening 102a if the shape of the opening 102a is rectangular.

[0112] Furthermore, the measurement coordinate system is a three-dimensional orthogonal coordinate system, an example of a first-dimensional coordinate system. By calculating the measurement coordinate system, the relative position and orientation of the range sensor 210 with the shelf 102 as a reference can be determined. That is, this allows the sensor coordinate system of the range sensor 210 to be matched with the measurement coordinate system, enabling calibration between the shelf 102 and the range sensor 210. Additionally, the sensor coordinate system is a three-dimensional orthogonal coordinate system.

[0113] Furthermore, in this embodiment, the cuboid-shaped shelf 102 has an opening 102a on one side, but is not limited to this. The shelf may also have openings on multiple faces of the cuboid shape, such as openings on both the front and rear surfaces, or openings on both the front and top surfaces. When the shelf has multiple openings, a predetermined reference position, as described later, may be set for one of the openings. This predetermined reference position may be set in the three-dimensional model of the shelf 102, i.e., in the space containing the three-dimensional model where no three-dimensional points or voxels exist.

[0114] Here, using Figure 6 and Figure 7The coordinate system calculation unit 222 of the first example will be explained.

[0115] Figure 6 This is a block diagram showing the structure of the coordinate system calculation unit in the first example. Figure 7 This is a diagram used to illustrate the calculation method of the measurement coordinate system performed by the coordinate system calculation unit in Example 1.

[0116] The coordinate system calculation unit 222 calculates the measurement coordinate system. The measurement coordinate system is a three-dimensional coordinate system that serves as the reference for the spatial three-dimensional model. For example, the distance sensor 210 is set at the origin of the measurement coordinate system, facing directly opposite the opening 102a of the shelf 102. In this case, the measurement coordinate system can also set the upward direction of the distance sensor 210 as the X-axis, the rightward direction as the Y-axis, and the forward direction as the Z-axis. The coordinate system calculation unit 222 includes an auxiliary unit 301 and a calculation unit 302.

[0117] like Figure 7 As shown in (a), the auxiliary unit 301 sequentially acquires images 2001, which are measurement results of the ranging sensor 210, acquired by the acquisition unit 221, in real time, and overlays adjustment marks 2002 onto each sequentially acquired image 2001. The auxiliary unit 301 sequentially outputs the overlaid image 2003, with the adjustment marks 2002 superimposed on the image 2001, to a display device (not shown). The display device sequentially displays the overlaid image 2003 output by the information processing device 220. Alternatively, the auxiliary unit 301 and the display device may be integrated into the ranging sensor 210.

[0118] The adjustment mark 2002 is a mark used to support the user in moving the range sensor 210 so that the position and orientation of the range sensor 210 relative to the shelf 102 are specific. The user changes the position and orientation of the range sensor 210 relative to the shelf 102 by aligning the adjustment mark 2002 with a predetermined reference position on the shelf 102 while viewing the overlay image 2003 displayed on the display device. The predetermined reference position of the shelf 102 is, for example, the position of the four corners of the quadrilateral opening 102a of the shelf 102.

[0119] When the ranging sensor 210 is positioned and oriented in a specific location relative to the shelf 102, an overlay image 2003 is generated, which superimposes four adjustment marks 2002 at four locations corresponding to the four corners of the opening 102a of the shelf 102. For example, the user moves the ranging sensor 210 so that the adjustment marks 2002 are aligned with the shelf 102. Figure 7 The arrow in (a) is moved in the direction shown in the middle, so that it can be as follows: Figure 7Align the four adjustment marks 2002 with the positions of the four corners of the opening 102a as shown in (b).

[0120] In addition, the auxiliary unit 301 can overlay the adjustment mark 2002 onto the image 2001, but it can also overlay the adjustment mark onto the spatial three-dimensional model, so that the spatial three-dimensional model with the adjustment mark overlaid is displayed on the display device.

[0121] like Figure 7 As shown in (c), the calculation unit 302 calculates a rotation matrix 2005 and a translation vector 2006 representing the positional relationship between the distance sensor 210 and the shelf 102 when the four adjustment marks 2002 are aligned with the four corners of the opening 102a. The calculation unit 302 uses the calculated rotation matrix 2005 and translation vector 2006 to transform the sensor coordinate system 2004 of the distance sensor 210, thereby calculating a measurement coordinate system 2000 with any one of the four corners of the opening 102a as its origin. Thus, the calculation unit 302 can establish a correspondence between the position of the contained three-dimensional model and the position of the spatial three-dimensional model. Furthermore, when the four adjustment marks 2002 are aligned with the four corners of the opening 102a, the user can also input data to an input device (not shown). The information processing device 220 can also acquire this input when it is received from the input device, thereby determining when the four adjustment marks 2002 are aligned with the four corners of the opening 102a. In addition, the information processing device 220 can also determine whether the four adjustment marks 2002 are aligned with the four corners of the opening 102a by analyzing the image 2001.

[0122] Next, use Figure 8 and Figure 9 The coordinate system calculation unit 222A of the second example will be explained.

[0123] Figure 8 This is a block diagram showing the structure of the coordinate system calculation unit in the second example. Figure 9 This diagram illustrates the calculation method of the measurement coordinate system performed by the coordinate system calculation unit in Example 2.

[0124] The coordinate system calculation unit 222A includes a detection unit 311, an extraction unit 312, and a calculation unit 313.

[0125] Testing Department 311 uses Figure 9 The three-dimensional spatial model 2011, which is the measurement result of the ranging sensor 210, is shown in (a) by the acquisition unit 221. Figure 9 The three-dimensional model 2012 shown in (b) is as follows. Figure 9The shelf area 2014 corresponding to the shelf 102 is detected as shown in (c). Additionally, the 3D model 2012 is a 3D model of the shelf 102 without goods 103, generated in advance using the measurement results of the ranging sensor 210 for the shelf 102 without goods 103. The 3D model 2012 is generated by the model generation unit 223 (described later) and stored in the storage unit 225. The 3D model 2012 may also include position information 2013 indicating the positions of the four corners of the opening 102a of the shelf 102.

[0126] Extraction unit 312, such as Figure 9 As shown in (d), using the position information 2013 that accommodates the three-dimensional model 2012, four opening endpoints 2016 are extracted as the positions of the four corners of the opening 2015 in the shelving area 2014. The shape of the opening 2015 defined by the four opening endpoints 2016 is an example of the shape that serves as part of the reference for calculating the measurement coordinate system.

[0127] like Figure 9 As shown in (e), the calculation unit 313 calculates a rotation matrix 2017 and a translation vector 2018 representing the positional relationship between the range sensor 210 and the shelf 102, based on the shapes of the four opening endpoints 2016 observed from the range sensor 210. Using the rotation matrix 2017 and the translation vector 2018, the calculation unit 313 calculates the measurement coordinate system 2000 by transforming the sensor coordinate system 2004 of the range sensor 210. Thus, the calculation unit 313 can establish a correspondence between the position of the three-dimensional model and the position of the spatial three-dimensional model. Specifically, if the rotation matrix 2017 is R and the translation vector 2018 is T, then the three-dimensional point x in the sensor coordinate system 2004 can be transformed into a three-dimensional point X in the measurement coordinate system 2000 using Equation 1 shown below. Therefore, the calculation unit 313 can calculate the measurement coordinate system 2000.

[0128] X = Rx + T…Equation 1

[0129] Next, use Figure 10 and Figure 11 The coordinate system calculation unit 222A of the third example will be explained.

[0130] Figure 10 This is a block diagram showing the structure of the coordinate system calculation unit in the third example. Figure 11 This is a diagram used to illustrate the calculation method of the measurement coordinate system performed by the coordinate system calculation unit in Example 3.

[0131] The coordinate system calculation unit 222B includes a detection unit 321, an extraction unit 322, and a calculation unit 323. In the third example, a mark 104 is arranged at a specific position (e.g., the position on the upper surface) of the shelf 102, and the coordinate system calculation unit 222B determines the measurement coordinate system 2000 based on the position of the mark 104. That is, the measurement coordinate system 2000 in this case is a coordinate system based on the position of the mark 104 provided on the shelf 102.

[0132] Additionally, mark 104 may have a checkerboard pattern, for example. Mark 104 is not limited to any calibration mark (positioning mark) with a prescribed shape.

[0133] Inspection Department 321 according to Figure 11 Image 2021, shown in (a), is the measurement result of the ranging sensor 210 acquired by the acquisition unit 221. Figure 11 As shown in (c), the marking area 2024 corresponding to the marking 104 set on the shelf 102 is detected.

[0134] Extraction unit 322 extracts from marked region 2024 on image 2021, such as Figure 11 The pattern outline 2025 is extracted as the outline of the checkerboard pattern, as shown in (d).

[0135] Based on the shape of the extracted pattern contour 2025, the calculation unit 323 calculates a rotation matrix 2026 and a translation vector 2027 representing the positional relationship between the ranging sensor 210 and the marker 104. The calculation unit 323 uses the rotation matrix 2026 and the translation vector 2027... Figure 11 The positional relationship between the accommodating 3D model 2022 and the marker 2023 shown in (b) is used to calculate the 3D positional relationship between the ranging sensor 210 and the shelf 102. By transforming the sensor coordinate system 2004 using the calculated 3D positional relationship, the measurement coordinate system 2000 is calculated. Thus, the calculation unit 323 can establish a correspondence between the position of the accommodating 3D model and the position of the spatial 3D model. Furthermore, the positional relationship between the accommodating 3D model 2022 and the marker 2023 can be measured in advance or generated in advance based on the design data of the shelf 102 with the marker 104.

[0136] Back Figure 2 The model generation unit 223 will be explained.

[0137] The model generation unit 223 generates a 3D model of the shelf 102 without the goods 103. The model generation unit 223 generates the 3D model by acquiring measurement results of the shelf 102 without the goods 103 from the ranging sensor 210. Specific processing performed by the model generation unit 223 will be described later. The generated 3D model is stored in the storage unit 225.

[0138] Here, using Figure 12 and Figure 13 The model generation unit 223 will be explained in detail.

[0139] Figure 12 This is a block diagram illustrating an example of the structure of the model generation unit. Figure 13 This is a flowchart of the process performed by the model generation unit to calculate the volume of the space to be accommodated.

[0140] The model generation unit 223 includes a detection unit 401, a generation unit 402, and a volume calculation unit 403.

[0141] The detection unit 401 detects the shelf area corresponding to the shelf 102 based on the spatial three-dimensional model measured by the distance sensor 210 (S101). When the three-dimensional measurement system 200 has multiple distance sensors 210, the detection unit 401 performs step S101 on each of the multiple distance sensors 210. Thus, the detection unit 401 detects multiple shelf areas corresponding to each of the multiple distance sensors 210.

[0142] In the case where the 3D measurement system 200 has multiple ranging sensors 210, the generation unit 402 integrates multiple shelf areas to generate a 3D model that accommodates them (S102). Specifically, in order to integrate the multiple shelf areas, the generation unit 402 can either align the 3D point cloud using ICP (Iterative Closest Point) or pre-calculate the relative positional relationships of the multiple ranging sensors 210 and integrate the multiple shelf areas based on the calculated relative positional relationships. The calculation of the relative positional relationships can also be performed by using SfM (Structure from Motion) to calculate multiple images acquired by the multiple ranging sensors 210 as multi-view images. The multiple ranging sensors 210 can also be set up based on a design drawing whose relative positional relationships are determined.

[0143] Alternatively, instead of multiple ranging sensors 210, one ranging sensor 210 can be moved, and multiple measurement results from multiple locations can be used to integrate multiple shelf areas obtained from the multiple measurement results to generate a three-dimensional model of the shelf 102.

[0144] Alternatively, the 3D model can be generated based on 3D CAD data from the design of the shelf 102, without the measurement results from the distance sensor 210, or based on dimensional measurement data of the shelf 102 or equipment specifications published by the manufacturer. Furthermore, the 3D model can also be generated by inputting manually measured dimensions of the shelf 102 into the information processing device 220.

[0145] Furthermore, if the 3D measurement system 200 has only one ranging sensor 210 instead of multiple ranging sensors 210, and uses a measurement result from a single location, the model generation unit 223 may not have a generation unit 402. That is, the model generation unit 223 may not perform step S102.

[0146] The volume calculation unit 403 uses the three-dimensional model of the shelf 102 to calculate the volume of the storage space 101 (S103).

[0147] Back Figure 2 The filling rate calculation unit 224 will be explained.

[0148] The fill rate calculation unit 224 calculates the fill rate of the cargo 103 relative to the accommodating space 101 of the shelf 102. For example, the fill rate calculation unit 224 may also use a spatial three-dimensional model, image and measurement coordinate system 2000 obtained by the ranging sensor 210 to calculate the ratio of the volume of the cargo 103 to the volume of the accommodating space 101 as the fill rate.

[0149] Here, using Figure 14 and Figure 15 The fill rate calculation unit 224 will be explained in detail.

[0150] Figure 14 This is a block diagram illustrating an example of the structure of the fill rate calculation unit. Figure 15 This is a diagram illustrating an example of the method used by the fill rate calculation unit to calculate the fill rate. Additionally, Figure 15 This example illustrates the case where the ranging sensor 210 is directly facing the opening 102a of the shelf 102. The ranging sensor 210 is positioned on the negative Z-axis side of the opening 102a of the shelf 102, and measures the accommodating space 101 of the shelf 102 via the opening 102a. This example shows the case where the coordinate system calculation unit 222 of the first example measures the measurement coordinate system 2000. That is, in this case, the sensor coordinate system 2004 is aligned with the measurement coordinate system 2000.

[0151] The fill rate calculation unit 224 includes an extraction unit 501, an estimation unit 502, and a calculation unit 503.

[0152] The extraction unit 501 uses the spatial three-dimensional model 2011 and the containing three-dimensional model to extract the cargo area 2033 in the spatial three-dimensional model, which corresponds to the cargo 103. Specifically, the extraction unit 501 extracts the cargo area 2033 in the spatial three-dimensional model. Figure 15 The data structure of the spatial three-dimensional model 2011, which is the measurement result of the ranging sensor 210, obtained by the acquisition unit 221 as shown in Figure (a), is transformed into voxel data to generate... Figure 15 Voxel data 2031 is shown in (b). Extraction unit 501 uses the generated voxel data 2031 and... Figure 15 The container 3D model 2032 shown in (c) is a voxelized container 3D model. The container 3D model 2032 is subtracted from the voxel data 2031 to extract the data. Figure 15 The cargo region 2033 in the voxel data 2031 shown in (d) is the region that represents the result of measuring cargo 103. The cargo region 2033 is an example of the object portion that corresponds to the object being measured.

[0153] The estimation unit 502 uses the extracted cargo region 2033 to estimate the cargo model 2034, which is a three-dimensional model of cargo 103 within the accommodating space 101. The cargo model 2034 is an example of a three-dimensional model of an object. Specifically, the estimation unit 502 uses the cargo region 2033 to interpolate the area of ​​cargo 103 that is obscured by the distance sensor 210 in the Z-axis direction, which is the arrangement direction of the distance sensor 210 and the shelf 102. That is, it interpolates the cargo region 2033 in the positive Z-axis direction. For example, the estimation unit 502 determines whether each voxel constituting the cargo region 2033 is positioned in the negative Z-axis direction than the voxel positioned in the farthest Z-axis direction. When a voxel is positioned on the negative Z-axis side compared to the farthest voxel, and no voxel is positioned on the positive Z-axis side compared to that voxel, the estimation unit 502 interpolates the voxel until it reaches a position on the same Z-axis as the farthest voxel. Thus, the estimation unit 502 estimates... Figure 15 The cargo model 2034 shown in (e) is as shown.

[0154] The calculation unit 503 uses the accommodating 3D model and the cargo model 2034 to calculate the first fill rate of the cargo 103 relative to the accommodating space 101. Specifically, the calculation unit 503 counts the number of multiple voxels constituting the cargo model 2034, and multiplies the obtained count by a preset voxel size to calculate the volume of the cargo 103. The calculation unit 503 calculates the ratio of the calculated volume of the cargo 103 to the volume of the accommodating space 101 of the shelf 102 calculated by the model generation unit 223 as the first fill rate.

[0155] Alternatively, the ranging sensor 210 may not be directly opposite the opening 102a of the shelf 102. Figure 16 This is another example of the method used to illustrate the calculation of the fill rate by the fill rate calculation unit. Figure 16 This example illustrates a case where the ranging sensor 210 is positioned at an angle relative to the opening 102a of the shelf 102. This example demonstrates a case where the measurement coordinate system 2000 is measured by either the coordinate system calculation unit 222A (in the second example) or the coordinate system calculation unit 222B (in the third example). That is, in this case, the sensor coordinate system 2004 is different from the measurement coordinate system 2000.

[0156] exist Figure 16 In the example case, the coordinate system used is the measurement coordinate system 2000. The estimation unit 502 uses the cargo area 2033 to interpolate the area of ​​the cargo 103 that is obscured by the distance sensor 210 in the Z-axis direction of the measurement coordinate system 2000, which is the arrangement direction of the distance sensor 210 and the shelf 102. That is, it interpolates the cargo area 2033 on the positive Z-axis direction side.

[0157] Other processing performed by the fill rate calculation unit 224 and Figure 15 The situation is the same, so the explanation is omitted.

[0158] Furthermore, the spatial three-dimensional model and image pair used in the coordinate system calculation performed by the coordinate system calculation unit 222 and the fill rate calculation performed by the fill rate calculation unit 224 can be either the result of measurement by the ranging sensor 210 at the same time or the result of measurement at different times.

[0159] The ranging sensor 210 and the information processing device 220 can also be connected to each other communicatively via a communication network. The communication network can be a public communication network such as the Internet, or a private communication network. Thus, the spatial three-dimensional model and image obtained by the ranging sensor 210 are transmitted from the ranging sensor 210 to the information processing device 220 via the communication network.

[0160] Furthermore, the information processing device 220 can also acquire the spatial 3D model and image from the ranging sensor 210 without using a communication network. For example, the spatial 3D model and image can be temporarily stored from the ranging sensor 210 to an external storage device such as a hard disk drive (HDD) or a solid-state drive (SSD), and the information processing device 220 can acquire the spatial 3D model and image from the external storage device. Alternatively, the external storage device can also be a cloud server.

[0161] The information processing device 220 includes, for example, at least a computer system. This computer system has a control program, processing circuitry including a processor or logic circuitry for executing the control program, and a recording device including internal memory or accessible external memory for storing the control program. The functions of each processing unit of the information processing device 220 can be implemented either by software or by hardware.

[0162] Next, the operation of the information processing device 220 will be explained.

[0163] Figure 17 This is a flowchart of a method for measuring fill rate using an information processing device.

[0164] The information processing device 220 acquires a three-dimensional spatial model from the ranging sensor 210 (S111). At this time, the information processing device 220 may also acquire an image of the measured object from the ranging sensor 210.

[0165] The information processing device 220 acquires the three-dimensional model stored in the storage unit 225 (S112).

[0166] The information processing device 220 calculates a measurement coordinate system based on the shape of the opening 102a of the shelf 102 (S113). Step S113 is the processing performed by the coordinate system calculation unit 222.

[0167] The information processing device 220 uses the voxel data 2031 of the spatial three-dimensional model 2011 and the accommodating three-dimensional model 2032 to extract the cargo region 2033 corresponding to the cargo 103 from the voxel data 2031 (S114). Step S114 is the processing performed by the extraction unit 501 of the fill rate calculation unit 224.

[0168] The information processing device 220 uses the extracted cargo area 2033 to infer a cargo model 2034, which is a three-dimensional model of the cargo 103 within the accommodating space 101 (S115). Step S115 is the processing performed by the estimation unit 502 of the fill rate calculation unit 224.

[0169] The information processing device 220 uses the accommodating 3D model and the cargo model 2034 to calculate the first fill rate of the cargo 103 for the accommodating space 101 (S116). Step S116 is the processing performed by the calculation unit 503 of the fill rate calculation unit 224.

[0170] Figure 18 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Example 1.

[0171] The coordinate system calculation unit 222 acquires images 2001, which are the measurement results of the ranging sensor 210, acquired by the acquisition unit 221 in real time, and superimposes adjustment marks 2002 onto the acquired images 2001 in sequence (S121). Step S121 is a process performed by the auxiliary unit 301 of the coordinate system calculation unit 222.

[0172] The coordinate system calculation unit 222 obtains the position and orientation of the ranging sensor 210 (S122). Step S121 is a process performed by the auxiliary unit 301 of the coordinate system calculation unit 222.

[0173] The coordinate system calculation unit 222 uses the position and orientation of the ranging sensor 210 when the four adjustment marks 2002 are aligned with the four corners of the opening 102a to determine the sensor coordinate system 2004 of the ranging sensor 210, and uses the determined sensor coordinate system 2004 to calculate the measurement coordinate system 2000 (S123). Step S123 is the processing performed by the calculation unit 302 of the coordinate system calculation unit 222.

[0174] Figure 19 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Example 2.

[0175] The coordinate system calculation unit 222A uses the spatial three-dimensional model 2011, which is the measurement result of the ranging sensor 210, and the accommodating three-dimensional model 2012, acquired by the acquisition unit 221, to detect the shelf area 2014 corresponding to the shelf 102 (S121A). Step S121A is the processing performed by the detection unit 311 of the coordinate system calculation unit 222A.

[0176] The coordinate system calculation unit 222A uses the position information 2013 of the three-dimensional model 2012 to extract the four opening endpoints 2016 of the shelf area 2014, which are the four corners of the opening 2015 (S122A). Step S122A is the processing performed by the extraction unit 312 of the coordinate system calculation unit 222A.

[0177] Based on the shapes of the four opening endpoints 2016 observed from the range sensor 210, the coordinate system calculation unit 222A calculates a rotation matrix 2017 and a translation vector 2018 representing the positional relationship between the range sensor 210 and the shelf 102. Furthermore, the coordinate system calculation unit 222A transforms the sensor coordinate system 2004 of the range sensor 210 using the rotation matrix 2017 and the translation vector 2018, thereby calculating the measurement coordinate system 2000 (S123A). Step S123A is the processing performed by the calculation unit 313 of the coordinate system calculation unit 222A.

[0178] Figure 20This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Example 3.

[0179] The coordinate system calculation unit 222B detects the marked area 2024 based on the image 2021, which is the measurement result of the ranging sensor 210, acquired by the acquisition unit 221 (S121B). Step S121B is the processing performed by the detection unit 321 of the coordinate system calculation unit 222B.

[0180] The coordinate system calculation unit 222B extracts the pattern outline 2025 from the marked area 2024 on the image 2021 (S122B). Step S122B is the processing performed by the extraction unit 322 of the coordinate system calculation unit 222B.

[0181] Based on the shape of the extracted pattern contour 2025, the coordinate system calculation unit 222B calculates a rotation matrix 2026 and a translation vector 2027 representing the positional relationship between the ranging sensor 210 and the mark 104. Furthermore, using the rotation matrix 2026 and translation vector 2027, and the positional relationship between the 3D model 2022 and the mark 2023, the coordinate system calculation unit 222B calculates the three-dimensional positional relationship between the ranging sensor 210 and the shelf 102. The calculated three-dimensional positional relationship is then used to transform the sensor coordinate system 2004, thereby calculating the measurement coordinate system 2000 (S123B). Step S123B is the processing performed by the calculation unit 323 of the coordinate system calculation unit 222B.

[0182] Additionally, the fill rate calculated by the information processing device 220 can be output from the information processing device 220. This fill rate can be displayed via a display device (not shown) included with the information processing device 220, or it can be sent to an external device different from the information processing device 220. For example, the calculated fill rate can be output to a freight transport system for control purposes.

[0183] According to the filling rate measurement method of this embodiment, the cargo model 2034 of cargo 103 is inferred using cargo area 2033. The cargo area 2033 is an area extracted using a spatial three-dimensional model obtained by measuring the shelf 102 in the state of containing cargo 103 and a three-dimensional model of the shelf 102 without containing cargo 103. Therefore, the first filling rate of cargo 103 in the storage space 101 can be easily calculated by measuring the shelf 102 in the state of containing cargo 103.

[0184] Furthermore, in the filling rate measurement method, the cargo model 2034 is estimated based on a three-dimensional coordinate system with the shape of a portion of the shelf 102 as a reference. Therefore, the processing amount of estimating the cargo model 2034 can be reduced.

[0185] Furthermore, in the filling rate measurement method, the cargo model 2034 is inferred based on a three-dimensional coordinate system that uses only a portion of the shelf 102's shape as a reference. This allows the shape of only a portion of the housing, easily extracted from an image, to be used in the calculation of the measurement coordinate system. Consequently, the processing speed for inferring the cargo model is improved, and the calculation accuracy of the measurement coordinate system is increased.

[0186] Furthermore, in the fill rate measurement method, the three-dimensional coordinate system is a three-dimensional orthogonal coordinate system with a Z-axis. In the estimation, the cargo model 2034 is estimated by interpolating the positive Z-axis direction side of the cargo region 2033, which is opposite to the negative Z-axis direction. Therefore, the processing amount of estimating the cargo model 2034 can be effectively reduced.

[0187] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system is a coordinate system based on the shape of the opening 102a of the shelf 102. Therefore, the coordinate system based on the shape of the opening 102a of the shelf 102 can be easily calculated, and the cargo model 2034 can be inferred based on the calculated coordinate system.

[0188] Furthermore, in the filling rate measurement method, the three-dimensional coordinate system can be a coordinate system based on the mark 104 set on the shelf 102. Therefore, the coordinate system based on the mark 104 can be easily calculated, and the cargo model 2034 can be inferred based on the calculated coordinate system.

[0189] (Implementation Method 2)

[0190] The information processing apparatus of Embodiment 2 differs in structure of the coordinate system calculation unit from that of Embodiment 1. This will be explained in detail below.

[0191] Figure 21 This is a block diagram showing the structure of the coordinate system calculation unit in Embodiment 2. Figure 22 This is a block diagram showing the structure of the extraction unit of the coordinate system calculation unit in Embodiment 2. Figure 23 This is a diagram used to illustrate the extraction method of the extraction section of embodiment 2 for extracting the end of the opening.

[0192] The coordinate system calculation unit 222C differs from the coordinate system calculation unit 222A in that it has an extraction unit 321C instead of the detection unit 311 and extraction unit 312.

[0193] Similar to the detection unit 311 and the extraction unit 312, the extraction unit 321C uses the measurement results from the ranging sensor 210 obtained by the acquisition unit 221 and the three-dimensional model 2012 to extract the four opening endpoints 2016, which are the four corner positions of the openings 2015 in the shelf area 2014. Furthermore, the extraction unit 321C only needs to determine the four opening endpoints 2016 of the openings 2015 in the shelf area 2014; it does not need to perform the process of extracting the four opening endpoints 2016. The extraction unit 321C includes a line segment detection unit 1331, an opening extraction unit 1332, and an endpoint calculation unit 1333.

[0194] The measurement results of the ranging sensor 210 in Embodiment 2 include an RGB image and a depth image. The RGB image is a two-dimensional image captured by a camera built into the ranging sensor 210. The RGB image is an image of the entire opening 2015, that is, a two-dimensional image of the opening 2015. Thus, the RGB image is an example of a two-dimensional image of the opening 2015 generated by measuring from the position of the ranging sensor 210 in a specific direction. The RGB image is an image captured (measured) by a camera positioned at the location of the ranging sensor 210 in a specific direction. The specific direction indicates the orientation of the camera when capturing the RGB image (e.g., the shooting direction), such as the direction from the position of the ranging sensor 210 toward the opening 2015. Alternatively, the specific direction may not be consistent with the direction from the position of the ranging sensor 210 toward the opening 2015, as long as the shooting direction of the camera includes the opening 2015 in the area captured by the camera. Alternatively, the position of the ranging sensor 210 and the specific direction may be used as the position and orientation (external parameters of the camera), respectively. Furthermore, the position and orientation of the camera may be preset. The camera's external parameters are the position and pose information corresponding to the RGB image.

[0195] The depth image is an image generated by the ranging sensor 210. The depth image is a two-dimensional image with pixel values ​​representing the distance measured by the ranging sensor 210 in the ranging direction (depth direction) to the measured object including the opening 2015. That is, the depth image is another example of a two-dimensional image of the opening 2015. The depth image is, for example, an image generated based on the measurement results of the ranging sensor 210. The depth image can be generated based on measurement results containing only the opening 2015 and its surrounding area, or it can be generated based on a spatial three-dimensional model or measurement results used as the basis for generating a spatial three-dimensional model. In this case, the remeasurement of the opening 2015 by the ranging sensor 210 is omitted. The specific direction represents the orientation of the ranging sensor 210 (e.g., the ranging direction) when the measurement results obtained by the ranging sensor 210 are measured, which is the basis for the generation of the depth image; for example, the direction from the position of the ranging sensor 210 toward the opening 2015. Furthermore, the specific direction does not necessarily have to be consistent with the direction from the position of the range sensor 210 towards the opening 2015, as long as the measurement range of the range sensor 210 includes the opening 2015, it is acceptable. The position and specific direction of the range sensor 210 represent the position and orientation of the range sensor 210, respectively, and are external parameters of the range sensor 210. The external parameters of the range sensor 210 are the position and orientation information corresponding to the depth image.

[0196] In addition, two-dimensional images are not limited to RGB images and depth images; grayscale images, infrared images, etc. can also be used.

[0197] The specific orientation of the camera and the specific orientation of the range sensor 210 can be the same or different. The external parameters of the camera can also be the same as the external parameters of the range sensor 210. Therefore, the external parameters of the range sensor 210 can also be used as the external parameters of the camera.

[0198] Line segment detection unit 1331 for Figure 23 The RGB image 2101 shown in (a) performs line segment detection processing. The line segment detection unit 1331 detects line segments based on the RGB image 2101, thereby... Figure 23 As shown in (b), a line segment image 2102 is generated that contains line segment 2103 in the RGB image 2101.

[0199] Similarly, the line segment detection unit 1331 for Figure 23 The depth image 2111 shown in (c) performs line segment detection processing. The line segment detection unit 1331 detects line segments based on the depth image 2111, thereby... Figure 23 As shown in (d), a line segment image 2112 is generated that contains line segment 2113 in the depth image 2111.

[0200] In addition, in the process of detecting line segments, the line segment detection unit 1331 detects edges based on the difference in pixel values ​​between adjacent pixels of each image, and detects the direction perpendicular to the direction of the detected edge, thereby detecting line segments.

[0201] The opening extraction section 1332 uses line segment images 2102, 2112 and Figure 23 The three-dimensional model 2012 shown in (e) is as follows. Figure 23 As shown in (f), line segments 2122 representing the shape of the opening 2015 are extracted from line segment images 2102 and 2112, generating a line segment image 2121 containing line segments 2122. Specifically, the opening extraction unit 1332 performs pattern matching with the shape of the opening accommodating the 3D model 2012 as a template on the line segment image obtained by combining line segment images 2102 and 2112, thereby extracting line segments 2122. Line segment images 2102 and 2112 can be aligned using external parameters of the camera and external parameters of the ranging sensor 210, and multiple line segments contained in one of the line segment images 2102 and 2112 can be positioned at the positions of the multiple line segments in the other image, thereby combining them. Line segment image 2121 can also be an image containing only line segments 2122 representing the shape of the opening 2015. Figure 23 In line segment image 2121 of (f), since it is determined that there are four shapes in line segment images 2102 and 2112 that match the three-dimensional model 2012, four line segments 2122 containing the opening 2015 are extracted.

[0202] like Figure 23 As shown in (g), the endpoint calculation unit 1333 extracts the four opening endpoints 2016 in the shelf area 2014, which are located at the four corners of each opening 2015, based on the external parameters of the camera and the external parameters of the range sensor 210, and the line segments 2122 contained in the line segment image 2121. The external parameters of the camera and the external parameters of the range sensor 210 can be pre-stored by the storage unit 225 of the information processing device 220. In addition, in Figure 23 In the example, because the shapes of four openings 2015 were detected, 16 opening endpoints 2016 were extracted. Figure 23 In (g), because some of the opening endpoints 2016 overlap, nine opening endpoints 2016 are shown in the figure.

[0203] Furthermore, if the opening extraction unit 1332 extracts a line segment 2122 containing the shape of the opening 2015, the position of the opening 2015 can be determined, so the processing performed by the endpoint calculation unit 1333 is not necessarily required. The shape of the opening 2015 can be defined by the line segment 2122, the opening endpoint 2016, or a combination of the line segment 2122 and the opening endpoint 2016. When the opening 2015 is a quadrilateral, it can be defined by four line segments, by four opening endpoints representing vertices, or by a combination of line segments and opening endpoints. That is, when the opening is a polygon, it can be defined by the line segments constituting the edges of the polygon, by the vertices of the polygon, or by a combination of line segments and vertices. In addition, when the opening is a circle, including ellipses and perfect circles, it can be defined by the shape of the curve of the outer edge of the circle.

[0204] Thus, the extraction unit 321C determines the line segments representing the shape of the opening 2015 based on the RGB image 2101 and the depth image 2111. Furthermore, the extraction unit 321C calculates the position of the opening 2015 in three-dimensional space based on the position and specific orientation of the ranging sensor 210 and the determined shape of the opening 2015 (i.e., the line segment 2122 in the line segment image 2121).

[0205] The processing of the calculation unit 313 after extracting the four opening endpoints 2016 is the same as in Implementation Method 1, so the description is omitted.

[0206] In addition, Figure 23 The example described is based on both the RGB image 2101 and the depth image 2111, which are two-dimensional images, but it is not limited to this. For example, the line segment of the opening 2015 can also be extracted based on either the RGB image 2101 or the depth image 2111.

[0207] Furthermore, the accuracy of line segment extraction from the RGB image 2101 is affected by environmental factors such as the brightness (illuminance) around the opening 2015. The RGB image 2101 contains more information than the depth image 2111, so more line segments are detected.

[0208] On the other hand, the accuracy of line segment extraction from depth image 2111 is less affected by the environment (brightness) compared to RGB image 2101. In the absence of distance information obtained through ranging, depth image 2111 may have deficiencies in areas where distance information is unavailable. Therefore, since RGB image 2101 and depth image 2111 have different characteristics, it is also possible to extract line segments of opening 2015 based on the characteristics of each image.

[0209] For example, in the process of extracting line segments, if the brightness near the opening 2015 exceeds the specified illuminance, then the line segment image 2102 obtained from the RGB image 2101 and the line segment image 2112 obtained from the depth image 2111 are combined, prioritizing the RGB image 2101 over the depth image 2111. The line segments of the opening 2015 are then extracted based on the combined result, thereby improving the accuracy of line segment extraction. Conversely, in the process of extracting line segments, if the brightness near the opening 2015 is below the specified illuminance, then the line segment image 2102 obtained from the RGB image 2101 and the line segment image 2112 obtained from the depth image 2111 are combined, prioritizing the depth image 2111 over the RGB image 2101, thereby improving the accuracy of line segment extraction of the opening 2015.

[0210] The brightness near the opening 2015 can also be inferred from the pixel values ​​of the pixels contained in the RGB image 2101.

[0211] Furthermore, there are cases where the higher the fill rate of the cargo 103 into the shelf 102's accommodating space 101, the lower the brightness near the opening 2015. In this case, the fill rate can be calculated first, and the two-dimensional image prioritized in the line segment extraction process can be determined based on the calculated fill rate. That is, in the line segment extraction process, if the calculated fill rate is below a specified fill rate, the line segment image 2102 obtained from the RGB image 2101 and the line segment image 2112 obtained from the depth image 2111 are combined, prioritizing the RGB image 2101 over the depth image 2111. The line segment of the opening 2015 is then extracted based on the combined result, thereby improving the line segment extraction accuracy. Conversely, in the process of extracting line segments, if the calculated fill rate exceeds the specified fill rate, the line segment image 2102 obtained from the RGB image 2101 is combined with the line segment image 2112 obtained from the depth image 2111 in a manner that prioritizes the depth image 2111 over the RGB image 2101, thereby improving the extraction accuracy of the line segments of the opening 2015.

[0212] Furthermore, in the process of extracting line segments, since it is difficult to measure the shape of the opening based on the depth image 2111 when there are more than a specified threshold in the spatial 3D model, the line segment image 2102 obtained from the RGB image 2101 is combined with the line segment image 2112 obtained from the depth image 2111, prioritizing the RGB image 2101 over the depth image 2111. The line segments of the opening 2015 are extracted based on the combined result, thereby improving the accuracy of line segment extraction. For example, the distance information of the opening 2015 located at a distance greater than a specified distance from the ranging sensor 210 in a direction orthogonal to the direction of ranging relative to the ranging sensor 210 is not sufficiently accurate or is incomplete because the laser from the ranging sensor 210 is difficult to reflect. Therefore, the line segment image 2102 obtained from the RGB image 2101 and the line segment image 2112 obtained from the depth image 2111 can be combined, prioritizing the RGB image 2101 over the depth image 2111. Conversely, in the process of extracting line segments, if the spatial three-dimensional model lacks a value below the specified threshold, the line segment extraction accuracy of the opening 2015 can be improved by combining the line segment image 2102 obtained from the RGB image 2101 with the line segment image 2112 obtained from the depth image 2111, prioritizing the depth image 2111 over the RGB image 2101.

[0213] In addition, as described above, assuming that the RGB image 2101 is prioritized over the depth image 2111, the line segment image 2102 obtained from the RGB image 2101 is combined with the line segment image 2112 obtained from the depth image 2111, and the line segment of the opening 2015 is extracted based on the result of the combination. Specifically, the following processing can also be performed.

[0214] The first example of this processing will be explained. If the RGB image 2101 is prioritized over the depth image 2111, the result of this combination may simply be a line segment image 2102. In this case, it is also possible to generate a line segment image 2112 without relying on the depth image 2111.

[0215] Next, the second example will be explained. In the second example, during the line segment extraction process, each extracted line segment can be assigned an evaluation value representing likelihood (accuracy). That is, in this case, evaluation values ​​are assigned to each of the multiple line segments contained in line segment images 2102 and 2112. In the combination of line segment images 2102 and 2112, the evaluation values ​​of each line segment in each line segment image 2102 and 2112 are weighted and summed with weights corresponding to the illumination around the opening 2015. Thus, line segment images 2102 and 2112 are integrated. At this time, assuming that the RGB image 2101 takes priority over the depth image 2111, the weight for line segment image 2102 in the weighted summation is set to be greater than the weight for line segment image 2112.

[0216] Furthermore, line segments with evaluation values ​​above a threshold are extracted from multiple line segments in the integrated image and used as candidates for line segments with opening 2015. Pattern matching is then performed on the extracted candidate line segments to extract the line segments with opening 2015. The evaluation value representing the likelihood can be a higher value for longer line segments, or a higher value for the difference in pixel values ​​between two adjacent pixels with the edge as the boundary when detecting the line segment, or a higher value for the difference in pixel values ​​between two pixels belonging to two adjacent regions with the edge as the boundary.

[0217] Furthermore, the case where the line segment image 2102 obtained from the RGB image 2101 and the line segment image 2112 obtained from the depth image 2111 are combined in a manner where the depth image 2111 takes precedence over the RGB image 2101, can be explained by replacing the RGB image 2101 and the depth image 2111, and replacing the line segment image 2102 and the line segment image 2112, in the explanation of when the RGB image 2101 takes precedence over the depth image 2111.

[0218] Next, the operation of the information processing device according to Embodiment 2 will be explained. The structure of the coordinate system calculation unit of the information processing device according to Embodiment 2 is different from that of the information processing device according to Embodiment 1, so the operation (S113) of the coordinate system calculation unit will be explained.

[0219] Figure 24 This is a flowchart of the coordinate system calculation process (S113) performed by the coordinate system calculation unit in Embodiment 2.

[0220] The coordinate system calculation unit 222C detects line segments based on the two-dimensional image (S1121). Specifically, the coordinate system calculation unit 222C detects line segments based on the RGB image 2101, thereby generating a line segment image 2102 containing the line segment 2103 of the RGB image 2101. Furthermore, the coordinate system calculation unit 222C detects line segments based on the depth image 2111, thereby generating a line segment image 2112 containing the line segment 2113 of the depth image 2111. Step S1121 is the processing performed by the line segment detection unit 1331 of the extraction unit 321C of the coordinate system calculation unit 222C.

[0221] The coordinate system calculation unit 222C extracts the line segment representing the opening 2015 from the detected line segments (S1122). Specifically, the coordinate system calculation unit 222C uses the line segment images 2102 and 2112 and the three-dimensional model 2012 to extract the line segment 2122 representing the shape of the opening 2015 from the line segment images 2102 and 2112, generating a line segment image 2121 containing the line segment 2122. Step S1122 is the processing performed by the opening extraction unit 1332 of the extraction unit 321C of the coordinate system calculation unit 222C.

[0222] The coordinate system calculation unit 222C extracts the four opening endpoints 2016 in the shelf area 2014, which are the four corner positions of each opening 2015, based on the position of the ranging sensor 210, the direction (i.e., a specific direction) of the RGB image 2101 and the depth image 2111, and the line segments 2122 contained in the line segment image 2121 (S1123). Step S1123 is the processing performed by the endpoint calculation unit 1333 of the extraction unit 321C of the coordinate system calculation unit 222C.

[0223] Based on the shapes of the four opening endpoints 2016 as seen from the range sensor 210, the coordinate system calculation unit 222C calculates a rotation matrix 2017 and a translation vector 2018 representing the positional relationship between the range sensor 210 and the shelf 102. Next, the coordinate system calculation unit 222A transforms the sensor coordinate system 2004 of the range sensor 210 using the rotation matrix 2017 and the translation vector 2018, thereby calculating the measurement coordinate system 2000 (S1124). Step S1124 is the processing performed by the calculation unit 313 of the coordinate system calculation unit 222C. That is, this processing is the same as the processing performed by the calculation unit 313 of the coordinate system calculation unit 222A. Thus, the coordinate system calculation unit 222C can establish a correspondence between the position of the accommodating three-dimensional model and the position of the spatial three-dimensional model.

[0224] Alternatively, step S1123 may not necessarily be performed. Without performing step S1123, in step S1124, the coordinate system calculation unit 222A calculates a rotation matrix 2017 and a translation vector 2018 representing the positional relationship between the distance sensor 210 and the shelf 102, based on the line segment 2122 representing the shape of the opening 2015. Next, the coordinate system calculation unit 222A transforms the sensor coordinate system 2004 of the distance sensor 210 using the rotation matrix 2017 and the translation vector 2018, thereby calculating the measurement coordinate system 2000.

[0225] (Variation Example 1)

[0226] In the information processing apparatus 220 of the above-described embodiments, the ratio of the volume of the goods 103 contained in the accommodating space 101 to the volume of the accommodating space 101 is calculated as the filling rate, but it is not limited thereto.

[0227] Figure 25 This is a diagram used to illustrate how the fill rate is calculated.

[0228] exist Figure 25 In (a) and (b), the shelf 102 has a capacity 101 that can accommodate exactly 16 goods 103. Figure 25 As shown in (a), even with eight goods 103 arranged without gaps, eight more goods 103 can still be accommodated in the empty storage space 101. On the other hand, as... Figure 25 As shown in (b), when goods are arranged with gaps, if eight goods 103 are to be accommodated in the remaining space of the accommodating space 101, the already accommodated goods 103 need to be moved. If the already accommodated goods 103 are not moved and the goods 103 are accommodated in the remaining space of the accommodating space 101, only six goods 103 can be accommodated.

[0229] Thus, in Figure 25 The situation in (a) and Figure 25 In case (b), although the amount of goods 103 that can be accommodated in the remaining space of the accommodating space 101 is different, the fill rate is calculated to be the same 50% in both cases. Therefore, it is conceivable to calculate the fill rate that takes into account the substantially accommodating space in a way that matches the shape of the remaining space of the accommodating space 101.

[0230] Figure 26 This is a block diagram illustrating an example of the structure of the calculation unit for the fill rate calculation unit in Modified Example 1. Figure 27 This is a flowchart illustrating the fill rate calculation process of the calculation unit in the relevant variation example 1.

[0231] like Figure 26As shown, the calculation unit 503 includes a cargo volume calculation unit 601, a region division unit 602, a predetermined cargo measurement unit 603, a region estimation unit 604, and a calculation unit 605.

[0232] The cargo volume calculation unit 601 calculates the cargo volume, which is the volume of cargo 103, based on the cargo model 2034 (S131). The cargo volume calculation unit 601 calculates the volume of cargo 103 contained in the accommodating space 101 using the same method as in Embodiment 1.

[0233] Next, the region segmentation unit 602 divides the accommodating space 101 of the spatial three-dimensional model 2011 into an occupied area 2041 occupied by the goods 103 and an empty area 2042 not occupied by the goods 103 (S132).

[0234] Next, the pre-selected cargo measuring unit 603 calculates the volume of the cargo to be housed (S133). The shape and dimensions of the cargo to be housed are as follows: Figure 25 In the case shown in (c), where there are multiple types, the planned cargo measuring unit 603 calculates the volume equivalent to one cargo for each type. For example, the planned cargo measuring unit 603 calculates the volume of cargo 103a, cargo 103b, and cargo 103c respectively.

[0235] Next, the area estimation unit 604 estimates the placement of the maximum number of pre-accommodated goods 103 that can be accommodated in the empty area 2042, and estimates the number of pre-accommodated goods 103 in that case. That is, the area estimation unit 604 estimates the maximum number of pre-accommodated goods 103 that can be accommodated in the empty area 2042. The area estimation unit 604 calculates the accommodating volume in the empty area 2042 by multiplying the volume of one goods by the number of goods that can be accommodated (S134).

[0236] Furthermore, when there are multiple types of goods, the area estimation unit 604 can estimate the number of goods that can be accommodated for each type, or it can estimate the number of goods that can be accommodated for a mixture of multiple types. When accommodating goods in a mixed manner, the area estimation unit 604 calculates the cumulative value of the volume obtained by multiplying the volume of one type of goods by the number of goods of that type that can be accommodated for each type, and uses this as the volume that can be accommodated in the empty area 2042. For example, if the area estimation unit 604 estimates that it can accommodate n1 goods 103a, n2 goods 103b, and n3 goods 103c, it calculates the cumulative value of the first volume obtained by multiplying the volume of goods 103a by n1, the second volume obtained by multiplying the volume of goods 103b by n2, and the third volume obtained by multiplying the volume of goods 103c by n3, and uses this as the volume that can be accommodated in the empty area 2042. In addition, n1, n2, and n3 are all integers greater than or equal to 0.

[0237] The calculation unit 605 applies the volume of the contained goods and the capacity to be contained to the following formula 2 to calculate the filling rate (S135).

[0238] Fill rate (%) = (Volume of goods already contained) / (Volume of goods already contained + Capacity to be contained) × 100… Equation 2

[0239] In this way, the filling rate calculation unit 224 can calculate the ratio of the volume of the goods 103 contained in the accommodating space 101 to the volume of the space in the accommodating space 101 that can accommodate the goods 103, as the filling rate.

[0240] Therefore, a first fill rate can be calculated to appropriately determine how much cargo 103 can be accommodated in the free space of the accommodating space 101.

[0241] Furthermore, if the types of goods to be accommodated in the accommodating space 101 are given in advance, the quantity of goods accommodated can be calculated by dividing the volume of the accommodated goods by the volume of the given types of goods. For example, the types of goods accommodated in the accommodating space 101 can be stored together with the ID identifying the shelf 102 having the accommodating space 101 in the storage unit 225 of the information processing device 220. The storage unit 225 can also store accommodating information that establishes a correspondence between the ID identifying the shelf 102 and the types of goods accommodated in the accommodating space 101 of the shelf 102. In addition, the storage unit 225 of the information processing device 220 can also store goods information that establishes a correspondence between the types of goods and the volumes of various types of goods. The volumes of various types of goods in the goods information are volumes calculated based on the dimensions of goods commonly used in the distribution industry. The accommodating information and goods information are, for example, tables. Therefore, based on the storage information stored in the storage unit 225, the information processing device 220 determines the type of goods 103 stored in the storage space 101 of the shelf 102 and the volume of that type of goods, and divides the calculated volume of the stored goods by the determined volume of the goods, thereby being able to calculate the quantity of the stored goods.

[0242] The calculated quantity of goods can also be output along with the fill rate. For example, if the goods being held are goods 103a, the quantity of goods being held can be calculated by dividing the volume of the held goods by the volume of goods 103a.

[0243] (Variation Example 2)

[0244] In the information processing apparatus 220 of the above-described embodiments, the fill rate of the cargo 103 in the accommodating space 101 of one shelf 102 is calculated, but the fill rate of the cargo 103 in the accommodating space 101 of two or more shelves 102 can also be calculated.

[0245] Figure 28 This diagram illustrates an example of a situation where two or more shelves are housed within the cargo space of a truck, etc. Figure 29 This is a table showing the relationship between the shelves housed in the cargo container and their fill rate.

[0246] like Figure 28 As shown, multiple cage-type trolleys 112 are housed in a cargo box 106 with a storage space 105. The cargo box 106 can be, for example, a truck box. The cargo box 106 is an example of a second storage unit. The second storage unit is not limited to the cargo box 106; it can also be a container or a warehouse.

[0247] The receiving space 105 is an example of the second receiving space. The receiving space 105 has a volume capable of accommodating multiple cage trolleys 112. In the modified example 2, the receiving space 105 can accommodate 6 cage trolleys 112. Since the receiving space 105 can accommodate multiple cage trolleys 112, the receiving space 105 is larger than the receiving space 111.

[0248] The cage-type trolley 112 has a storage space 111 capable of accommodating multiple goods 103. The cage-type trolley 112 is one example of a storage section. The storage section of the modified example 2 is not limited to the cage-type trolley 112 or a roller container trolley, as long as it is a movable container. The storage space 111 is an example of a first storage space. Alternatively, the shelf 102 described in embodiment 1 can also be accommodated in the storage space 105.

[0249] The multiple goods 103 are not directly contained in the cargo box 106, but are contained in multiple cage trolleys 112. Furthermore, the cage trolleys 112 containing the multiple goods 103 are contained in the cargo box 106.

[0250] The structure of the calculation unit 503 of the filling rate calculation unit 224 under this condition will be explained.

[0251] Figure 30 This is a block diagram illustrating an example of the structure of the calculation unit for the fill rate calculation unit in Modified Example 2. Figure 31 This is a flowchart of the fill rate calculation process of the calculation unit in the calculation unit of Modified Example 2.

[0252] like Figure 30 As shown, the calculation unit 503 of Modified Example 2 has an acquisition unit 701, a counting unit 702 and a calculation unit 703.

[0253] The acquisition unit 701 acquires the number of cage trolleys 112 that can be accommodated in the cargo box 106 (S141). In the case of Modification 2, since the maximum number of cage trolleys 112 that can be accommodated in the cargo box 106 is 6, 6 are acquired.

[0254] The counting unit 702 counts the number of cage-type trolleys 112 housed in the cargo box 106 (S142). Figure 29 When the cage trolley 112 shown is housed in the cargo box 106, the counting unit 702 counts the number of cage trolleys 112 as 3.

[0255] The calculation unit 703 calculates the second filling rate (S143) of one or more cage trolleys 112 in the cargo box 106. Specifically, the calculation unit 703 can calculate the ratio of the number of cage trolleys 112 contained in the cargo box 106 to the maximum number of cage trolleys 112 that can be contained in the cargo box 106 as the second filling rate. For example, if the cargo box 106 can accommodate a maximum of 6 cage trolleys 112, and 3 cage trolleys 112 are contained in the cargo box 106, then the calculation unit 703 calculates the second filling rate to be 50%.

[0256] Alternatively, the calculation unit 703 can calculate the filling rate of the cargo 103 relative to each cage trolley 112 housed in the cargo box 106, and use the calculated filling rate to calculate the filling rate of the cargo 103 relative to the second accommodating space. Specifically, the calculation unit 703 can also calculate the average filling rate of the cargo 103 relative to the cage trolley 112 as the filling rate of the cargo 103 relative to the second accommodating space. In this case, if there is remaining space in the accommodating space 105 of the cargo box 106 that can accommodate the cage trolley 112, the calculation unit 703 can set the filling rate of the remaining space that can accommodate the cage trolley 112 to 0%, and then calculate the average value.

[0257] For example, in Figure 29 The filling rates of the three cage trolleys 112 shown are 70%, 30%, and 20%, respectively. If the cargo box 106 can accommodate a maximum of 6 cage trolleys 112, the average of the filling rates of the 6 cage trolleys 112 (70%, 30%, 20%, 0%, 0%, 0%) can be calculated as 20%, which is the filling rate of the cargo 103 for the second accommodating space.

[0258] Therefore, it is possible to appropriately calculate the second filling rate when more than one cage trolley 112 is accommodated in the accommodating space 105.

[0259] (Variation Example 3)

[0260] Next, variation example 3 will be explained.

[0261] Figure 32 This is a diagram used to illustrate the structure of the cage-type trolley in variation 3.

[0262] Figure 32 Figure (a) shows the cage trolley 112 with the opening and closing part 113 in the closed state. Figure 32 Figure (b) shows the cage trolley 112 with the opening and closing part 113 in the open state.

[0263] The cage carriage 112 of Modification 3 has an opening / closing part 113 for opening and closing the opening 112a. The opening / closing part 113 is a grid-like or mesh-like cover with multiple through holes 113a. Therefore, even when the opening / closing part 113 of the cage carriage 112 is closed, the ranging sensor 210 can measure the three-dimensional shape of the interior of the accommodating space 111 of the cage carriage 112 through the multiple through holes 113a and the opening 112a.

[0264] This is because the electromagnetic waves emitted by the range sensor 210 pass through multiple through holes 113a and openings 112a. Furthermore, since the infrared pattern illuminated by the range sensor 210A passes through multiple through holes 113a and openings 112a, even when the opening / closing part 113 of the cage carriage 112 is closed, the range sensor 210A can still measure the three-dimensional shape of the interior of the housing space 111 of the cage carriage 112 through the multiple through holes 113a and openings 112a. Moreover, with the range sensor 210B, since the two cameras 211B and 212B can capture images of the interior of the housing space 111 through the multiple through holes 113a and openings 112a, the three-dimensional shape of the interior of the housing space 111 of the cage carriage 112 can also be measured.

[0265] Therefore, the information processing device 220 can determine whether the goods 103 are contained in the accommodating space 111. However, when the opening / closing part 113 is closed, it is difficult to obtain the correct filling rate unless the method for calculating the filling rate is switched to a different method than when the opening / closing part 113 is open or when it is not present. Therefore, in Modification 3, the filling rate calculation unit 224 calculates the filling rate using the first method when the opening / closing part 113 is open, and calculates the filling rate using the second method when the opening / closing part 113 is closed.

[0266] Figure 33 This is a block diagram illustrating an example of the structure of the fill rate calculation unit related to Modified Example 3. Figure 34 This is a flowchart of the fill rate calculation process in the fill rate calculation section of Modified Example 3.

[0267] like Figure 33 As shown, the fill rate calculation unit 224 of the modified example 3 includes a detection unit 801, a switching unit 802, a first fill rate calculation unit 803, and a second fill rate calculation unit 804.

[0268] The detection unit 801 uses a spatial three-dimensional model to detect the opening and closing state of the opening and closing part 113 (S151). Specifically, the detection unit 801 uses the spatial three-dimensional model to detect that the opening and closing part 113 is in a closed state when a three-dimensional point cloud exists at its respective positions inside and outside the accommodating space 111 in the front-back direction of the area of ​​the opening 112a of the cage trolley 112 (i.e., the arrangement direction of the ranging sensor 210 and the cage trolley 112). When a three-dimensional point cloud exists only inside the accommodating space 111, the detection unit 801 detects that the opening and closing part 113 is in an open state.

[0269] The switching unit 802 determines whether the opening / closing unit 113 is in an open or closed state (S152), and switches to the next processing based on the determination result.

[0270] When the switching unit 802 determines that the opening / closing unit 113 is in the open state (open state in S152), the first filling rate calculation unit 803 calculates the filling rate using the first method (S153). Specifically, the first filling rate calculation unit 803 performs the same processing as that performed by the filling rate calculation unit 224 in Embodiment 1, thereby calculating the filling rate of the cage trolley 112.

[0271] If the switching unit 802 determines that the opening / closing unit 113 is in the closed state (closed state in S152), the second filling rate calculation unit 804 calculates the filling rate using the second method (S154). Figure 35 Please provide details about the second method.

[0272] Figure 35 This is a diagram used to illustrate an example of the second method for calculating the fill rate.

[0273] like Figure 35 As shown in (a), we envision a scenario where a spatial three-dimensional model 2051 is obtained.

[0274] Figure 35 Image (b) is an enlarged view of region R2 in the 3D spatial model 2051. For example... Figure 35 As shown in (b), the second filling rate calculation unit 804 divides the region R2 into a second part where the opening and closing part 113 is detected and a first part where the cargo 103 is detected.

[0275] The first part is the region containing a three-dimensional point cloud inside the area of ​​the opening 112a. Furthermore, the first part is the portion of the ranging sensor 210 facing the cargo 103 in the direction from the ranging sensor 210 toward the cargo 103. That is, the first part is the portion of the closed opening / closing portion 113 facing the through hole 113a in the direction from the ranging sensor 210 toward the cargo 103. Alternatively, the opening / closing portion 113 may also have a structure with a through hole 113a.

[0276] The second part is the area containing a three-dimensional point cloud on the front side of the area in the front-rear direction of the opening 112a of the cage trolley 112. Furthermore, the second part is the portion where the ranging sensor 210 is not aligned with the cargo 103 in the direction from the ranging sensor 210 toward the cargo 103. That is, the second part is the portion obstructed by the closed opening / closing part 113 in the closed state in the direction from the ranging sensor 210 toward the cargo 103.

[0277] The second fill rate calculation unit 804 voxels the first part and the second part respectively, thereby generating Figure 35 Voxel data 2052 is shown in (c). In voxel data 2052, the white areas without shadows are the voxelized areas of the second part, and the shadowed areas are the voxelized areas of the first part.

[0278] Furthermore, the second fill rate calculation unit 804, for the white area corresponding to the area of ​​the opening / closing section 113, infers whether goods 103 exist inside the opening / closing section 113. Specifically, the second fill rate calculation unit 804 assigns scores based on the probability of goods presence for the 26 voxels adjacent to the voxels of the point shadow where goods 103 exist in the voxelized area. Furthermore, for voxels represented by white areas adjacent to multiple voxels containing goods 103, the scores are summed. The second fill rate calculation unit 804 performs this processing on all voxels containing goods 103, and determines that goods 103 exist for voxels represented by white areas whose total score is above any threshold. For example, if the second fill rate calculation unit 804 sets any threshold to 0.1, since goods 103 are determined to exist in all areas, therefore... Figure 35 As shown in (e), a cargo model 2053 is capable of calculating the shape of the area obscured by the opening / closing part 113.

[0279] Thus, since the information processing device 220 infers the shape of the second part of the distance sensor that is not aligned with the object being measured based on the shape of the first part of the distance sensor 210 that is aligned with the cargo 103, it can properly infer the three-dimensional model of the object even when there is a second part.

[0280] Furthermore, when a rule is attached that the goods 103 are arranged without gaps inside the cage-type trolley 112, such as Figure 36 As shown, the second fill rate calculation unit 804 can extract the outline R3 of a region where one or more goods 103 are disposed, and determine that the area inside the extracted outline R3 is a region where goods 103 exist. Furthermore, the second fill rate calculation unit 804 can also use the three-dimensional point cloud in the region of the multiple through holes 113a of the opening and closing part 113 to infer the region of the opening and closing part 113 inside the outline R3.

[0281] In the filling rate measurement method of Modified Example 3, the cage trolley 112 also has multiple through holes 113a and an opening / closing part 113 for opening and closing the openings 112a. In the filling rate measurement method, it is further determined whether the opening / closing part 113 is in an open state or a closed state. When the opening / closing part 113 is in an open state, the cargo model 2034 is estimated by extraction and estimation, similar to the filling rate calculation unit 224 in Embodiment 1. When the opening / closing part 113 is in a closed state, the filling rate calculation unit 224 estimates the second part that is blocked by the opening / closing part 113 based on multiple first parts corresponding to the multiple through holes 113a in the voxel data 2031 of the spatial three-dimensional model 2011, and uses the multiple first parts and the estimated second parts to assemble the three-dimensional model 2032 to estimate the cargo model 2034.

[0282] Therefore, even when the cargo 103 is housed in a cage-type trolley 112 which has an opening and closing section 113 for opening and closing the opening 112a, the method for inferring the cargo model 2034 is switched between the first method and the second method according to the opening and closing state of the opening and closing section 113, so the three-dimensional model of the object can be appropriately inferred.

[0283] (Variation Example 4)

[0284] Figure 37 This is a diagram used to illustrate the method for generating the spatial three-dimensional model of variation example 4.

[0285] like Figure 37 As shown, when generating a spatial 3D model, the 3D measurement system 200 processes the same methods as the model generation unit 223, and can also integrate the measurement results of multiple range sensors 210. In this case, the 3D measurement system 200 determines the position and orientation of the multiple range sensors 210 through pre-calibration, and integrates the multiple measurement results based on the determined position and orientation of the multiple range sensors 210, thereby generating a spatial 3D model containing 3D point clouds with minimal occlusion.

[0286] (Variation Example 5)

[0287] Figure 38 and Figure 39 This is a diagram used to illustrate the method for generating the spatial three-dimensional model of variation example 5.

[0288] like Figure 38As shown, even when generating a spatial 3D model, the 3D measurement system 200 can move at least one of the cage carriage 112 and a ranging sensor 210 across the measurement area R1 of the ranging sensor 210, and during the movement, multiple timing intervals will integrate multiple measurement results obtained by the ranging sensor 210. The cage carriage 112 can be moved, for example, by being transported by an Automated Guided Vehicle (AGV) 1101 across the measurement area R1 of the ranging sensor 210.

[0289] In this case, the information processing device 220 calculates the relative position and orientation between the cage carriage 112 and a ranging sensor 210 at each timing when the measured results are obtained. For example, as Figure 39 As shown, the information processing device 220 obtains measurement results 2010 from the ranging sensor 210 and position information 2061 of the unmanned transport vehicle 1101 from the unmanned transport vehicle 1101. Measurement results 2010 include a first measurement result measured by the ranging sensor 210 at a first timing and a second measurement result measured at a second timing. The first timing and the second timing are different from each other. Position information 2061 includes a first position of the unmanned transport vehicle 1101 at the first timing and a second position of the unmanned transport vehicle 1101 at the second timing. The first position and the second position are different from each other. Position information 2061 is the self-position of the unmanned transport vehicle 1101 at multiple timings inferred from the unmanned transport vehicle 1101.

[0290] The vehicle's own position can be inferred using existing methods. For example, the unmanned transport vehicle 1101 is positioned at a specific location, and its position can be inferred by reading specific location information from a marker or tag containing specific location information indicating that specific location. Alternatively, the specific location information can be read from the marker or tag, and the position can be inferred based on the distance and direction traveled from the inferred specific location. Alternatively, the unmanned transport vehicle 1101 can send the read specific location information and the distance and direction traveled from the specific location to the information processing device 220, which infers the position of the unmanned transport vehicle 1101 based on the specific location information and the distance and direction traveled from the specific location. Furthermore, the position of the unmanned transport vehicle 1101 can also be inferred using images captured by a camera located on the exterior of the unmanned transport vehicle 1101.

[0291] The information processing device 220 extracts four opening endpoints 2016 in the shelf area 2014, which are the four corner positions of each opening 2015, based on measurement results 2010 including multiple measurement results obtained from measuring the cage trolley 112 from different viewpoints and position information 2061. The information processing device 220 can also determine, based on the first position included in the position information 2061, an area where the opening 112a of the cage trolley 112 in the first measurement result is more likely to exist, and process the determined area to calculate the measurement coordinate system. The information processing device 220 can also determine, based on the second position included in the position information 2061, an area where the opening 112a of the cage trolley 112 in the second measurement result is more likely to exist, and process the determined area to calculate the measurement coordinate system.

[0292] Alternatively, the information processing device 220 can integrate the first and second measurement results based on the first and second positions contained in the location information 2061 to generate a spatial 3D model containing a 3D point cloud with less occlusion. This allows for the calculation of a more accurate fill rate.

[0293] Alternatively, location information 2061 may contain only a location at a specific time, and measurement result 2010 may contain only a measurement result at a specific time.

[0294] When multiple unmanned transport vehicles pass through the measurement area R1 of the ranging sensor 210 one by one, the information processing device 220 can also calculate the filling rate of each cage trolley 112 based on the measurement results of the cage trolleys 112 transported by each unmanned transport vehicle.

[0295] (Variation Example 6)

[0296] In Variation 6, the measurement area of ​​the ranging sensor is described.

[0297] Figure 40 This diagram illustrates an example of using a single ranging sensor to measure multiple cage trolleys.

[0298] like Figure 40 As shown, a single ranging sensor 210 is configured such that all of the multiple cage carriages 112 of the measured object are contained within the measuring area R10 of the ranging sensor 210. For example, the ranging sensor 210 may be configured such that the side of the cage carriage 112 furthest from the ranging sensor 210 is included in the maximum length of the measuring area R10 in the measuring direction.

[0299] Figure 41This diagram illustrates an example of using two ranging sensors to measure multiple cage trolleys.

[0300] like Figure 41 As shown, the two ranging sensors 210a and 210b are configured such that the entire range of the multiple cage-like trolleys 112 of the measured object is contained within the measurement areas R11 and R12 of these ranging sensors 210a and 210b. Furthermore, the two ranging sensors 210a and 210b are configured, for example, such that the length 902 of the overlapping area R13 of the measurement areas R11 and R12 in the ranging direction D1 is longer than the length 901 of the cage-like trolley 112 in the ranging direction D1. Additionally, lengths 901 and 902 are lengths (heights) in the ranging direction D1 based on the arrangement surface of the cage-like trolley 112. That is, the overlapping area R13 has a length 902 in the ranging direction D1 that is greater than the length 901 of the cage-like trolley 112. This maximizes the number of cage-like trolleys 112 that can be measured by the two ranging sensors 210a and 210b. Additionally, the ranging direction D1 is the direction in which ranging sensors 210a and 210b measure distances. Figure 40 and Figure 41 In this diagram, the ranging direction D1 is along the vertical direction, but the direction along which the ranging direction D1 is along is not limited to the vertical direction. The ranging direction D1 can also be along the horizontal direction.

[0301] Each ranging sensor 210a, 210b is related to Figure 40 The ranging sensor 210 is the same sensor, and its measurement area is also the same size. Figure 40 In this context, a single ranging sensor 210 can measure a maximum of four cage-type trolleys 112. Figure 41 In this configuration, by arranging two range sensors 210a and 210b such that the height 902 of the overlapping area R13 is higher than the height 901 of the cage trolley 112, one more cage trolley 112 can be configured. As a result, nine cage trolleys 112 can be measured using two range sensors 210a and 210b, which is more than twice the number of cage trolleys 112 that can be measured using one range sensor 210.

[0302] Figure 42 This diagram illustrates an example of using three distance sensors to measure multiple cage trolleys.

[0303] like Figure 42As shown, three ranging sensors 210a, 210b, and 210c are configured such that all the multiple cage-like trolleys 112 of the measured object are contained within the measurement areas R21, R22, and R23 of these ranging sensors 210a, 210b, and 210c. Furthermore, the three ranging sensors 210a, 210b, and 210c are configured, for example, such that all the multiple cage-like trolleys 112 of the measured object are contained within an area R24 where at least two of the measurement areas R21, R22, and R23 overlap. Thus, all of the multiple cage-like trolleys 112 are measured by the multiple ranging sensors. Therefore, a spatial 3D model containing a 3D point cloud with less occlusion can be generated.

[0304] (Variation Example 7)

[0305] The three-dimensional measurement system 200A of the modified example 7 will be described.

[0306] Figure 43 This is a block diagram representing the characteristic structure of the three-dimensional measurement system related to variation 7.

[0307] The three-dimensional measurement system 200A of Modification 7 differs from the three-dimensional measurement system 200 of Embodiment 1 in that it includes two ranging sensors 210a and 210b. Furthermore, the information processing device 220A of Modification 7 differs from the three-dimensional measurement system 200 of Embodiment 1 in that it includes the constituent elements of the information processing device 220 of Embodiment 1, and also includes an integration unit 226. Here, we will mainly explain the differences from Embodiment 1.

[0308] The acquisition unit 221 acquires measurement results from multiple ranging sensors 210a and 210b. Specifically, the acquisition unit 221 acquires a first measurement result obtained from the ranging sensor 210a and a second measurement result obtained from the ranging sensor 210b. The first measurement result includes a first spatial three-dimensional model generated by the ranging sensor 210a. The second measurement result includes a second spatial three-dimensional model generated by the ranging sensor 210b.

[0309] Integration unit 226 integrates the first spatial 3D model with the second spatial 3D model. Specifically, integration unit 226 integrates the first spatial 3D model with the second spatial 3D model based on the position and orientation (external parameters) of range sensor 210a and range sensor 210b stored in storage unit 225. Thus, integration unit 226 generates an integrated spatial 3D model. The position and orientation of range sensor 210a and range sensor 210b stored in storage unit 225 are generated through pre-calibrated data.

[0310] The coordinate system calculation unit 222, the model generation unit 223, and the fill rate calculation unit 224 use the integrated spatial three-dimensional model as the spatial three-dimensional model and perform the processing described in Embodiment 1.

[0311] Figure 44 This is a flowchart of a method for measuring the fill rate using the information processing device related to Modification 7.

[0312] The information processing device 220A acquires multiple spatial three-dimensional models from the ranging sensors 210a and 210b (S111a). The multiple spatial three-dimensional models include a first spatial three-dimensional model and a second spatial three-dimensional model. At this time, the information processing device 220 also acquires images of the measured object from the ranging sensors 210a and 210b.

[0313] The information processing device 220A integrates multiple spatial three-dimensional models to generate an integrated spatial three-dimensional model (S111b).

[0314] The information processing device 220A acquires the three-dimensional model stored in the storage unit 225 (S112).

[0315] Steps S113 to S116 are the same as in Implementation Method 1, except that an integrated spatial three-dimensional model is used instead of a spatial three-dimensional model, so the explanation is omitted.

[0316] (Variation Example 8)

[0317] In the information processing apparatus 220 of Embodiment 2, line segments are detected based on a two-dimensional image including an RGB image and a depth image. From the detected line segments, the shape of the opening accommodating the three-dimensional model is determined. However, the method is not limited to determining the shape of the opening based on a two-dimensional image. The information processing apparatus 220 may also determine the shape of the opening based on the measurement results of the ranging sensor 210 or a spatial three-dimensional model. For example, from the measurement results of the ranging sensor 210 or the spatial three-dimensional model, a larger number of three-dimensional point clouds arranged at a certain interval or less along a certain direction may be detected as line segments, and the shape of the opening accommodating the three-dimensional model may be determined from the detected line segments.

[0318] (Other implementation methods)

[0319] The above description relates to the filling rate measurement method of this disclosure based on the above embodiments, but this disclosure is not limited to the above embodiments.

[0320] For example, in the above embodiments, it is described that each processing unit of an information processing device, etc., is implemented by a CPU and a control program. Alternatively, the constituent elements of this processing unit may each be composed of one or more electronic circuits. These one or more electronic circuits can be general-purpose circuits or dedicated circuits. Among these one or more electronic circuits, for example, semiconductor devices, integrated circuits (ICs), or large-scale integration (LSIs) may be included. ICs or LSIs can be integrated onto a single chip or onto multiple chips. Here they are referred to as ICs or LSIs, but depending on the degree of integration, they are called system LSIs, very large-scale integration (VLSIs), or ultra-large-scale integration (ULSIs). Furthermore, field-programmable gate arrays (FPGAs) that are programmable after LSI manufacturing can also be used for the same purpose.

[0321] Furthermore, the general or specific form of this disclosure can also be implemented by a system, apparatus, method, integrated circuit, or computer program. Alternatively, it can be implemented by a computer-readable, non-transitory recording medium such as an optical disc, HDD (Hard Disk Drive), or semiconductor memory storing the computer program. Furthermore, it can be implemented by any combination of system, apparatus, method, integrated circuit, computer program, and recording medium.

[0322] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that can be conceived by those skilled in the art, or forms achieved by arbitrarily combining the constituent elements and functions of the embodiments without departing from the spirit of this disclosure.

[0323] Industrial applicability

[0324] This disclosure is applicable as a fill rate measurement method, information processing device, program recording medium, etc., capable of calculating the fill rate of a measured object.

[0325] Explanation of icon numbers

[0326] 101, 105, 111 Accommodation space; 102 Shelf; 102a, 112a Openings; 103, 103a-103c Goods; 104 Marking; 106 Cargo box; 112 Cage trolley; 113 Opening and closing part; 113a Through hole; 200, 200A Three-dimensional measurement system; 210, 210A, 210B, 210a, 210b Distance sensors; 211 Laser irradiation unit; 211A Infrared pattern irradiation unit; 211B, 212B Camera; 212 Laser light receiving unit; 212A Infrared camera; 213A Infrared pattern; 220, 220A Information processing device; 221, 701 Acquisition unit; 22 2. Coordinate system calculation unit (222A, 222B, 222C); Model generation unit (223); Fill rate calculation unit (224); Storage unit (225); Integration unit (226); Auxiliary unit (301); Calculation units (302, 313, 323, 503, 605, 703); Detection units (311, 321, 401); Extraction units (312, 322, 501, 321C); Generation unit (402); Volume calculation unit (403); Estimation unit (502); Cargo volume calculation unit (601); Region segmentation unit (602); Predetermined cargo measurement unit (603); Region estimation unit (604); Counting unit (702); Detection unit (801); Switching unit (802); First fill rate calculation unit (803) ; 804 Second Fill Rate Calculation Unit; 901 Length; 902 Width; 1101 Unmanned Transport Vehicle; 1331 Line Segment Detection Unit; 1332 Opening Extraction Unit; 1333 Endpoint Calculation Unit; 2000 Measurement Coordinate System; 2001, 2021 Images; 2002 Adjustment Marks; 2003 Overlay Image; 2004 Sensor Coordinate System; 2005, 2017, 2026 Rotation Matrices; 2006, 2018, 2027 Translation Vectors; 2010 Measurement Results; 2011, 2051 Spatial 3D Model; 2012, 2022, 2032 Accommodation 3D Model; 2013 Position Information; 2014 Shelf Area; 2015 Opening; 2016 Opening endpoint; 2023 Marker; 2024 Marker area; 2025 Pattern outline; 2031, 2052 Voxel data; 2033 Cargo area; 2034, 2053 Cargo model; 2041 Occupied area; 2042 Empty area; 2061 Location information; 2101 RGB image; 2102, 2112, 2121 Line segment images; 2103, 2113, 2122 Line segments; 2111 Depth image; Point P1; Measurement areas R1, R10, R11, R12, R21, R22, R23; R2 area; R3 Outline; Overlapping area of ​​R13, R24.

Claims

1. A method for measuring fill rate, A spatial three-dimensional model is obtained by measuring a distance sensor opposite to the receiving part through the opening of a receiving part that has a receiving space for accommodating the object to be measured. Obtain a three-dimensional model of the container, which is a three-dimensional model of the container that does not contain the object to be measured. Obtain a two-dimensional image of the opening and the position and pose information corresponding to the two-dimensional image; Using the aforementioned three-dimensional model, the line segments representing the shape of the aforementioned opening in the aforementioned two-dimensional image are determined; Based on the above position and posture information and the determined line segment, calculate the position of the above opening in three-dimensional space; Based on the calculated location of the opening, establish a correspondence between the location of the three-dimensional model and the location of the spatial three-dimensional model; Based on the above-established corresponding three-dimensional model of the containment space and the above-established three-dimensional model of the space, the three-dimensional model of the object that serves as the three-dimensional model of the object to be measured within the containment space is inferred. Using the aforementioned three-dimensional model of the container and the aforementioned three-dimensional model of the object, the fill rate of the measured object in the aforementioned container space is calculated.

2. The filling rate measurement method as described in claim 1, The aforementioned two-dimensional image includes an RGB image generated by capturing the aforementioned opening with a camera; The above position and posture information indicates the position and posture of the camera when it is taking pictures of the opening.

3. The filling rate measurement method as described in claim 1, The aforementioned two-dimensional image includes a depth image generated based on the measurement of the aforementioned opening performed by the aforementioned ranging sensor; The above position and posture information indicates the position and posture of the ranging sensor when measuring the opening.

4. The filling rate measurement method as described in claim 1, The aforementioned two-dimensional image includes at least one of RGB image, grayscale image, infrared image, and depth image; The aforementioned RGB image is generated by capturing the aforementioned opening with a camera; The depth image described above is generated based on the measurement results from the aforementioned ranging sensor.

5. The filling rate measurement method as described in claim 4, In determining the above-mentioned line segments, the line segments are determined based on both the line segments determined according to the above-mentioned RGB image and the line segments determined according to the above-mentioned depth image.

6. The method for measuring filling rate as described in any one of claims 1 to 5, The aforementioned ranging sensors include at least one of a Time-of-Flight (ToF) sensor and a stereo camera.

7. The method for measuring filling rate as described in any one of claims 1 to 5, The aforementioned ranging sensors include a first ranging sensor and a second ranging sensor; The first measurement area of ​​the first ranging sensor and the second measurement area of ​​the second ranging sensor have an overlapping area.

8. The filling rate measurement method as described in claim 7, The aforementioned overlapping area has a length greater than or equal to the length of the object being measured in the direction of the ranging sensor.

9. The filling rate measurement method as described in claim 7, The aforementioned overlapping area includes the entire range of the measured object.

10. The method for measuring filling rate as described in any one of claims 1 to 5, The aforementioned receiving portion moves relative to the aforementioned ranging sensor in a direction that intersects with the ranging direction of the aforementioned ranging sensor; The aforementioned three-dimensional spatial model is generated using the first measurement result obtained by the aforementioned ranging sensor at the first timing and the second measurement result obtained at the second timing.

11. The method for measuring filling rate as described in any one of claims 1 to 5, The positions of the aforementioned three-dimensional models and the aforementioned spatial three-dimensional models are established using rotation matrices and translation vectors.

12. The method for measuring filling rate as described in any one of claims 1 to 5, Calculate the second filling rate of one or more of the multiple accommodating portions for the second accommodating portion, wherein the second accommodating portion has a second accommodating space for accommodating one or more of the aforementioned accommodating portions.

13. The method for measuring filling rate as described in any one of claims 1 to 5, The third filling rate of the measured object contained in one or more of the above-mentioned accommodating portions with respect to the second accommodating portion is calculated, wherein the second accommodating portion has a second accommodating space for accommodating one or more of the above-mentioned accommodating portions.

14. The filling rate measurement method as described in claim 1, Based on the calculated position of the opening, a correspondence is established between the position of the three-dimensional model and the position of the spatial three-dimensional model.

15. An information processing device, It has a processor and memory; The processor described above uses the memory described above. A spatial three-dimensional model is obtained by measuring a distance sensor opposite to the receiving part through the opening of a receiving part that has a receiving space for accommodating the object to be measured. Obtain a three-dimensional model of the container, which is a three-dimensional model of the container that does not contain the object to be measured. Obtain a two-dimensional image of the opening and the position and pose information corresponding to the two-dimensional image; Using the aforementioned three-dimensional model, the line segments representing the shape of the aforementioned opening in the aforementioned two-dimensional image are determined; Based on the above position and posture information and the determined line segment, calculate the position of the above opening in three-dimensional space; Based on the calculated location of the opening, establish a correspondence between the location of the three-dimensional model and the location of the spatial three-dimensional model; Based on the above-established corresponding three-dimensional model of the containment space and the above-established three-dimensional model of the space, the three-dimensional model of the object that serves as the three-dimensional model of the object to be measured within the containment space is inferred. Using the aforementioned three-dimensional model of the container and the aforementioned three-dimensional model of the object, the fill rate of the measured object in the aforementioned container space is calculated.

16. A program recording medium storing a program for causing a computer to execute a fill rate measurement method, In the above-mentioned method for measuring fill rate, A spatial three-dimensional model is obtained by measuring a distance sensor opposite to the receiving part through the opening of a receiving part that has a receiving space for accommodating the object to be measured. Obtain a three-dimensional model of the container, which is a three-dimensional model of the container that does not contain the object to be measured. Obtain a two-dimensional image of the opening and the position and pose information corresponding to the two-dimensional image; Using the aforementioned three-dimensional model, the line segments representing the shape of the aforementioned opening in the aforementioned two-dimensional image are determined; Based on the above position and posture information and the determined line segment, calculate the position of the above opening in three-dimensional space; Based on the calculated location of the opening, establish a correspondence between the location of the three-dimensional model and the location of the spatial three-dimensional model; Based on the above-established corresponding three-dimensional model of the containment space and the above-established three-dimensional model of the space, the three-dimensional model of the object that serves as the three-dimensional model of the object to be measured within the containment space is inferred. Using the aforementioned three-dimensional model of the container and the aforementioned three-dimensional model of the object, the fill rate of the measured object in the aforementioned container space is calculated.

Images