Material pile measurement system and material pile measurement method

Abstract

The shape and volume of the material pile are measured in a simple manner. [Solution] The measurement system 10 is provided on a movable body that can move on the surface of the board and includes an image capture unit 21 that photographs the pile of material and acquires a captured image, an image capture direction acquisition unit 25 that acquires the shooting direction of the image capture unit, a transport device direction acquisition unit 12 that acquires the direction of the transport device that the transport device faces when unloading the material, and a measurement unit 30 that measures at least one of the shape and volume of the pile of material in the captured image. The measurement unit 30 identifies the region of the pile of material in the captured image, determines the relative shooting direction which is the shooting direction with respect to the device direction from the shooting direction and the device direction, and measures at least one of the shape and volume of the pile of material based on the region of the pile of material and the relative shooting direction.

Description

Technical Field

[0001] The present invention relates to a measurement system for a material pile and a method for measuring a material pile.

Background Art

[0002] Patent Document 1 discloses a measurement method for measuring the shape of a raw material pile in a raw material yard using an aerial photograph of the raw material yard continuously taken from above the raw material yard using an unmanned aircraft.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The measurement method described in Patent Document 1 acquires a plurality of photographed images using an unmanned aircraft and analyzes the plurality of photographed images to measure the shape of the raw material pile. In such a measurement method, since it is necessary to acquire an aerial photograph before measurement, not only does the time required to obtain the measurement result become long, but measurement is difficult when an unmanned aircraft cannot be used in rainy weather or the like.

[0005] An object of the present invention is to easily measure at least one of the shape and volume of a material pile.

Means for Solving the Problems

[0006] The present invention is a measurement system for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device are dropped from the transport device and unloaded onto a platform. The measurement system is provided on a movable body that can move on the platform and comprises: an image capture unit that photographs the pile of materials and acquires an image; an image direction acquisition unit that acquires the shooting direction of the image capture unit; a transport device direction acquisition unit that acquires the direction of the transport device when unloading materials; and a measurement unit that measures at least one of the shape and volume of the pile of materials in the image captured by the image capture unit. The measurement unit identifies the region of the pile of materials in the image captured by the image capture unit, determines the relative shooting direction, which is the shooting direction with respect to the device direction, from the shooting direction acquired by the image direction acquisition unit and the device direction acquired by the transport device direction acquisition unit, and measures at least one of the shape and volume of the pile of materials based on the region of the pile of materials and the relative shooting direction.

[0007] The present invention is a measurement system for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device are dropped from the transport device and unloaded onto a platform. The measurement system is provided on a mobile body that can move on the platform and includes a 3D scanner that irradiates the pile of materials with laser light to acquire point cloud coordinate data indicating the presence or absence of objects, an irradiation direction acquisition unit that acquires the irradiation direction of the 3D scanner, a transport device direction acquisition unit that acquires the direction the transport device faces when unloading materials, and a measurement unit that measures at least one of the shape and volume of the pile of materials in the point cloud coordinate data acquired by the 3D scanner. The measurement unit identifies the region of the pile of materials in the point cloud coordinate data acquired by the 3D scanner, determines the relative irradiation direction, which is the irradiation direction with respect to the device direction, from the irradiation direction acquired by the irradiation direction acquisition unit and the device direction acquired by the transport device direction acquisition unit, and measures at least one of the shape and volume of the pile of materials based on the region of the pile of materials and the relative irradiation direction.

[0008] The present invention relates to a measurement method for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device are dropped from the transport device and unloaded onto a surface. In this measurement method, an image capture unit provided on a movable body that can move on the surface of the device is used to capture an image of the pile of materials, the shooting direction of the image capture unit is obtained, the direction of the transport device facing when unloading the materials is obtained, the region of the pile of materials in the image captured by the image capture unit is identified, the relative shooting direction, which is the shooting direction with respect to the device direction, is determined from the obtained shooting direction and the obtained device direction, and at least one of the shape and volume of the pile of materials is measured based on the region of the pile of materials and the relative shooting direction.

[0009] The present invention relates to a measurement method for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device are dropped from the transport device and unloaded onto a surface. In this measurement method, a 3D scanner mounted on a movable body that can move on the surface of the device is used to irradiate the pile of materials with laser light to acquire point cloud coordinate data indicating the presence or absence of objects, the irradiation direction of the 3D scanner is acquired, the direction of the transport device facing when unloading the materials is acquired, the region of the pile of materials is identified within the point cloud coordinate data acquired by the 3D scanner, the relative irradiation direction, which is the irradiation direction with respect to the device direction, is determined from the acquired irradiation direction and the acquired device direction, and at least one of the shape and volume of the pile of materials is measured based on the region of the pile of materials and the relative irradiation direction. [Effects of the Invention]

[0010] According to the present invention, at least one of the shape and volume of a material pile can be easily measured. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram of the material pile measurement system common to each embodiment. [Figure 2] This diagram shows the shooting direction of the image capture unit installed on the mobile device. [Figure 3] This is a block diagram showing the measurement process of the measurement system according to the first embodiment. [Figure 4] It is a flowchart showing the measurement process by the measurement system. [Figure 5] It is a flowchart showing the details of the measurement process in step S16 of FIG. 4. [Figure 6] It is a diagram showing an example of a correct answer example compared with the measurement result by the measurement system. [Figure 7] It is a diagram showing the first measurement result by the measurement system. [Figure 8] It is a diagram showing the second measurement result by the measurement system. [Figure 9] It is a diagram showing the third measurement result by the measurement system. [Figure 10] It is a block diagram showing the measurement process of the measurement system according to the first modification example that does not use position information. [Figure 11] It is a block diagram showing the measurement process of the measurement system according to the second modification example that does not use distance information. [Figure 12] It is a block diagram showing the measurement process of the measurement system according to the second embodiment. [Figure 13] It is a block diagram showing the measurement process of the measurement system according to the third modification example including both a camera and a 3D scanner.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] (First Embodiment) Referring to FIGS. 1 to 9, the measurement system 10 and the measurement method for the material pile 4 according to the first embodiment of the present invention will be described. In this embodiment, the transport device for transporting materials such as embankment materials is a self-propelled dump-up device 1, and the case where the moving body for processing the material pile 4 formed by unloading from the dump-up device 1 is a construction machine 2 such as a bulldozer and a wheel loader will be described. Note that the transport device may be a fixed device such as a belt conveyor.

[0014] FIG. 1 is a schematic diagram of a measurement system 10 for a material pile 4 according to the first embodiment. FIG. 1 is a view of the dump-up device 1, the construction machine 2, and the material pile 4 on the work surface as seen from above the work surface. The dump-up device 1, the construction machine 2, and the measurement unit 3 are configured to be able to communicate with each other via a network. In this embodiment, the construction machine 2 may be operated by an operator or may be capable of automatic (autonomous) operation without operator operation.

[0015] When the material loaded on the dump-up device 1 is unloaded from the dump-up device 1 onto the work surface, a material pile 4 is formed on the work surface. Then, the material pile 4 is leveled on the work surface by the construction machine 2. Here, the construction machine 2 uses the shape and volume of the material pile 4 measured by the measurement unit 3 and performs a leveling operation manually or automatically according to the acquired self-position and orientation.

[0016] The measurement unit 3 is, for example, a computer and includes a CPU (Central Processing Unit) that performs arithmetic processing, a ROM (Read-Only Memory) that stores a control program and the like executed by the CPU, and a RAM (Random Access Memory) that stores the arithmetic results of the CPU and the like. Note that the arithmetic processing is not limited to the CPU, and a GPU (Graphics Processing Unit) may be used alternatively or additionally.

[0017] The construction machine 2 is provided with a camera 21 (image capturing unit), and the measurement unit 3 measures the shape and volume of the material pile 4 using the captured image acquired by the camera 21. The camera 21 may be a monocular camera or may be a compound eye camera as in the first modification example (FIG. 10) described later. In addition, a plurality of sensors are provided on the dump-up device 1 and the construction machine 2, and the data acquired by these sensors is used for the measurement processing of the measurement unit 3.

[0018] In this embodiment, when the transport device is a dump-up device 1, the cargo bed is dumped up with the materials loaded on it, and then the dump-up device 1 moves forward, causing the materials to fall onto the platform and be unloaded. Hereinafter, the direction in which the dump-up device 1 faces when unloading by such a dump-up device 1 will be referred to as "device direction A".

[0019] The pile of materials 4 unloaded from the dump-up device 1 is configured symmetrically with respect to the device direction A, as shown in Figure 1. In the example in Figure 1, the shape of the pile of materials 4 is configured to consist of two overlapping ellipses of different sizes when viewed from above, but this shape is just one example. The pile of materials 4 unloaded from the dump-up device 1 can be any shape configured symmetrically with respect to the device direction A.

[0020] After the material pile 4 is formed by unloading using the dump-up device 1, the camera 21 takes an image of the material pile 4. Here, since the material pile 4 is symmetrical with respect to the device direction A, it is ideal that the shooting direction B of the camera 21 is perpendicular to the device direction A. By shooting in this way, the shape of one side of the material pile 4 can be captured with respect to the device direction A, which is the center of symmetry of the material pile 4, and the shape of the material pile 4 on the opposite side will be the same as the other side, thus improving the accuracy of measuring the overall shape and volume of the material pile 4.

[0021] The shooting direction B of camera 21 is preferably the direction toward the highest point O of the material pile 4 in the direction of the board surface, i.e., the direction of the paper in Figure 2, that is, the direction in which the apparatus direction A and the shooting direction B intersect at the highest point O of the material pile 4. By having such a shooting direction B, the area on one side of the material pile 4 that is not photographed relative to the apparatus direction A can be reduced. In the height direction, the shooting direction B may be horizontal to the board surface or it may be the direction toward the highest point of the material pile 4.

[0022] However, it is difficult to orient the construction machine 2 so that the shooting direction B of the camera 21 is perfectly perpendicular to the device direction A. Furthermore, if the work surface is unpaved, the posture of the construction machine 2 may be unstable, and the shooting direction B of the camera 21 may be tilted. Therefore, in this embodiment, the difference between the shooting direction B and the device direction A is obtained as a three-dimensional rotation angle, and the measurement unit 3 performs measurement processing so that this difference is taken into consideration.

[0023] Figure 2 shows the three-dimensional rotation angle of the camera 21 in shooting direction B. In this figure, the direction on the work surface where the construction machine 2 operates is shown in the xy plane, and the height direction is shown in the z axis. For the sake of explanation, the direction in front of the construction machine 2 is referred to as the x axis, and the direction perpendicular to the x axis on the work surface is referred to as the y axis. The rotation angles around the xyz axes are referred to as the roll angle, pitch angle, and yaw angle, respectively. The yaw angle allows us to obtain the orientation (direction around the z axis) of the construction machine 2, and the roll angle and pitch angle allow us to obtain the attitude angle of the construction machine 2. Using these rotation angles around the xyz axes, the relative shooting direction, which is shooting direction B with respect to the device direction A, can be determined.

[0024] Figure 3 is a block diagram showing the processing of the measurement system 10.

[0025] The dump-up device 1 has a first GNSS sensor 11 capable of acquiring the position of the dump-up device 1 and a first geomagnetic sensor 12 (transport device direction acquisition unit) capable of acquiring the orientation of the dump-up device 1. The first GNSS sensor 11 acquires positional information of the dump-up device 1 using a satellite positioning system (GNSS: Global Navigation Satellite System). The first geomagnetic sensor 12 is, for example, an electronic compass and acquires the orientation of the dump-up device 1. Therefore, the first geomagnetic sensor 12 can acquire the orientation A of the dump-up device 1 when unloading materials.

[0026] Furthermore, the rear end of the center of the dump bed of the dump-up device 1 in the vehicle width direction often corresponds to the highest point O of the material pile 4 during unloading. Therefore, by positioning the first GNSS sensor 11 at the rear end of the center of the dump bed of the dump-up device 1 in the vehicle width direction, the position of the highest point O of the material pile 4 can be determined using the position information acquired by the first GNSS sensor 11 during unloading. If the first GNSS sensor 11 is not located at the center rear end of the dump-up device 1, the position information acquired by the first GNSS sensor 11 should be corrected by the difference in position between the first GNSS sensor 11 and the center rear end of the dump-up device 1.

[0027] In addition to the camera 21, the construction machine 2 is equipped with a second GNSS sensor 22, a second geomagnetic sensor 23 (construction machine orientation acquisition unit), and an attitude angle sensor 24 (attitude angle acquisition unit). The second geomagnetic sensor 23 and the attitude angle sensor 24 may be integrally provided as a shooting direction acquisition unit 25. The second GNSS sensor 22 and the second geomagnetic sensor 23 acquire the position information and orientation of the construction machine 2, respectively. The attitude angle sensor 24 is, for example, an inertial measurement unit (IMU) that acquires the attitude angle of the construction machine 2.

[0028] The second GNSS sensor 22 is located near the camera 21, and the position of the camera 21 can be determined using the position information acquired by the second GNSS sensor 22. If the second GNSS sensor 22 is not located near the camera 21, the acquired position information only needs to be corrected by the difference in position between the camera 21 and the second GNSS sensor 22.

[0029] The measurement unit 3 includes a region identification unit 31, a relative direction calculation unit 32, a shooting distance calculation unit 33, and a shape volume measurement unit 34. The measurement process of the measurement unit 3 using data acquired by the sensors of the dump-up device 1 and the construction machine 2 will be described below.

[0030] First, the camera 21 captures an image of the material pile 4 and generates a captured image. When the region identification unit 31 receives the captured image from the camera 21, it performs segmentation processing on the captured image to identify the region of the material pile 4. Then, the region identification unit 31 performs binarization processing to generate a binarized image indicating the presence or absence of the material pile 4. In this way, measurement results that are independent of the environment around the material pile 4 can be obtained through segmentation processing, and measurement results that are independent of the type and color of the material can be obtained through binarization processing. These processes of the region identification unit 31 may be performed using a trained model which is the result of machine learning that has been performed in advance. For example, deep learning using a neural network can be considered for machine learning.

[0031] Next, the orientations of the dump-up device 1 and the construction machine 2 (the orientations of the dump-up device 1 and the construction machine 2 on the plane of Figure 1) are indicated by the yaw angle, which is the rotation angle around the z-axis, and are acquired by the first geomagnetic sensor 12 and the second geomagnetic sensor 23, respectively. The relative direction calculation unit 32 calculates the difference in yaw angles between the dump-up device 1 during unloading and the construction machine 2 during photography. This difference in yaw angles is the difference in orientation between the dump-up device 1 during unloading and the construction machine 2 during photography.

[0032] The rotational direction of the construction machine 2 in the horizontal plane is indicated by the roll angle and pitch angle, which are acquired by the attitude angle sensor 24. However, if the work surface is unpaved, the attitude of the construction machine 2 may be unstable, and the shooting direction B of the camera 21 may be tilted. In that case, the shooting direction B will be tilted relative to the device direction A by the amount of the attitude angle of the construction machine 2. Therefore, the relative direction calculation unit 32 acquires the attitude angle, which is indicated by the roll angle and pitch angle of the construction machine 2 at the time of shooting, acquired by the attitude angle sensor 24, as the tilt of the shooting direction B. In this way, the relative direction calculation unit 32 calculates the relative shooting direction, which is the shooting direction B relative to the device direction A, from the rotation angles around the xyz axes (the difference between the roll angle, pitch angle, and yaw angle).

[0033] The shooting distance calculation unit 33 receives position information of the dump-up device 1 during unloading, acquired by the first GNSS sensor 11, and position information of the construction machine 2 during shooting, acquired by the second GNSS sensor 22. Since this position information indicates the drop point from the dump-up device 1 and the position of the camera 21 in the xy plane, the shooting distance calculation unit 33 calculates the Euclidean distance between this position information as the shooting distance between the camera 21 and the pile of materials 4.

[0034] The shape-volume measurement unit 34 receives a binarized image showing the region of the material pile 4 identified by the region identification unit 31, the relative shooting direction calculated by the relative direction calculation unit 32, and the shooting distance calculated by the shooting distance calculation unit 33. The shape-volume measurement unit 34 then uses these inputs to measure the shape and volume of the material pile 4.

[0035] Here, the shape-volume measurement unit 34 performs measurement processing using a trained model, which is the result of machine learning performed in advance. In machine learning, training data is used in which the region of the material pile 4 in the binarized image, the relative shooting direction, and the shooting distance are taken as input values, and the shape and volume of the material pile 4 are taken as output values. By using such a trained model, the shape-volume measurement unit 34 can measure the shape and volume of the material pile 4. The training data is obtained by taking multiple images of multiple material piles 4 from different directions multiple times for the input, and by actually measuring the shape and volume of the material pile 4 for the output. The shape of the material pile 4 can be determined, for example, by SfM (Structure from Motion) using multiple images taken from above.

[0036] Furthermore, the shape-volume measurement unit 34 does not necessarily have to use a pre-trained model. The shape-volume measurement unit 34 may measure the shape and volume of the material pile 4 using a mathematical model. For example, the shape-volume measurement unit 34 may correct the binarized image using the relative shooting direction, determine the shape of the material pile 4 by identifying the object from the corrected binarized image, and then measure the volume of the material pile 4 by determining its size using the shooting distance. When such a mathematical model is used, the binarization process in the region identification unit 31 may be omitted. When the binarization process is omitted, the shape of the material pile 4 is displayed by shading, so the shape-volume measurement unit 34 may perform the measurement process using the shading information.

[0037] The shape volume measurement unit 34 may use images other than the binarized image showing the presence or absence of the material pile 4 for measurement. For example, the region identification unit 31 may, after identifying the region of the material pile 4 in the captured image, whiten the area outside the identified region of the material pile 4 to create an RGB image in which the region of the material pile 4 is shown in RGB. The shape volume measurement unit 34 can also measure the volume and capacity of the material pile 4 using such an RGB image.

[0038] In this manner, the measurement unit 3 uses the data acquired by the sensors of the dump-up device 1 and the construction machine 2 to measure the shape and volume of the material pile 4. Alternatively, the measurement unit 3 may measure either the shape or the volume of the material pile 4.

[0039] Figure 4 is a flowchart showing the measurement process performed by the measurement system 10.

[0040] In step S11, the camera 21 captures an image of the material pile 4 and acquires the captured image. In step S12, the second geomagnetic sensor 23 acquires the orientation (yaw angle) of the construction machine 2, and the attitude angle sensor 24 acquires the attitude angle (roll angle and pitch angle) of the construction machine 2. In this way, the shooting direction B (roll angle, pitch angle, and yaw angle) expressed in three dimensions is acquired. In step S13, the second GNSS sensor 22 of the construction machine 2 acquires the construction machine position information (XY coordinates). In step S14, the first geomagnetic sensor 12 acquires the device direction A (yaw angle) of the dump-up device 1 during unloading. In step S15, the first GNSS sensor 11 of the dump-up device 1 acquires the position information (XY coordinates) of the dump-up device 1. In step S16, the measurement unit 3 executes the measurement process.

[0041] Figure 5 is a flowchart showing the details of the measurement process in step S16 of Figure 4.

[0042] In step S161, the region identification unit 31 performs segmentation processing on the captured image acquired in step S11 to identify the region of the material pile 4, and then performs binarization processing to generate a binarized image in which the region of the material pile 4 has been identified.

[0043] In step S162, the relative direction calculation unit 32 calculates the difference in orientation (difference in yaw angle) between the construction machine 2 and the dump-up device 1 from the difference between the orientation (yaw angle) of the construction machine 2 acquired in step S12 and the device direction A (yaw angle) of the dump-up device 1 acquired in step S14. The relative direction calculation unit 32 calculates the relative shooting direction, which is the shooting direction B relative to the device direction A, by combining the difference in orientation with the attitude angles (roll angle and pitch angle) of the construction machine 2 acquired in step S12.

[0044] In step S163, the shooting distance calculation unit 33 calculates the shooting distance between the camera 21 and the material pile 4 by taking the Euclidean distance between the position information of the dump-up device 1 during unloading, which is acquired by the first GNSS sensor 11 in step S13, and the position information of the construction machine 2 during shooting, which is acquired by the second GNSS sensor 22 in step S15.

[0045] In step S164, the shape and volume measurement unit 34 measures the shape and volume of the material pile 4 using the binarized image in which the region of the material pile 4 calculated in step S161 is identified, the relative shooting direction calculated in step S162, and the shooting distance calculated in step S163.

[0046] Next, we will explain the measurement results obtained by the shape volume measurement unit 34 using the trained model, with reference to Figures 6 to 9. Figure 6 shows the correct data, and Figures 7 to 9 show the measurement results.

[0047] The input data used to calculate the measurement results in Figures 7 to 9 are the captured image, the position and orientation of camera 21, and the tilt. The camera position is shown in XY coordinates with the material pile 4 as the origin, and the orientation of camera 21 is shown as the yaw angle relative to the ideal shooting direction (perpendicular to the device direction A). The attitude angle is shown as the roll angle and pitch angle acquired by the attitude angle sensor 24.

[0048] Figure 6 shows the ground truth data. On the left is a height map showing the actual height of material pile 4 obtained by SfM, indicated by shades of gray, and on the right is the measured volume of material pile 4, which is 34.6 m³. 3 This indicates that it was the case.

[0049] Figure 7 shows the results of the first measurement. Figure 7(A) shows the data input to the measurement unit 3. The captured image is shown on the left, and other data is shown on the right. In this example, the position information of the construction machine 2 at the time of shooting (coordinates of camera 21) is X: -12.657m, Y: -0.934m. The shooting direction is 1.530 degrees. The attitude angles of camera 21 are roll angle: 0.200 degrees, pitch angle: -0.700 degrees.

[0050] Figure 7(B) shows a binarized image in which the region of the material pile 4 has been identified, obtained by the region identification unit 31 performing segmentation and binarization processing on the captured image on the left side of Figure 7(A). Figure 7(C) shows the measurement results of the shape volume measurement unit 34 using the data on the right side of Figure 7(A) and the binarized image in which the region of the material pile 4 has been identified in Figure 7(B). In this figure, the height map is shown on the left, and the measured volume of 35.1 m is shown on the right. 3 This indicates that it was the case.

[0051] Similarly, Figure 8 shows the results of the second measurement. As shown on the right side of Figure 8(A), the position information of the construction machine 2 at the time of shooting (coordinates of camera 21) is X: -11.016m, Y: -1.457m. The shooting direction is 3.120 degrees. The attitude angles of camera 21 are roll angle: 0.300 degrees, pitch angle: 0.100 degrees. Figure 8(C) shows the height map obtained from the measurement and the measured volume of 34.9m³. 3 It has been shown that this was the case.

[0052] Similarly, Figure 9 shows the results of the third measurement. As shown on the right side of Figure 9(A), the position information of construction machine 2 at the time of shooting (coordinates of camera 21) is X: -10.290m, Y: 4.367m. The shooting direction is -25.580 degrees. The attitude angles of camera 21 are roll angle: 1.500 degrees, pitch angle: 0.200 degrees. Figure 9(C) shows the height map obtained from the measurement and the measured volume of 34.9m³. 3 It has been shown that this was the case.

[0053] As shown in Figures 7(C), 8(C), and 9(C), the measurement results obtained by the shape volume measurement unit 34 using the trained model were close to the ground truth data in Figure 6. This demonstrates that highly accurate measurement processing can be performed by utilizing a trained model obtained through machine learning.

[0054] In the example described above, the transport device was described as a movable dump-up device 1, but it may also be a fixed device such as a belt conveyor. When the transport device is a belt conveyor, the transport direction of the belt conveyor is set as the device direction, and the position information of the unloading point of the belt conveyor is used as the device position information, thereby enabling measurement processing by the measurement unit 3.

[0055] In the example described above, the measurement unit 3 is provided separately from the dump-up device 1 and the construction machine 2, but the measurement unit 3 may be provided in any manner. For example, the measurement unit 3 may be provided integrally within the construction machine 2, or it may be provided in the cloud. The measurement unit 3 only needs to be able to perform measurement processing by executing a stored program.

[0056] In the above example, the shape-volume measurement unit 34 measured the shape and volume of the material pile 4 using a binarized image, relative shooting direction, and shooting distance, but is not limited to this. Specifically, the relative direction calculation unit 32 may measure the shape and volume of the material pile 4 using a binarized image in which the region of the material pile 4 is identified and the relative shooting direction, without using the shooting distance. An example of measurement processing without using the shooting distance will be explained in the second modified example (Figure 11) described later. The relative direction calculation unit 32 may also measure either the shape or the volume of the material pile 4. Furthermore, the shape-volume measurement unit 34 may measure the shape and volume of the material pile 4 using an RGB image instead of a binarized image.

[0057] In the example described above, the transport device is described as a dump-up device 1. The construction machine 2, which is the mobile unit, may be operated by an operator or operated automatically. When the construction machine 2 is operated automatically, the measured shape and volume of the material pile 4 are used to create a work procedure, including the movement path of the construction machine 2 for processing the material pile 4 on the work surface. When the construction machine 2 is operated by an operator, the efficiency of processing the material pile 4 is improved by the operator knowing the measurement results of the shape and volume of the material pile 4.

[0058] According to the first embodiment described above, the following effects and advantages are achieved.

[0059] In the measurement system 10 of the first embodiment, the measurement unit 3 (region identification unit 31) identifies the region of the material pile 4 in the captured image taken by the camera 21 installed on the construction machine 2. The measurement unit 3 (relative direction calculation unit 32) determines the relative shooting direction B of the camera 21 with respect to the device direction A of the dump-up device 1. Then, the measurement unit 3 (shape volume measurement unit 34) measures at least one of the shape and volume of the material pile 4 based on the identified region of the material pile 4 and the relative shooting direction. In this way, the information necessary for measuring the shape and volume of the material pile 4 can be obtained in a simple manner, so the shape and volume of the material pile 4 can be measured easily.

[0060] Furthermore, the measurement unit 3 (shape and volume measurement unit 34) stores the learning results of machine learning that measure the shape and volume of the material pile 4 based on the identified region of the material pile 4 and the relative shooting direction. By utilizing the learning results of machine learning in this way, the man-hours required to construct the measurement unit 3 (shape and volume measurement unit 34) can be reduced compared to when a mathematical model is used.

[0061] Furthermore, the measurement unit 3 (shape and volume measurement unit 34) measures at least one of the shape and volume of the material pile 4 based on the area of ​​the material pile 4 identified and the relative shooting direction, as well as the shooting distance between the camera 21 and the material pile 4. By additionally utilizing the shooting distance in this way, the measurement unit 3 (shape and volume measurement unit 34) can calculate the measurement results with high accuracy.

[0062] Furthermore, the measurement unit 3 (shooting distance calculation unit 33) determines the shooting distance based on the position information of the dump-up device 1 during unloading, acquired by the first GNSS sensor 11, and the position information of the construction machine 2 during shooting, acquired by the second GNSS sensor 22. By calculating the shooting distance using the measured position information of the dump-up device 1 and the construction machine 2 in this way, the shooting distance becomes closer to the true value, and the measurement unit 3 (shape volume measurement unit 34) can calculate the measurement result with high accuracy.

[0063] Furthermore, the transport device is the dump-up device 1, and the moving body is the construction machine 2. The pile of materials 4 unloaded from the dump-up device 1 is spread and leveled on the work surface by the construction machine 2. When the construction machine 2 processes the pile of materials 4, it is necessary to create a work procedure, including the movement route of the construction machine 2 on the work surface, before starting the work. By measuring and understanding the shape and volume of the pile of materials 4 before the construction machine 2 performs the leveling, an efficient work procedure can be created with high precision, allowing the construction machine 2 to perform efficient leveling.

[0064] (First variation) In the first embodiment, an example was described in which the dump-up device 1 and the construction machine 2 are equipped with a GNSS sensor and a geomagnetic sensor, respectively, but other embodiments are also possible. Below, a first modified example is described in which the dump-up device 1 and the construction machine 2 are not equipped with a GNSS sensor and a geomagnetic sensor, respectively, and the measurement unit 3 calculates the relative shooting direction and shooting distance by other means.

[0065] Figure 10 is a block diagram showing the processing of the first modified measurement system 10A. Compared to the measurement system 10 of the first embodiment shown in Figure 3, the measurement system 10A shown in Figure 10 omits the first GNSS sensor 11 and the first geomagnetic sensor 12 of the dump-up device 1, as well as the second GNSS sensor 22 and the second geomagnetic sensor 23 of the construction machine 2. The captured image acquired by the camera 21 is input to the relative direction calculation unit 32 and the shooting distance calculation unit 33, in addition to the area identification unit 31.

[0066] Camera 21 photographs the dump-up device 1 from before unloading. Here, since the center of the captured image is the shooting direction B, the orientation of the dump-up device 1 shown in the captured image is the relative direction of the device direction A with respect to the shooting direction B. Therefore, the relative direction calculation unit 32 can obtain the relative shooting direction by combining the captured images of the dump-up device 1 before and after unloading.

[0067] The shooting distance calculation unit 33 calculates the shooting distance from the camera 21 to the material pile 4 using the captured image. Here, it is preferable that the camera 21 is a compound camera (for example, a stereo camera) composed of multiple cameras. When the camera 21 is a compound camera, distance information can be obtained from multiple images captured at the same time using trigonometry, so the relative direction calculation unit 32 and the shooting distance calculation unit 33 can accurately calculate the relative shooting direction and shooting distance.

[0068] Even if camera 21 is a monocular camera, the relative direction calculation unit 32 and the shooting distance calculation unit 33 can calculate the shooting distance based on multiple images acquired by camera 21 by utilizing a trained model obtained through machine learning.

[0069] This first modified example provides the following advantages and benefits. In the measurement system 10A of the first modified example, the dump-up device 1 and the construction machine 2 require fewer sensors than the first embodiment, thus increasing the flexibility of the configuration. In particular, the dump-up device 1 does not require the first GNSS sensor 11 and the first geomagnetic sensor 12, so any dump-up device 1 can be used. Furthermore, the construction machine 2 only needs to be equipped with a camera 21 and an attitude angle sensor 24, and other sensors can be omitted, thus simplifying the configuration.

[0070] (Second variation) In the first modified example, the measurement unit 3 performed measurements based on the binarized image, relative shooting direction, and shooting distance, but other configurations are also possible. Below, a second modified example will be described in which the measurement unit 3 does not include a shooting distance calculation unit 33, and the measurement unit 3 performs measurement processing based on the binarized image and relative shooting direction.

[0071] Figure 11 is a block diagram showing the processing of the second modified measurement system 10B. Compared to the first modified measurement system 10A shown in Figure 10, the measurement system 10B shown in Figure 11 omits the shooting distance calculation unit 33 of the measurement unit 3. The shape and volume measurement unit 34 of the measurement unit 3 measures the shape and volume of the material pile 4 based on the binarized image in which the region of the material pile 4 obtained by the region identification unit 31 is identified, and the relative shooting direction acquired by the relative direction calculation unit 32.

[0072] In the measurement process of the material pile 4, the shooting distance contributes relatively more to volume measurement and less to shape measurement. Therefore, the shape-volume measurement unit 34 can measure the shape of the material pile 4 without using the shooting distance. Furthermore, depending on the type of material, the shape of the material pile 4 may differ according to its volume. Therefore, the shape-volume measurement unit 34 can measure the volume of the material pile 4 from the differences in its shape without using the measurement distance by using a machine learning model. In addition, the maximum load capacity of the dump-up device 1 used is often fixed. If the construction machine 2 is controlled to stop at a predetermined distance from the dump-up device 1 when unloading from the dump-up device 1, the shooting distance will be a predetermined distance. Therefore, even without a shooting distance calculation unit 33, the shape-volume measurement unit 34 can measure the volume of the material pile 4 with a certain degree of accuracy by using the maximum load capacity and predetermined distance in the measurement process.

[0073] This second modified version provides the following effects. In the measurement system 10B of the second modified version, the shooting distance calculation unit 33 is omitted in the measurement unit 3, which allows for faster measurement of the shape and volume of the material pile 4. Furthermore, when constructing the trained model and mathematical model of the shape-volume measurement unit 34, the shooting distance is not used, and the number of parameters used is reduced, thus shortening the time required to construct the shape-volume measurement unit 34.

[0074] (Second Embodiment) In the first embodiment, the measurement unit 3 performed measurements using images captured by the camera 21, but other configurations are also possible. Below, a second embodiment will be described in which the measurement unit 3 performs measurement processing using point cloud coordinate data obtained using a 3D scanner.

[0075] Figure 12 is a block diagram showing the processing of the measurement system 20 of the second embodiment. Compared to the measurement system 10 of the first embodiment shown in Figure 3, the measurement system 20 shown in Figure 12 has a 3D scanner 26 instead of a camera 21 in the construction machine 2.

[0076] The 3D scanner 26 is, for example, a LiDAR (Light Detection and Ranging) scanner. The 3D scanner 26 comprises an emitter that emits laser light while scanning, and a receiver that receives the reflected light of the laser light emitted from the emitter. Since the emitter emits laser light over a wide area while scanning, it is determined that no object exists at coordinates where reflected light could not be received.

[0077] Therefore, the 3D scanner 26 calculates the distance to the object in the direction in which the reflected light was received using the difference between the laser light irradiation time and the reflected light reception time. By performing this process over the entire laser light irradiation area, the 3D scanner 26 generates point cloud coordinate data indicating the presence or absence of an object.

[0078] In the measurement unit 3, the region identification unit 31 performs segmentation processing on the point cloud coordinate data to remove objects other than the material pile 4 and noise, thereby creating point cloud coordinate data that indicates the presence or absence of the material pile 4. In the first embodiment, the region identification unit 31 performed binarization processing, but since the presence or absence of objects is already displayed in binarization in such point cloud coordinate data, the region identification unit 31 in the second embodiment does not need to perform binarization processing.

[0079] In the second embodiment, the laser emission unit of the 3D scanner 26 irradiates while scanning with laser light, so unlike the shooting direction B in the first embodiment, the laser light emission direction is not unidirectional. However, given the characteristics of the material pile 4 shape as shown in Figure 1, it is ideal for the 3D scanner 26 to irradiate laser light within a predetermined range centered perpendicular to the device direction A. Therefore, by obtaining the difference between the actual irradiation area of ​​the laser light of the 3D scanner 26 and the ideal irradiation area, the measurement unit 3 can improve the measurement accuracy of the shape and volume of the material pile 4 by taking this difference into consideration.

[0080] Here, if we consider a specific irradiation direction within the irradiation area as the reference irradiation direction (for example, the exit direction when scanning stops or the scanning center direction), the above-mentioned differences can be taken into account by using the relative irradiation direction, which indicates the reference irradiation direction with respect to the device direction A, in the measurement processing of the measurement unit 3. Since the reference irradiation direction is a specific irradiation direction within the irradiation area, the relative direction calculation unit 32 can calculate the relative irradiation direction from the orientation of the dump-up device 1, the orientation and attitude angle of the construction machine 2, similar to the relative imaging direction in the first embodiment.

[0081] The shape and volume measurement unit 34 then uses a trained model or mathematical model to measure the shape and volume of the material pile 4 based on point cloud coordinate data that identifies the region of the material pile 4, the relative irradiation direction, and the shooting distance. The second geomagnetic sensor 23 and the attitude angle sensor 24 are assumed to be integrally configured as the irradiation direction acquisition unit 27.

[0082] Furthermore, the point cloud coordinate data obtained by the region identification unit 31, which identifies the region of the material pile 4, includes the distance from the 3D scanner 26 to any point on the surface of the material pile 4. Therefore, the shooting distance calculation unit 33 may be omitted in the measurement unit 3. In that case, the shape volume measurement unit 34 can perform measurement processing based on the point cloud coordinate data identifying the region of the material pile 4 and the relative shooting direction.

[0083] According to the second embodiment described above, the following effects and advantages are achieved.

[0084] In the measurement system 20 of the second embodiment, the measurement unit 3 measures the shape and volume of the material pile 4 using point cloud coordinate data instead of captured images. The measurement unit 3 (region identification unit 31) identifies the region of the material pile 4 within the point cloud coordinate data acquired by the 3D scanner 26 installed on the construction machine 2. The measurement unit 3 (relative direction calculation unit 32) determines the relative irradiation direction, which is the irradiation direction of the 3D scanner 26 with respect to the device direction A of the dump-up device 1. Then, the measurement unit 3 (shape volume measurement unit 34) measures at least one of the shape and volume of the material pile 4 based on the identified region of the material pile 4 and the relative irradiation direction. In this way, the information necessary for measuring the shape and volume of the material pile 4 can be acquired in a simple manner, so the shape and volume of the material pile 4 can be measured easily.

[0085] Furthermore, while the captured images used in the first embodiment are two-dimensional data, the point cloud coordinate data used in the second embodiment is three-dimensional data. The point cloud coordinate data includes distance information from the 3D scanner 26 to any point on the material pile 4 that is irradiated by laser light. Therefore, using point cloud coordinate data allows for more accurate measurement results regarding the shape and volume of the material pile 4 than using captured images. In addition, the acquisition time of captured images from the camera 21 is often shorter than the acquisition time of point cloud coordinate data from the 3D scanner 26. Therefore, by using captured images, the measurement time can be shortened compared to using point cloud coordinate data.

[0086] (Third variation) In the second embodiment, an example was described in which the construction machine 2 has a 3D scanner 26 instead of a camera 21, but other embodiments are also possible. Below, a third modified example in which the construction machine 2 has both a camera 21 and a 3D scanner 26 will be described.

[0087] Figure 13 is a block diagram showing the processing of the third modified measurement system 20A. Compared to the measurement system 20 of the second embodiment shown in Figure 12, the measurement system 20A shown in Figure 13 omits the first GNSS sensor 11 and the first geomagnetic sensor 12 of the dump-up device 1, and the second GNSS sensor 22 and the second geomagnetic sensor 23 of the construction machine 2, and adds a camera 21. Furthermore, compared to the first modified measurement system 10A, this configuration adds a 3D scanner 26.

[0088] The region identification unit 31 receives captured images acquired by the camera 21 and point cloud coordinate data acquired by the 3D scanner 26. By using the captured images as an auxiliary tool in the segmentation process of the point cloud coordinate data, the region identification unit 31 can remove objects other than the material pile 4 and noise with high accuracy.

[0089] The relative direction calculation unit 32 similarly receives captured images and point cloud coordinate data. As described in the first modified example, the relative direction calculation unit 32 can determine the direction A of the dump-up device 1 by combining captured images of the dump-up device 1 before and after unloading, but the accuracy of calculating the direction A of the device can be improved by using point cloud coordinate data as an auxiliary.

[0090] Similarly, when the shooting distance calculation unit 33 receives the captured image and point cloud coordinate data, it receives the distance from the camera 21 or 3D scanner 26 to the shooting reference point (for example, the highest point) of the material pile 4 as the shooting distance. Here, the shooting distance calculation unit 33 may directly determine the distance to the shooting reference point of the material pile 4 using the point cloud coordinate data.

[0091] The shape and volume measurement unit 34 then uses a trained model or mathematical model to measure the shape and volume of the material pile 4 based on point cloud coordinate data in which the region of the material pile 4 is identified, the relative shooting direction, and the shooting distance.

[0092] In the second modification described above, the region identification unit 31 performs a predetermined process on the point cloud coordinate data to create point cloud coordinate data in which the material pile 4 is identified, but other configurations are also possible. The region identification unit 31 may perform segmentation processing and binarization processing on the captured image, and use the point cloud coordinate data as an auxiliary in these processes to generate a binarized image in which the material pile 4 is identified. In such a case, the shape volume measurement unit 34 will use the binarized image instead of point cloud coordinate data, and can therefore use a trained model or mathematical model similar to that in the first embodiment.

[0093] This third modified version provides the following effects. In the measurement system 20A of the third modified version, the measurement unit 3 performs measurements using the captured image acquired by the camera 21 and the point cloud coordinate data acquired by the 3D scanner 26. By using both the captured image and the point cloud coordinate data, the amount of input information to the measurement unit 3 increases, thereby improving the accuracy of the measurement results. In particular, in the processing of the region identification unit 31, using both together makes it possible to efficiently remove objects other than the material pile 4 and noise.

[0094] Although embodiments and modifications of the present invention have been described above, these embodiments and modifications only represent a part of the application of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the embodiments and modifications described above. For example, in the third modification, when the camera 21 and the 3D scanner 26 are used in combination, the measurement unit 3 can perform measurement processing by arbitrarily combining inputs from sensors provided on the dump-up device 1 and the construction machine 2. [Explanation of Symbols]

[0095] 10, 10A, 10B, 20, 20A... Measurement System 1. Dump-up device 2. Construction machinery 3. Measurement section 4 ···Materials mountain 11 ···First GNSS sensor 12 ···First Geomagnetic Sensor 21 ···Camera 22 ···Second GNSS sensor 23 ···Second Geomagnetic Sensor 24 ···Attitude angle sensor 25 ···Shooting direction acquisition unit 26 ···3D scanner 27...Irradiation direction acquisition unit 31...Area identification part 32 ···Relative Direction Calculation Unit 33 ···Shooting distance calculation unit 34 ···Shape and volume measurement unit

Claims

1. A measuring system for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device fall from the transport device and are unloaded onto a surface, An image capture unit is provided on a movable body that can move on the aforementioned surface, and captures an image of the material pile by photographing it, A shooting direction acquisition unit that acquires the shooting direction of the image acquisition unit, A transport device direction acquisition unit that acquires the direction of the transport device when unloading the material, The system includes a measuring unit that measures at least one of the shape and volume of the material pile in the captured image acquired by the image acquisition unit, The aforementioned measuring unit is The region of the material pile in the captured image acquired by the image acquisition unit is identified, From the shooting direction acquired by the shooting direction acquisition unit and the device direction acquired by the transport device direction acquisition unit, the relative shooting direction, which is the shooting direction with respect to the device direction, is determined. A material pile measurement system for measuring the shape and volume of a material pile based on the region of the material pile and the relative imaging direction.

2. The material pile measurement system according to claim 1, wherein the measurement unit stores the results of machine learning performed in advance, and measures at least one of the shape and volume of the material pile based on the learning results based on the region of the material pile and the relative shooting direction.

3. The aforementioned measuring unit is The shooting distance between the image capture unit and the material pile is determined. A material pile measurement system according to claim 1, comprising measuring at least one of the shape and volume of the material pile based on the region of the material pile, the relative shooting direction, and the shooting distance.

4. The aforementioned measuring unit is The material pile measurement system according to claim 3, wherein the shooting distance is determined based on the position of the image shooting unit and the unloading position where the transport device unloaded the material.

5. The aforementioned measuring unit is The material pile measurement system according to claim 3, wherein the shooting distance is determined based on an image taken by the image capturing unit of the transport device unloading the material.

6. The material pile measurement system according to claim 1, wherein the transport device is a dump-up device and the moving body is a construction machine.

7. A measuring system for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device fall from the transport device and are unloaded onto a surface, A 3D scanner is provided on a movable body that can move on the aforementioned surface, and irradiates the material pile with laser light to acquire point cloud coordinate data indicating the presence or absence of an object. The irradiation direction acquisition unit acquires the irradiation direction of the 3D scanner, A transport device direction acquisition unit that acquires the direction of the transport device when unloading the material, The system includes a measuring unit that measures at least one of the shape and volume of the material pile in the point cloud coordinate data acquired by the 3D scanner, The aforementioned measuring unit is The region of the material peak is identified within the point cloud coordinate data acquired by the 3D scanner. From the irradiation direction acquired by the irradiation direction acquisition unit and the device direction acquired by the transport device direction acquisition unit, the relative irradiation direction, which is the irradiation direction with respect to the device direction, is determined. A material pile measurement system for measuring at least one of the shape and volume of the material pile based on the region of the material pile and the relative irradiation direction.

8. A measurement method for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device fall from the transport device and are unloaded onto a surface, An image capture unit is provided on a movable body that can move on the aforementioned surface to photograph the material pile and acquire an image. The shooting direction of the image capture unit is obtained, The direction of the transport device when unloading the material is obtained, The region of the material pile in the captured image acquired by the image acquisition unit is identified, From the acquired shooting direction and the acquired device direction, the relative shooting direction, which is the shooting direction with respect to the device direction, is determined. A method for measuring a material pile, comprising measuring at least one of the shape and volume of the material pile based on the region of the material pile and the relative imaging direction.

9. A measurement method for measuring at least one of the shape and volume of a pile of materials formed when materials loaded onto a transport device fall from the transport device and are unloaded onto a surface, A 3D scanner mounted on a movable body that can move on the aforementioned surface irradiates the material pile with laser light to acquire point cloud coordinate data indicating the presence or absence of an object. The irradiation direction of the 3D scanner is acquired, The direction of the transport device when unloading the material is obtained, The region of the material peak is identified within the point cloud coordinate data acquired by the 3D scanner. From the acquired irradiation direction and the acquired device direction, the relative irradiation direction, which is the irradiation direction with respect to the device direction, is determined. A method for measuring a material pile, comprising measuring at least one of the shape and volume of the material pile based on the region of the material pile and the relative irradiation direction.

Images