Particle size measurement device and particle size measurement program

Abstract

To accurately measure a particle size of a granular material.SOLUTION: A depth image acquisition section 26 acquires a depth image which expresses a front surface of a loaded matter of a granular material and where each one of pixels is a depth value. Then, a detection section 30 defines each one of partial images of the depth image as an input so as to detect the granular material from the depth image with the use of a pre-learnt neural network model for determining whether the partial images express the granular material. A measurement section 34 measures a particle size of the granular material from the detection result of the granular material.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a particle size measuring device and a particle size measuring program. [Background technology]

[0002] Conventionally, a particle size detection algorithm that applies two-dimensional image processing developed for the purpose of particle size measurement has been known, which targets three-dimensional measurement results obtained by three-dimensional measurement and the 256-level depth image generated thereby (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6281014 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the particle size measurement algorithms of the conventional technology have problems such as being unable to detect overlapping pebbles, being unable to detect foreign objects other than pebbles, and being unable to perform particle size detection processing unless there are a certain number of measurement points in the three-dimensional measurement.

[0005] In consideration of the above, an object of the present invention is to enable accurate measurement of the particle size of a granular material. [Means for solving the problem]

[0006] The particle size measuring device of the present invention is a particle size measuring device that measures the particle size of a granular material from an accumulation of the granular material, and includes an acquisition unit that acquires a depth image, which is an image representing the surface of the accumulation of the granular material, in which each pixel has a depth value, a detection unit that receives each partial image of the depth image as input and detects the granular material from the depth image using a pre-trained neural network model that determines whether the partial image represents a granular material, and a measurement unit that measures the particle size of the granular material from the detection result of the granular material.

[0007] According to the particle size measuring device of the present invention, an acquisition unit acquires a depth image, which is an image representing the surface of the accumulation of the granular material, and each pixel is a depth value. A detection unit receives each partial image of the depth image as an input, and detects the granular material from the depth image using a pre-trained neural network model that determines whether the partial image represents a granular material. A measurement unit measures the particle size of the granular material from the detection result of the granular material.

[0008] In this way, by using a neural network model to detect granular material from a depth image representing the surface of the accumulation of granular material and measuring the particle size of the granular material, the particle size of the granular material can be measured with high accuracy.

[0009] The particle size measuring device of the present invention further includes a point cloud acquisition unit that acquires a three-dimensional point cloud representing the surface of the accumulation of granular material, and a pre-processing unit that corrects the acquired three-dimensional point cloud so that a plane that approximates the surface represented by the three-dimensional point cloud corresponds to the measurement surface of the three-dimensional point cloud, and the acquisition unit acquires the depth image from the corrected three-dimensional point cloud.

[0010] The granular material according to the present invention is gravel.

[0011] The particle size measuring device of the present invention further includes a shape estimation unit that estimates the shape of the granular material from the detection result of the granular material, and the measuring unit measures the particle size of the granular material from the estimated shape of the granular material.

[0012] The measuring unit according to the present invention measures the particle size of the granular material based on the area, minor axis, or major axis calculated from the estimated shape of the granular material.

[0013] The particle size measurement program of the present invention is a particle size measurement program for measuring the particle size of a granular material from an accumulation of the granular material, and is a program for causing a computer to function as an acquisition unit that acquires a depth image, which is an image representing the surface of the accumulation of the granular material, in which each pixel has a depth value, a detection unit that detects the granular material from the depth image using a pre-trained neural network model that receives as input each partial image of the depth image and determines whether the partial image represents a granular material, and a measurement unit that measures the particle size of the granular material from the detection result of the granular material. Effect of the Invention

[0014] As described above, the particle size measuring device and particle size measuring program of the present invention have the effect of detecting granular material using a neural network model from a depth image representing the surface of the accumulation of granular material, and measuring the particle size of the granular material, thereby enabling the particle size of the granular material to be measured with high accuracy. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a particle size measuring device and a learning device according to an embodiment of the present invention. [Diagram 2] 1 is a functional block diagram showing a particle size measuring device according to an embodiment of the present invention. [Diagram 3] 1A and 1B are diagrams illustrating an example in which an angle misalignment occurs between a camera sensor plane and a measurement target surface. [Figure 4] FIG. 13 is a diagram showing an example in which the distance resolution of each gravel is different. [Diagram 5]FIG. 1A is a diagram showing an example in which the angle between the camera sensor plane and a three-dimensional plane that approximates the point cloud obtained as a three-dimensional measurement result is misaligned, and FIG. 1B is a diagram showing an example in which a three-dimensional point cloud is transformed so that the three-dimensional plane that approximates the point cloud is parallel to the camera sensor plane. [Figure 6] This figure shows an example of a converted 3D point cloud and an example in which the distance resolution of each pebble is adjusted to an optimal one. [Figure 7] FIG. 2A is a diagram showing an example of a measurement result of a three-dimensional point cloud before preprocessing, and FIG. 2B is a diagram showing an example of a measurement result of a three-dimensional point cloud after preprocessing. [Figure 8] FIG. 13 is a diagram for explaining a method for calculating the particle size of a granular material. [Figure 9] FIG. 13 is a diagram for explaining a method for calculating the particle size of a granular material. [Figure 10] 1 is a functional block diagram showing a learning device according to an embodiment of the present invention; [Figure 11] 4 is a flowchart showing the contents of a particle size measurement processing routine of the particle size measuring device according to the embodiment of the present invention. [Figure 12] FIG. 1 shows experimental results using a prior art approach. [Figure 13] FIG. 11 is a diagram showing experimental results using the method of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0017] <Principle of the embodiment of the present invention> In the embodiment of the present invention, a particle size measurement process is performed using an object detection process by a neural network.

[0018] To detect gravel, we apply neural network-based object detection technology (e.g., YOLO (You Only Look Once), Fast R-CNN (Regions with Convolutional Neural Networks), SSD (Single Shot Multibox Detector)) to detect rectangular areas representing gravel in the depth image, and then apply two-dimensional image processing to estimate the gravel shape in more detail from within the detected rectangular areas.

[0019] The data used for object detection by the neural network is a two-dimensional depth image after three-dimensional measurement, which makes it possible to ignore information unrelated to the pebble shape, such as the irregular patterns on the pebble surface, and creates a model that takes into account only the pebble shape, making it possible to achieve high object detection accuracy even with limited learning data. Limited data is used for learning data, with no restrictions on the amount of gravel.

[0020] <Configuration of a particle size measuring device according to an embodiment of the present invention> As shown in FIG. 1, a particle size measuring device 100 according to an embodiment of the present invention includes a CPU (Central Processing Unit) 12, a graphics card 13, a GPU (Graphics Processing Unit) 14, a RAM (Random Access Memory) 16, a HDD (Hard Disk Drive) 18, a communication interface (I / F) 21, and a bus 23 for interconnecting these.

[0021] The CPU 12 and the GPU 14 execute various programs. The RAM 16 is used as a work area when the CPU 12 executes the various programs. The HDD 18 serving as a recording medium stores various programs and various data, including a program for executing a particle size measurement processing routine described later.

[0022] The particle size measuring device 100 in this embodiment is represented by functional blocks along with a program for executing a particle size measuring processing routine, as shown in Fig. 2. The particle size measuring device 100 includes an input unit 10, a calculation unit 20, and an output unit 50.

[0023] The input unit 10 receives as input data for performing three-dimensional measurement of an accumulation of granular material.

[0024] Specifically, the granular material is gravel, for example, excavation waste (tunnel excavation waste). The location of the accumulation of the granular material is not important, but as an example, the loading platform of a dump truck (transport vehicle) for transporting the material to the embankment construction area is assumed.

[0025] In addition, in this embodiment, as a means for realizing three-dimensional measurement, a spatial coding method is adopted, which uses one projector (compact projector) and one CCD camera, as shown in Fig. 3. This spatial coding method has the advantage that it is resistant to changes in the environmental light at the measurement site, and the entire system can be kept low-cost and compact.

[0026] The projector and the CCD camera are each connected via a cable to the particle size measuring device 100. The projector and the CCD camera are controlled by the particle size measuring device 100, making it possible to perform three-dimensional measurement of the surface of the object to be photographed.

[0027] The pattern projection by the projector and the image capture operation by the CCD camera are synchronously controlled by the particle size measuring device 100, and the image capturing time can be kept short (several seconds to about 5 seconds).

[0028] The calculation unit 20 includes a point cloud acquisition unit 22, a preprocessing unit 24, a depth image acquisition unit 26, a trained model storage unit 28, a detection unit 30, a shape estimation unit 32, and a measurement unit 34.

[0029] The point cloud acquisition unit 22 acquires, from the data input to the input unit 10, a three-dimensional point cloud representing the surface of the accumulation of granular material.

[0030] Specifically, based on data from a projector and a CCD camera, a spatial coding method is used to perform three-dimensional measurements (calculation of three-dimensional coordinate values) of each pixel in the camera coordinate system, and a three-dimensional point cloud is obtained.

[0031] More specifically, the spatial coding method is a method in which a projector projects multiple slits onto the measurement target, a CCD camera simultaneously captures the target, and these captured images are used to assign codes to the captured space, which are then used to perform three-dimensional measurement. This method is a generally established three-dimensional measurement method, so an explanation of its principles will be omitted, but the method combines each sample of the pattern projected from the projector during shooting, assigns a spatial code, and measures the three-dimensional position of each pixel on the captured image by triangulation between the projector coordinate system, camera coordinate system, and the measurement target.

[0032] The preprocessing unit 24 corrects the acquired three-dimensional point cloud so that a plane that approximates the surface represented by the three-dimensional point cloud corresponds to the measurement surface of the three-dimensional point cloud.

[0033] Here, the principle of the pre-processing unit 24 will be described. In the three-dimensional measurement process in the particle size measurement process, an angle deviation occurs between the measurement target plane of the accumulation of granular material and the sensor plane of the CCD camera due to the hardware installation deviation of the particle size measurement device 100 and the condition of the measurement target surface. For example, as shown in FIG. 3, an angle deviation occurs between the sensor plane of the CCD camera and the measurement target surface. Note that this angle deviation is originally a three-dimensional angle deviation, but is displayed two-dimensionally in FIG. 3. This angle difference occurring as a result of the three-dimensional measurement causes a decrease in the distance resolution of each measured pebble when normalization is performed to 256 gradations between the maximum and minimum values ​​of the measurement distance when creating a depth image performed in the subsequent particle size measurement process (FIG. 4). FIG. 4 shows an example in which the distance resolution of each pebble present on the upper dashed line is different.

[0034] This problem causes a decrease in accuracy in both detection rate and detected particle size when applying particle size measurement algorithms that are performed as two-dimensional image processing.

[0035] In order to solve the above problems, in this embodiment, a pre-processing having an automatic tilt correction processing function is performed. In this pre-processing, first, a three-dimensional plane that approximates the point cloud obtained as the three-dimensional measurement result is generated (see FIG. 5(A)). FIG. 5(A) shows an example in which there is an angular misalignment between the camera sensor plane and the three-dimensional plane that approximates the point cloud obtained as the three-dimensional measurement result. Note that this angular misalignment is actually a three-dimensional angular misalignment, but is shown in two dimensions in FIG. 5.

[0036] Next, a coordinate transformation matrix is ​​calculated so that this estimated three-dimensional plane is parallel to the CCD camera sensor plane, and this coordinate transformation matrix is ​​applied to all measurement points (see Fig. 5(B)). Fig. 5(B) shows an example in which a three-dimensional point cloud is transformed so that a three-dimensional plane that approximates the point cloud is parallel to the CCD camera sensor plane.

[0037] This allows the angle between the measurement target surface and the CCD camera sensor surface to be optimally adjusted, making it possible to convert to a depth image while maintaining optimal distance resolution when generating a 256-level depth image (see Figure 6). The top of Figure 6 shows an example of a converted 3D point cloud, and the bottom of Figure 6 shows an example where the distance resolution of each of the pebbles present on the upper dashed line has been optimally adjusted.

[0038] Fig. 7(A) shows an example of a depth image obtained without performing preprocessing, and Fig. 7(B) shows an example of a depth image obtained after performing preprocessing. In this way, it can be confirmed that the depth is properly displayed in the depth image by performing preprocessing.

[0039] The depth image acquisition unit 26 acquires a depth image from the three-dimensional point cloud corrected by the pre-processing unit 24. Specifically, the three-dimensional point cloud corrected by the pre-processing unit 24 is converted into a depth image, which is an image representing the surface of the accumulation of granular material, and in which each pixel is a depth value.

[0040] The trained model storage unit 28 stores a trained model, which is a pre-trained neural network model that receives each partial image of the depth image as an input and determines whether the partial image represents a granular material or not. This trained model has been trained in advance by a training device 150, which will be described later.

[0041] The detection unit 30 uses a pre-trained neural network model to detect each of the granular materials from the depth image acquired by the depth image acquisition unit 26. For example, while scanning a rectangular region that is a bounding box on the depth image, a partial image of the rectangular region is input to the neural network model, an output of the neural network is acquired, and each of the rectangular regions that are indicated as being granular materials by the output of the neural network is detected as a rectangular region representing the granular material.

[0042] Next, the principle of measuring the particle size of a granular material will be described.

[0043] In conventional techniques, the particle size of granular material is measured by approximating the size of the granular material with a circle, but in the present embodiment, a curve is used to more accurately measure the shape of the granular material within a detected rectangular area.

[0044] Here, the concept of particle size will be explained.

[0045] Even if the particle size distribution obtained from the grain size test using a standard sieve is almost the same, (1) the maximum dry density when compacted differs depending on the soil particle shape, and (2) the soil particle size estimated by the grain size measurement method for granular materials also differs depending on the particle shape.

[0046] Regarding (2) above, if the shape of a soil particle, such as a rugby ball with a jagged surface, could be accurately measured in three dimensions, it would be possible to determine which part of the diameter should be judged as the engineering particle diameter (i.e., the particle size measured by a particle size test). This would allow for a similar approach to the conventional sieve analysis test, making it possible to determine particle size more accurately.

[0047] Therefore, in this embodiment, the shape estimation unit 32 estimates the shape of each of the granular materials from the detection result of the granular materials. Specifically, the shape estimation unit 32 estimates the shape of the granular material within the detected rectangular region by using a curve.

[0048] The measuring unit 34 measures the particle size of each of the granular materials from the estimated shape of the granular material, and outputs the measured particle size by the output unit 50.

[0049] Specifically, as shown in FIG. 8, the shape of the granular material estimated using a curve is approximated to a circle of the same area, and the particle size of the granular material is calculated using the following formula. d=2×√(T / π) Here, d is the diameter (particle size) and T is the area.

[0050] Alternatively, as shown in Figure 9, the short diameter a and long diameter b are calculated as the particle size of the granular material from the shape of the granular material estimated using the curve, and at least one of the short diameter a and long diameter b is determined to be the particle size of the granular material.

[0051] <Configuration of the learning device according to the embodiment of the present invention> As shown in Figure 1 above, the learning device 150 of an embodiment of the present invention, like the particle size measuring device 100, is equipped with a CPU 12, a graphics card 13, a GPU 14, a RAM 16, a HDD 18, a communication interface 21, and a bus 23 for connecting these to each other.

[0052] As shown in FIG. 10, a learning device 150 in this embodiment includes an input unit 110 and a calculation unit 120.

[0053] The input unit 110 receives as input a plurality of learning data including a partial image of a depth image obtained by the same method as the particle size measurement device 100 and a label indicating whether the partial image is a granular material or not.

[0054] The calculation unit 120 includes a learning data storage unit 122 , a learning unit 124 , and a model storage unit 126 .

[0055] The learning data storage unit 122 stores a plurality of input learning data.

[0056] The learning unit 124 learns a neural network model that receives a partial image of a depth image as input based on a plurality of learning data and determines whether the partial image represents a granular material or not, and creates a learned model.

[0057] Specifically, when a partial image of the depth image of the learning data is input to the neural network model, the parameters of the neural network model are learned so that the output of the neural network model is the same as the label.

[0058] The model storage unit 126 stores a trained model, which is a neural network model trained by the training unit 124.

[0059] <Learning device operation> Next, the operation of learning device 150 according to the embodiment of the present invention will be described.

[0060] First, a 3D point cloud representing the surface of the granular material accumulation is obtained from data of a projector and a CCD camera for performing 3D measurement of the granular material accumulation, and the obtained 3D point cloud is converted into a depth image after correcting the plane approximating the surface represented by the 3D point cloud to correspond to the measurement surface of the 3D point cloud. Then, a label indicating whether or not the partial image represents a granular material is assigned to each partial image of the depth image in advance, and multiple learning data are generated.

[0061] Then, when the input unit 110 of the learning device 150 accepts a plurality of pieces of learning data as input, the input plurality of pieces of learning data are stored in the learning data storage unit 122. Then, the learning unit 124 learns parameters of the neural network model based on the plurality of pieces of learning data, and stores in the model storage unit 126 a trained model, which is the trained neural network model.

[0062] <Operation of particle size measuring device> Next, the operation of the particle size measuring device 100 according to the embodiment of the present invention will be described.

[0063] First, the trained model trained by the learning device 150 is stored in the trained model storage unit 28 of the particle size measurement device 100.

[0064] Then, when the input unit 10 receives as input data from a projector and a CCD camera for performing three-dimensional measurement of an accumulation of granular material, the particle size measuring device 100 executes a particle size measuring processing routine shown in FIG.

[0065] First, in step S100, the point cloud acquisition unit 22 acquires, from the data input to the input unit 10, a three-dimensional point cloud representing the surface of the accumulation of granular material.

[0066] Then, in step S102, the preprocessing unit 24 performs preprocessing to correct the acquired three-dimensional point cloud so that a plane approximating a surface represented by the three-dimensional point cloud corresponds to a measurement surface of the three-dimensional point cloud.

[0067] In step S104, the depth image acquisition unit 26 converts the three-dimensional point group corrected by the pre-processing unit 24 into a depth image.

[0068] In step S106, the detection unit 30 detects each of the granular materials from the depth image acquired by the depth image acquisition unit 26 using a pre-trained neural network model.

[0069] In step S108, the shape estimation unit 32 estimates the shape of each of the granular materials from the detection results of the granular materials.

[0070] In step S110, the measuring unit 34 measures the particle size of each of the granular materials from the estimated shape of the granular material.

[0071] In step S112, the measurement result of the particle size of the granular material is output by the output unit 50, and the particle size measurement processing routine ends. For example, the output unit 50 outputs the particle size distribution in the accumulation of the granular material.

[0072] <Example> Using particle size measurement results (two-dimensional depth images) obtained on-site, a neural network model was trained and experiments on gravel detection accuracy were conducted. Figure 12 shows the results of gravel detection using the method described in Patent Document 1 above as a conventional technique. Figure 13 shows the results of gravel detection using the method of the above embodiment. Figure 12 shows an example in which 115 gravel were detected, of which 10 had large errors and 20 were false positives. On the other hand, Figure 13 shows gravel being detected with good accuracy.

[0073] Although the number of two-dimensional depth images used as learning data was limited, and the learning time was also limited, it was confirmed that the method of the above embodiment was sufficient to detect gravel.

[0074] As described above, with the particle size measuring device of an embodiment of the present invention, the particle size of the granular material can be accurately measured by detecting the granular material from a depth image representing the surface of an accumulation of granular material using a neural network model and measuring the particle size of the granular material.

[0075] In addition, the detection of overlapping granular materials (e.g. gravel), adjacent granular materials, and granular materials with rounded and unclear boundaries, which were previously detected incorrectly, is improved, and it is expected that the accuracy of particle size measurement will be improved. In other words, it will be possible to measure the particle size of granular materials more accurately than with conventional methods.

[0076] The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0077] For example, although the embodiment has been described with respect to the case where the particle size of gravel is measured as a granular material, the present invention is not limited to this. The particle size of granular materials other than gravel may be measured. For example, the particle size of stones having a particle size of 75 mm to 300 mm or more may be measured.

[0078] In the above embodiment, the case where the shape of the granular material is estimated and then the particle size of the granular material is measured is described as an example, but the present invention is not limited to this. The particle size of the granular material may be measured without estimating the shape of the granular material from the detection result of the granular material. For example, as in the above Patent Document 1, the particle size of the granular material may be measured by approximating the detected granular material with a circle.

[0079] Also, the 3D point cloud may be converted into a depth image without pre-processing.

[0080] Furthermore, the program of the present invention may be provided in a form stored on a storage medium. [Explanation of symbols]

[0081] 10, 110 Input section 20, 120 calculation section 22 Point cloud acquisition section 24 Pretreatment section 26 Image acquisition section 28 Trained model memory 30 Detection unit 32 Shape estimation section 34 Measurement section 50 Output section 100 Particle size measuring device 122 Learning data storage unit 124 Learning Department 126 Model Memory Section 150 Learning Device

Claims

1. A particle size measuring device for measuring a particle size of a granular material from an accumulation of the granular material, comprising: a point cloud acquisition unit for acquiring a three-dimensional point cloud representing a surface of the accumulation of granular material; a pre-processing unit that corrects the acquired three-dimensional point cloud so that a plane that approximates a surface represented by the three-dimensional point cloud corresponds to a measurement surface of the three-dimensional point cloud; an acquisition unit that acquires a depth image from the corrected three-dimensional point cloud, the depth image being an image representing a surface of the accumulation of granular material, each pixel being a depth value; a detection unit that detects the granular material from the depth image using a pre-trained neural network model that receives each partial image of the depth image as an input and determines whether the partial image represents a granular material; A measuring unit that measures a particle size of the granular material based on a detection result of the granular material; A particle size measuring device comprising:

2. 2. The particle size measuring apparatus according to claim 1, wherein the granular material is gravel.

3. The method further includes a shape estimation unit that estimates a shape of the granular material from a detection result of the granular material, 3. The particle size measuring device according to claim 1, wherein the measuring unit measures a particle size of the granular material from the estimated shape of the granular material.

4. The particle size measuring device according to claim 3 , wherein the measuring unit measures a particle size of the granular material based on an area, a minor axis, or a major axis determined from the estimated shape of the granular material.

5. A particle size measurement program for measuring a particle size of a granular material from an accumulation of the granular material, Computer, a point cloud acquisition unit for acquiring a three-dimensional point cloud representing a surface of the accumulation of granular material; a pre-processing unit that corrects the acquired three-dimensional point cloud so that a plane that approximates a surface represented by the three-dimensional point cloud corresponds to a measurement surface of the three-dimensional point cloud; an acquisition unit for acquiring, from the corrected three-dimensional point cloud, a depth image representing a surface of the accumulation of granular material, each pixel being a depth value; a detection unit that detects the granular material from the depth image using a pre-trained neural network model that receives each sub-image of the depth image and determines whether the sub-image represents a granular material; and a measuring unit for measuring a particle size of the granular material based on a result of the detection of the granular material; A particle size measurement program to function as a.

Images