Sand and gravel grain shape analysis method based on depth estimation of monocular camera

Abstract

The present invention discloses a sand and gravel grain shape analysis method based on monocular camera depth estimation, comprising the following steps: step one, image acquisition; step two, monocular depth estimation; step three, instance segmentation; step four, calculation of three-dimensional coordinates; step five, grain shape characterization; step six, outputting results; compared with the existing method of manually evaluating sand and gravel grain shape, the present invention collects two-dimensional image information of sand and gravel aggregate piles through automated image acquisition equipment, analyzes and processes the two-dimensional image information using computer algorithm technology, and realizes depth estimation of the two-dimensional aggregate image in a non-contact, intelligent and green way, and characterizes the aggregate grain shape, thereby obtaining the distribution of the sand and gravel aggregate grain shape and realizing quality evaluation of the sand and gravel aggregate. The method eliminates the subjective influence brought by manual evaluation, reduces the labor intensity of evaluators, and helps mining enterprises improve work efficiency and enhance economic benefits.

Description

Technical Field

[0001] The present invention relates to the technical field of mining industry, and in particular to a sand and gravel grain shape analysis method based on monocular camera depth estimation. Background Art

[0002] Different particle shapes correspond to different qualities of sand and gravel aggregates, serving different construction uses and directly determining the economic value of the products. Many types of building materials, including concrete, asphalt, and mortar, use sand and gravel aggregates as primary components. In concrete manufacturing, using spherical sand and gravel aggregates can reduce production costs and increase the resistance of the concrete mixture. In building material production, materials made with lower-quality needle-shaped and flaky sand and gravel aggregates have a lower safety factor, while those made with more spherical sand and gravel aggregates have a higher safety factor. Therefore, the economic value of needle-shaped and flaky aggregates is far lower than that of spherical aggregates. Therefore, aggregate particle shape analysis and particle size distribution are crucial for both economic value and safety.

[0003] In the existing technology, domestic mines still mainly rely on manual evaluation for the evaluation of the particle shape of crushed sand and gravel aggregates. Although manual evaluation is technically very mature, the number of samples calculated each time is limited, the manual process is highly subjective, labor-intensive, and inefficient. Summary of the Invention

[0004] The purpose of the present invention is to provide a sand and gravel grain shape analysis method based on monocular camera depth estimation to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a sand and gravel grain shape analysis method based on monocular camera depth estimation, comprising the following steps: step 1, image acquisition; step 2, monocular depth estimation; step 3, instance segmentation; step 4, calculation of three-dimensional coordinates; step 5, grain shape characterization; step 6, output of results;

[0006] In the above step 1, an industrial camera is used to capture a two-dimensional image of the sand and gravel aggregate pile;

[0007] In the above step 2, a monocular depth estimation algorithm is used to process the two-dimensional sand and gravel aggregate pile image obtained in step 1, and a corresponding depth map is output, where each pixel value in the depth map is the distance from the camera to the sand and gravel aggregate pile;

[0008] In the above step 3, the instance segmentation algorithm based on Mask R-CNN is used to perform instance segmentation on the two-dimensional sand and gravel aggregate pile image obtained in step 1, and the segmented mask image is output;

[0009] In the above step 4, under the condition of knowing the camera intrinsic parameter matrix, according to the depth value and the mask map of each sand and gravel aggregate, the three-dimensional coordinates (in the camera coordinate system) of all pixels in each mask area can be calculated according to the coordinate system transformation principle;

[0010] In the above step 5, the 3D point cloud of each sand and gravel aggregate particle is obtained according to the 3D coordinates of all pixels in each mask area obtained in step 4, and the major axis L, median axis I and minor axis S of the sand and gravel aggregate particle are estimated based on this. The particle shape of the sand and gravel aggregate is characterized by three particle shape characterization methods: flattening, Wentworh flattening coefficient, and KrumBein sphericity, and the particle shape characterization value of each sand and gravel aggregate is calculated based on this.

[0011] In the above step 6, according to the grain shape characterization value of the sand and gravel aggregate obtained in step 5, the distribution of the grain shape of the sand and gravel aggregate in the original input two-dimensional image is statistically analyzed, and the analysis result is output.

[0012] Preferably, in step one, an industrial camera is vertically deployed above the sand and gravel aggregate conveyor belt through a metal frame. Two industrial lighting devices are also deployed on both sides of the industrial camera for lighting. At the same time, a dust removal fan is also fixed on one side of the camera for dust removal. The image information collected by the industrial camera is transmitted to an embedded computing platform and analyzed using sand and gravel aggregate particle shape analysis software.

[0013] Preferably, in the step 2, the specific algorithm model adopted is the ZoeDepth model pre-trained on 12 indoor and outdoor depth estimation datasets, and fine-tuned on the self-constructed sand and gravel depth dataset. The sand and gravel depth dataset is constructed using the Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms in COLMAP. By taking pictures from different angles of sand and gravel aggregates, the depth of the collected images is estimated using the above method to obtain a more accurate depth map, which is then used as a fine-tuning dataset for the ZoeDepth pre-training model.

[0014] Preferably, in the step 4, specifically: given the two-dimensional pixel coordinates (u, v) of the input two-dimensional image of the sand and gravel aggregate pile, the intrinsic parameter matrix K of the shooting camera, and the depth value Z of each pixel point of the two-dimensional image, c In the case of , the three-dimensional camera coordinates (X c ,Y c ,Z c ),Right now:

[0015]

[0016] Z c =Zc

[0017] Among them, f x , f y is the camera intrinsic focal length, (u0, v0) is the principal point coordinate in the camera intrinsic parameter, the principal point is the intersection of the optical axis and the imaging plane, and is the center point of the pixel coordinate when the camera is imaging.

[0018] Preferably, in the step five, specifically: after obtaining the three-dimensional coordinates of each pixel point of the sand and gravel aggregate image, extract the corresponding mask area from the original two-dimensional image according to the mask result obtained after instance segmentation, each mask area represents a segmented sand and gravel aggregate particle, that is, the pixel point set of each sand and gravel aggregate particle in the original two-dimensional image is obtained, and the coordinates of these pixel point sets are converted into three-dimensional coordinates to obtain a three-dimensional point cloud of each sand and gravel aggregate particle; after obtaining the three-dimensional point cloud of the sand and gravel aggregate particle, first center the point cloud, that is, place the sand and gravel aggregate particle in the center of the coordinate system, and then construct the covariance matrix of the point cloud and use principal component analysis (PCA) to estimate the major axis L, medial axis I, and minor axis S of the sand and gravel aggregate particle, and then estimate the particle shape of the sand and gravel aggregate particle.

[0019] Preferably, in step 5, the calculation method of the flatness is: The value range is (0, 1); the calculation method of Wentworh flatness coefficient is: The value range is (1, +∞); the calculation method of KrumBein sphericity is: The value range is (0, 1).

[0020] Preferably, in step five, for the flatness, the closer the particles are to flatness, the closer the flatness is to 0, and the closer the particles are to spherical, the closer the flatness is to 1; for the Wentworh flatness coefficient, the closer the particles are to flatness, the larger the flatness coefficient value is, and compared with sand and gravel aggregate particles of other shapes, the flatness coefficient of flaky sand and gravel aggregate is the highest; for the KrumBein sphericity, the sphericity of spherical particles such as blocky sand and gravel aggregate is closer to 1, and vice versa.

[0021] Compared with the existing technology, the beneficial effects of the present invention are: compared with the existing method of manually evaluating the grain shape of sand and gravel, the present invention collects two-dimensional image information of the sand and gravel aggregate pile through automated image acquisition equipment, and uses computer algorithm technology to analyze and process the two-dimensional image information, thereby realizing a non-contact, intelligent and green method to perform depth estimation on the two-dimensional aggregate image and characterize the aggregate grain shape, thereby obtaining the distribution of the sand and gravel aggregate grain shape and realizing the quality evaluation of the sand and gravel aggregate. This method eliminates the subjective influence brought by manual evaluation, reduces the labor intensity of evaluators, and helps mining companies improve work efficiency and enhance economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the steps of the present invention;

[0023] Figure 2 is a flow chart of the method of the present invention;

[0024] Figure 3 is a two-dimensional diagram of the sand and gravel aggregate of the present invention;

[0025] Figure 4 is the sand and gravel aggregate depth map of the present invention;

[0026] Figure 5 It is the sand and gravel aggregate segmentation diagram of the present invention;

[0027] Figure 6 This is a representation diagram of the particle shape of the sand and gravel aggregate of the present invention;

[0028] Figure 7 This is the interface diagram of the sand and gravel aggregate particle shape analysis software of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] See also Figure 1-7 The present invention provides an embodiment: a sand and gravel grain shape analysis method based on monocular camera depth estimation, comprising the following steps: step 1, image acquisition; step 2, monocular depth estimation; step 3, instance segmentation; step 4, calculation of three-dimensional coordinates; step 5, grain shape characterization; step 6, outputting results;

[0031] In step 1, an industrial camera is used to capture a two-dimensional image of the sand and gravel aggregate pile. The industrial camera is vertically positioned above the sand and gravel aggregate conveyor belt via a metal frame. Two industrial lighting devices are also deployed on both sides of the industrial camera for illumination. A dust removal fan is also fixed to one side of the camera for dust removal. The image information captured by the industrial camera is transmitted to an embedded computing platform and analyzed using sand and gravel aggregate grain shape analysis software.

[0032] In the above step 2, a monocular depth estimation algorithm is used to process the two-dimensional sand and gravel aggregate pile image obtained in step 1, and a corresponding depth map is output. The value of each pixel in the depth map is the distance from the camera to the sand and gravel aggregate pile. The specific algorithm model adopted is the ZoeDepth model pre-trained on 12 indoor and outdoor depth estimation datasets, and fine-tuned on the self-constructed sand and gravel depth dataset. The sand and gravel depth dataset is constructed using the Structure from Motion (SfM) and Multi-View Stereo Matching (MVS) algorithms in COLMAP. By taking pictures from different angles of the sand and gravel aggregate, the above method is used to perform depth estimation on the collected images to obtain a more accurate depth map, which is then used as a fine-tuning dataset for the ZoeDepth pre-training model.

[0033] In the above step 3, the instance segmentation algorithm based on Mask R-CNN is used to perform instance segmentation on the two-dimensional sand and gravel aggregate pile image obtained in step 1, and the segmented mask image is output;

[0034] In the above step 4, under the condition of knowing the camera intrinsic parameter matrix, according to the depth value and the mask map of each sand and gravel aggregate, the three-dimensional coordinates of all pixels in each mask area (in the camera coordinate system) can be calculated according to the coordinate system conversion principle. Specifically, given the two-dimensional pixel coordinates (u, v) of the input two-dimensional image of the sand and gravel aggregate pile, the intrinsic parameter matrix K of the shooting camera, and the depth value Z of each pixel in the two-dimensional image, c In the case of , the three-dimensional camera coordinates (X c ,Y c ,Z c ),Right now:

[0035]

[0036] Z c =Z c

[0037] Among them, f x , f yis the camera intrinsic focal length, (u0, v0) is the principal point coordinate in the camera intrinsic parameter, the principal point is the intersection of the optical axis and the imaging plane, and is the center point of the pixel coordinate when the camera is imaging;

[0038] In the above step 5, the 3D point cloud of each sand and gravel aggregate particle is obtained according to the 3D coordinates of all pixels in each mask area obtained in step 4, and the long axis L, median axis I and short axis S of the sand and gravel aggregate particle are estimated accordingly, and the grain shape of the sand and gravel aggregate is characterized by three grain shape characterization methods: flattening rate, Wentworh flattening coefficient and KrumBein sphericity. Based on this, the grain shape characterization value of each sand and gravel aggregate is calculated. Specifically, after obtaining the 3D coordinates of each pixel point of the sand and gravel aggregate image, the corresponding mask area is extracted from the original 2D image according to the mask result obtained after instance segmentation. Each mask area represents a segmented sand and gravel aggregate particle, that is, the pixel point set of each sand and gravel aggregate particle in the original 2D image is obtained, and the coordinates of these pixel point sets are converted into 3D coordinates to obtain the 3D point cloud of each sand and gravel aggregate particle; after obtaining the 3D point cloud of the sand and gravel aggregate particle, the point cloud is first centered, that is, the sand and gravel aggregate particle is placed in the center of the coordinate system, and then the covariance matrix of the point cloud is constructed and the principal component analysis (Principal Component Analysis) is used. Component Analysis (PCA) is used to estimate the major axis L, median axis I, and minor axis S of the sand and gravel aggregate particles, and then the particle shape of the sand and gravel aggregate particles is estimated; the calculation method of the flattening rate is: The value range is (0, 1); the calculation method of Wentworh flatness coefficient is: The value range is (1, +∞); the calculation method of KrumBein sphericity is: The value range is (0, 1); for the flatness, the closer the particle is to being flat, the closer the flatness is to 0, and the closer the particle is to being spherical, the closer the flatness is to 1; for the Wentworh flatness coefficient, the closer the particle is to being flat, the larger the flatness coefficient is. Compared with other shapes of sand and gravel aggregate particles, flaky sand and gravel aggregate has the highest flatness coefficient; for the KrumBein sphericity, the sphericity of spherical particles, such as blocky sand and gravel aggregate, is closer to 1, and vice versa.

[0039] In the above step 6, according to the grain shape characterization value of the sand and gravel aggregate obtained in step 5, the distribution of the grain shape of the sand and gravel aggregate in the original input two-dimensional image is statistically analyzed, and the analysis result is output.

[0040] Based on the above, the advantage of the present invention is that the present invention first uses an automated image acquisition device to capture two-dimensional images of the sand and gravel aggregates on the conveyor belt, and then uses the captured two-dimensional images as input to perform monocular depth estimation to predict depth and instance segmentation to obtain a mask, and then uses the selected aggregate particle shape characterization method to calculate and characterize the aggregate particle shape to obtain the particle shape distribution of the sand and gravel aggregate. This evaluation method has the advantages of high efficiency and high precision, and can solve the problems of strong subjectivity, high labor intensity and low efficiency in manual evaluation.

[0041] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A sand and gravel grain shape analysis method based on depth estimation using a monocular camera includes the following steps: Step 1: image acquisition; Step 2: monocular depth estimation; Step 3: instance segmentation; Step 4: calculation of three-dimensional coordinates; Step 5: particle shape characterization; Step 6: output of results; characterized by: In the above step 1, an industrial camera is used to capture a two-dimensional image of the sand and gravel aggregate pile; In the above step 2, a monocular depth estimation algorithm is used to process the two-dimensional sand and gravel aggregate pile image obtained in step 1, and a corresponding depth map is output, where each pixel value in the depth map is the distance from the camera to the sand and gravel aggregate pile; In the above step 3, the instance segmentation algorithm based on Mask R-CNN is used to perform instance segmentation on the two-dimensional sand and gravel aggregate pile image obtained in step 1, and the segmented mask image is output; In the above step 4, under the condition of knowing the camera intrinsic parameter matrix, according to the depth value and the mask map of each sand and gravel aggregate, the three-dimensional coordinates of all pixels in each mask area can be calculated according to the coordinate system transformation principle; In the above step 5, the 3D point cloud of each sand and gravel aggregate particle is obtained according to the 3D coordinates of all pixels in each mask area obtained in step 4, and the major axis L, median axis I and minor axis S of the sand and gravel aggregate particle are estimated based on this. The particle shape of the sand and gravel aggregate is characterized by three particle shape characterization methods: flattening, Wentworh flattening coefficient, and KrumBein sphericity, and the particle shape characterization value of each sand and gravel aggregate is calculated based on this. In the above step 6, according to the grain shape characterization value of the sand and gravel aggregate obtained in step 5, the distribution of the grain shape of the sand and gravel aggregate in the original input two-dimensional image is statistically analyzed, and the analysis result is output.

2. The sand and gravel grain shape analysis method based on monocular camera depth estimation according to claim 1 is characterized in that: In step one, an industrial camera is deployed vertically above the sand and gravel aggregate conveyor belt through a metal frame. Two industrial lighting devices are also deployed on both sides of the industrial camera for lighting. At the same time, a dust removal fan is also fixed on one side of the camera for dust removal. The image information collected by the industrial camera is transmitted to the embedded computing platform and analyzed using sand and gravel aggregate particle shape analysis software.

3. The sand and gravel grain shape analysis method based on monocular camera depth estimation according to claim 1, characterized in that: In step 2, the specific algorithm model adopted is the ZoeDepth model pre-trained on 12 indoor and outdoor depth estimation datasets, and fine-tuned on a self-constructed sand and gravel depth dataset. The sand and gravel depth dataset is constructed using the structure from motion recovery and multi-view stereo matching algorithms in COLMAP. By taking photos from different angles of sand and gravel aggregates, the depth of the collected images is estimated using the above method to obtain an accurate depth map, which is then used as a fine-tuning dataset for the ZoeDepth pre-trained model.

4. The sand and gravel grain shape analysis method based on monocular camera depth estimation according to claim 1, characterized in that: In the step 4, specifically: given the two-dimensional pixel coordinates (u, v) of the input sand and gravel aggregate pile two-dimensional image, the intrinsic parameter matrix K of the shooting camera and the depth value Z of each pixel point of the two-dimensional image, c In the case of , the three-dimensional camera coordinates (X c ,Y c ,Z c ),Right now: WITH c =Z c Among them, f x , f y is the camera intrinsic focal length, (u0, v0) is the principal point coordinate in the camera intrinsic parameter, the principal point is the intersection of the optical axis and the imaging plane, and is the center point of the pixel coordinate when the camera is imaging.

5. The sand and gravel grain shape analysis method based on monocular camera depth estimation according to claim 1, characterized in that: In the step five, specifically, after obtaining the three-dimensional coordinates of each pixel point of the sand and gravel aggregate image, extracting the corresponding mask area from the original two-dimensional image according to the mask result obtained after instance segmentation, each mask area represents a segmented sand and gravel aggregate particle, that is, obtaining the pixel point set of each sand and gravel aggregate particle in the original two-dimensional image, converting the coordinates of these pixel point sets into three-dimensional coordinates, and obtaining a three-dimensional point cloud of each sand and gravel aggregate particle; after obtaining the three-dimensional point cloud of the sand and gravel aggregate particle, first centering the point cloud, that is, placing the sand and gravel aggregate particle in the center of the coordinate system, and then constructing the covariance matrix of the point cloud and using principal component analysis to estimate the major axis L, medial axis I, and minor axis S of the sand and gravel aggregate particle, and then estimating the particle shape of the sand and gravel aggregate particle.

6. The sand and gravel grain shape analysis method based on monocular camera depth estimation according to claim 1, characterized in that: In step 5, the calculation method of the flatness is: The value range is (0, 1); the calculation method of Wentworh flatness coefficient is: The value range is (1, +∞); the calculation method of KrumBein sphericity is: The value range is (0, 1).

7. The method for analyzing sand and gravel shape based on depth estimation using a monocular camera according to claim 1, wherein: In step 5, for the flatness, the closer the particle is to a flat shape, the closer the flatness is to 0, and the closer the particle is to a spherical shape, the closer the flatness is to 1; for the Wentworh flatness coefficient, the closer the particle is to a flat shape, the larger the flatness coefficient value is, and compared with sand and gravel aggregate particles of other shapes, flaky sand and gravel aggregate has the highest flatness coefficient; for the KrumBein sphericity, the sphericity of spherical particles is closer to 1, and vice versa, it is closer to 0.

Images