Method for identifying needle-like and flaky aggregate particles based on multi-view image fusion

Abstract

The present invention provides a method for identifying flaky and elongated aggregate particles based on multi-view image fusion, including: collecting an aggregate image sequence from multiple perspectives, preprocessing the images and performing multi-morphology data association of the same aggregate particle, adjusting the angle of the axis where the minimum moment of inertia of the aggregate particle is located, fusing the multi-view morphologies of the aggregate particle into one aggregate image, establishing an aggregate shape classification model based on a deep learning network and classifying the aggregate shape. By fusing the aggregate morphologies in different perspectives, the present invention constructs a multi-view fusion image capable of reflecting the three-dimensional morphological characteristics of the aggregate, improves the three-dimensional morphological representation ability of the two-dimensional aggregate image, and uses the adaptive feature extraction ability of the deep learning model to construct a classification model for aggregate shape classification, effectively improving the recognition accuracy of flaky and elongated aggregates.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and particularly but not limited to a method for identifying needle-like and flaky aggregate particles based on multi-view image fusion. Background Art

[0002] Aggregates are an important component of asphalt mixtures, and their shape has an important impact on the performance of asphalt mixtures. In road engineering, in order to ensure the comprehensive performance of asphalt pavements, it is necessary to detect the content of needle-like and flaky aggregates.

[0003] Traditional methods for detecting needle-like and flaky aggregates require inspectors to measure the length and thickness of aggregate particles one by one using vernier calipers, which consumes a large amount of labor costs and has low efficiency. In order to improve the detection efficiency and reduce labor costs, more and more researchers have used machine vision technology to study the size and shape of aggregate particles. At present, aggregate image detection methods are mainly divided into static aggregate image detection methods and dynamic aggregate image detection methods. Pei Lili et al. from Chang'an University proposed in "Neural network model for calculating the particle size of road aggregates based on multi-feature factors" and "Pavement aggregate shape classification based on extreme gradient boosting" that an industrial camera is used to analyze the images of flat-laid aggregate particles, and multiple geometric parameters such as the minor axis of the equivalent ellipse, the short side of the equivalent rectangle, and the diameter of the equivalent circle of the aggregates are extracted to form multi-feature factors of the aggregates, and a multi-layer perceptron neural network is constructed to realize the calculation of the aggregate particle size, and the extreme gradient boosting algorithm is used to conduct classification research on the aggregate shape. Chen Sijia from Huaqiao University analyzed the images of manufactured sand particles during the falling process in "Development and experimental research on the detection system for the particle size and shape of manufactured sand" and developed a set of detection systems for the particle size and shape of manufactured sand.

[0004] At present, the aggregate image detection method mainly conducts particle size and shape analysis based on the morphology of aggregates from a single perspective. Since aggregates are irregular three-dimensional particles, traditional two-dimensional images can only reflect the shape characteristics of aggregates from a specific perspective. Conducting aggregate particle size and shape analysis based on the morphology of aggregates in a single perspective will seriously affect the accuracy of the detection results. To solve this problem, Fan Weijun et al. from China Jiliang University proposed a method for identifying flaky and needle-shaped aggregates based on multi-perspective image analysis under a single camera in Patent CN 114723694 A. They collected images of aggregate particles during the falling process and utilized the rotational motion of aggregates during the falling process to obtain the morphology of aggregates from multiple perspectives. They extracted the characteristics of the multi-perspective morphology of aggregates and built an integrated generalized regression network model, which greatly improved the accuracy of detecting flaky and needle-shaped aggregates. However, manually extracting the shape characteristics of aggregate particles often relies on the experience of researchers, is highly subjective, and easily ignores some effective shape information, unable to maximize the value of the multi-perspective morphology of aggregates.

[0005] In view of this, a new identification method is needed to solve at least some of the above problems. Summary of the Invention

[0006] Aiming at the problem that the low utilization rate of the multi-perspective morphology of existing aggregates restricts the accuracy of identifying flaky and needle-shaped aggregates, the present invention proposes a method for identifying flaky and needle-shaped aggregate particles based on multi-perspective image fusion. By fusing the morphologies of aggregates from different perspectives and using a deep learning model with self-adaptive feature extraction ability, a two-dimensional aggregate image that can reflect the three-dimensional morphological characteristics of aggregates is constructed, thereby improving the identification rate of flaky and needle-shaped aggregates.

[0007] The technical solution for achieving the purpose of the present invention is as follows:

[0008] According to one aspect of the present invention, a method for identifying flaky and needle-shaped aggregate particles based on multi-perspective image fusion includes:

[0009] Step 1: Collect an aggregate image sequence: The vibrating feeder controls the falling of aggregates, and images of aggregates during the falling process are collected to obtain an aggregate image sequence from multiple perspectives;

[0010] Step 2: Image preprocessing and multi-morphology data association: Threshold segmentation is performed on the aggregate image sequence, aggregate particles with incomplete morphologies at the upper and lower boundaries of the image are removed, and the morphological data of the same aggregate particle are associated to obtain the multi-perspective morphological data of the aggregate particle;

[0011] Step 3: Adjust the morphological angle of the aggregate particle: Calculate the counterclockwise angle between the axis where the minimum moment of inertia of the aggregate particle morphology is located and the horizontal axis, and rotate clockwise by an equal angle to make the axis where the minimum moment of inertia of the aggregate particle is located in a horizontal state;

[0012] Step 4: Fuse the multi - perspective shapes of aggregate particles: Based on the centroid coordinates of the aggregate particles, translate the multiple shapes of the aggregate particles to the target position in the same image for pixel value superposition, so that the multi - perspective shapes of the aggregate particles are fused into one aggregate image, and map the pixel value range of this aggregate image proportionally to [0, 255] to construct a multi - perspective shape fusion image of the aggregate particles;

[0013] Step 5: Construct a classification model and classify: Establish an aggregate shape classification model based on a deep learning network and classify the aggregate shapes.

[0014] Furthermore, in the method for identifying needle - like and flaky aggregate particles based on multi - perspective image fusion of the present invention, in Step 1, an industrial camera is used to collect the aggregate images during the falling process, and the backlight shooting method is adopted, and the industrial camera and the light source are installed on both sides of the aggregate falling position.

[0015] Furthermore, in the method for identifying needle - like and flaky aggregate particles based on multi - perspective image fusion of the present invention, the frame rate of the industrial camera is 70fps, and the height of the shooting field of view is 24cm.

[0016] Furthermore, in the method for identifying needle - like and flaky aggregate particles based on multi - perspective image fusion of the present invention, the specific steps of image pre - processing and multi - shape data association in Step 2 include:

[0017] Step 2 - 1: Perform threshold segmentation on the aggregate image sequence using the single - threshold segmentation method to obtain a binary aggregate image;

[0018] Step 2 - 2: Use a particle analysis operator to extract the position information of the aggregate particles, and the position information includes the minimum pixel y - coordinate Bounding Rect Top and the maximum pixel y - coordinate Bounding Rect Bottom;

[0019] Step 2 - 3: Identify the aggregate particles with Bounding Rect Top equal to 1 or Bounding Rect Bottom equal to the image pixel height as incomplete particles, and remove the incomplete particles from the image;

[0020] Step 2 - 4: Track the aggregate particles according to the movement law of the aggregate particles in the image sequence, associate the image sequences of multiple shapes belonging to the same aggregate particle, and obtain the multi - perspective shape data of the aggregate particles.

[0021] Furthermore, in the method for identifying needle - like and flaky aggregate particles based on multi - perspective image fusion of the present invention, the calculation formula for the counter - clockwise angle between the axis where the minimum moment of inertia of the aggregate particle shape lies and the horizontal axis in Step 3 is as follows:

[0022]

[0023] Among them, n is the number of pixels included in the aggregate particle, and (x i , y i ) is the set of pixel coordinates of the aggregate particle, x i is the horizontal pixel coordinate, and y i is the vertical pixel coordinate.

[0024] Furthermore, for the method for identifying flaky and elongated aggregate particles based on multi-view image fusion of the present invention, the formula for proportionally mapping the pixel value range of the image to [0, 255] in step 4 is as follows:

[0025]

[0026] Among them, p is the pixel value after image superposition, p max is the maximum pixel value after image superposition, and p * is the pixel value after pixel mapping.

[0027] Furthermore, for the method for identifying flaky and elongated aggregate particles based on multi-view image fusion of the present invention, the specific steps of constructing a classification model and classifying in step 5 include:

[0028] Step 5-1: Divide the multi-view morphological fusion image of the aggregate particle into a training set and a test set;

[0029] Step 5-2: Input the multi-view morphological fusion image of the aggregate particle in the training set into an 18-layer deep residual network for training to establish an aggregate particle shape classifier;

[0030] Step 5-3: Use the aggregate shape classifier to perform shape classification on the multi-view morphological fusion image of the aggregate particle in the test set and verify the accuracy of the model.

[0031] Compared with the prior art by adopting the above technical solutions, the present invention has the following technical effects:

[0032] 1. For the method for identifying flaky and elongated aggregate particles based on multi-view image fusion of the present invention, by continuously and rapidly collecting images of the aggregate particles during the falling process, the morphology of the aggregate particles from multiple perspectives can be obtained, which is beneficial to the identification of flaky and elongated aggregate particles.

[0033] 2. For the method for identifying flaky and elongated aggregate particles based on multi-view image fusion of the present invention, by fusing multiple morphologies of the aggregate particles from different perspectives, a multi-view fusion image capable of reflecting the three-dimensional morphological characteristics of the aggregate is constructed, improving the three-dimensional morphological representation ability of the two-dimensional aggregate image.

[0034] 3. The method for identifying flaky and elongated aggregates based on multi-view image fusion of the present invention uses the adaptive feature extraction ability of the deep residual network to classify the shapes of aggregates, effectively improving the recognition accuracy of flaky and elongated aggregates. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings are used to provide a further understanding of the present invention and, together with the description, are used to explain the embodiments of the present invention and do not constitute a limitation on the present invention. In the drawings:

[0036] Figure 1 The schematic diagram shows the angle between the axis where the minimum moment of inertia of the aggregate particle in the method for identifying flaky and elongated aggregates based on multi-view image fusion of the present invention and the horizontal axis.

[0037] Figure 2 The schematic diagram shows the process of multi-view morphological superposition of the binarized aggregate in the method for identifying flaky and elongated aggregates based on multi-view image fusion of the present invention.

[0038] Figure 3 The multi-view fusion image of the aggregate particle in the method for identifying flaky and elongated aggregates based on multi-view image fusion of the present invention is shown.

[0039] Figure 4 The schematic diagram shows the classification results of the shapes of aggregate particles in the method for identifying flaky and elongated aggregates based on multi-view image fusion of the present invention and the multi-view image analysis method.

[0040] Figure 5 The implementation flowchart of the method for identifying flaky and elongated aggregates based on multi-view image fusion of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To further understand the present invention, the preferred implementation embodiments of the present invention will be described below in conjunction with the embodiments. However, it should be understood that these descriptions are only for further explaining the features and advantages of the present invention and not for limiting the claims of the present invention.

[0042] The description of this part only focuses on typical embodiments, and the present invention is not limited to the scope described in the embodiments. Combinations of different embodiments, mutual replacement of some technical features in different embodiments, and mutual replacement of the same or similar prior art means and some technical features in the embodiments are also within the scope of description and protection of the present invention.

[0043] In this embodiment, the method for identifying flaky and elongated aggregate particles based on multi-view image fusion of the present invention is adopted. For a concrete mixing plant in Shandong Province, an industrial camera is used to shoot the falling process of aggregate particles to obtain the morphological data of the aggregates from multiple perspectives. By fusing the multi-view morphologies of the aggregates, a multi-view image of the aggregates is constructed, and the shape classification of the aggregates is completed in combination with a deep learning model to realize the detection of flaky and elongated aggregates. The specific implementation steps are as followsFigure 5 As shown, including:

[0044] Step 1: Collect aggregate image sequence: Use a vibrating feeder to control the uniform and dispersed fall of aggregate particles to provide good image acquisition conditions for the imaging equipment; install an industrial camera and light source on both sides of the aggregate falling position, and use a backlight shooting method to use an industrial camera to collect aggregate images during the falling process to obtain aggregate image sequences from multiple perspectives.

[0045] In one embodiment, the image acquisition frame rate of the industrial camera is 70fps, the field of view height is 24cm, and each aggregate particle appears 6 times or more in the image sequence, so as to obtain sufficient aggregate shape information to ensure the accuracy of recognition.

[0046] Step 2: Image preprocessing and multi-morphological data association: Threshold segmentation is performed on the aggregate image sequence to remove aggregate particles with incomplete morphology at the upper and lower boundaries of the image, and the morphological data of the same aggregate particles are associated to obtain multi-view morphological data of aggregate particles. The specific steps include:

[0047] Step 2-1: Read the aggregate image sequence, and perform threshold segmentation on the aggregate image sequence using a single threshold segmentation method to obtain an aggregate binary image;

[0048] Step 2-2: using a particle analysis operator to extract position information of aggregate particles, the position information including a minimum pixel y coordinate Bounding Rect Top and a maximum pixel y coordinate Bounding Rect Bottom;

[0049] Step 2-3: Identify aggregate particles whose Bounding Rect Top is equal to 1 or whose Bounding Rect Bottom is equal to the image pixel height as incomplete particles, and remove the incomplete particles from the image;

[0050] Step 2-4: According to the average movement speed of the aggregate particles in the image sequence, the aggregate particles are tracked, and the image sequences of multiple forms belonging to the same aggregate particle are associated to obtain multi-view morphological data of the aggregate particles.

[0051] Step 3: Adjust the shape angle of aggregate particles: Calculate the counterclockwise angle between the axis of the minimum moment of inertia of the aggregate particle shape and the horizontal axis, such as Figure 1 The position relationship between the aggregate particles and the horizontal axis in the image is shown; and the aggregate particles are rotated clockwise by an equal angle so that the axis with the minimum moment of inertia of the aggregate particles is in a horizontal state. Among them:

[0052] The calculation formula for the counterclockwise angle between the axis of the minimum moment of inertia of the aggregate particle shape and the horizontal axis is as follows:

[0053]

[0054] In formula (1), n is the number of pixels included in the aggregate particle, and (x i , y i ) is the set of pixel coordinates of the aggregate particle, where x i is the horizontal pixel coordinate and y i is the vertical pixel coordinate.

[0055] Step 4: Fuse the multi-view shapes of the aggregate particles: Based on the centroid coordinates of the aggregate particles, translate the multiple binary shapes of the aggregate particles to the target positions in the same image for pixel value superposition, so that the multi-view shapes of the aggregate particles are fused into one aggregate image, as shown in Figure 2 which shows the specific process of aggregate shape superposition; adjust the pixel values of the image obtained by multi-view shape superposition, proportionally map the pixel value range to [0, 255], and construct a multi-view shape fusion image of the aggregate particles, as shown in Figure 3 which shows the effect of the aggregate image after fusing multiple shapes of the target aggregate. Among them:

[0056] The formula for proportionally mapping the pixel value range of the image to [0, 255] is as follows:

[0057]

[0058] where p is the pixel value after image superposition, p max is the maximum pixel value after image superposition, and p * is the pixel value after pixel mapping.

[0059] Step 5: Construct a classification model and classify: Establish an aggregate shape classification model based on a deep learning network and classify the aggregate shapes. The specific steps include:

[0060] Step 5-1: Divide the multi-view shape fusion image of the aggregate particles into a training set and a test set;

[0061] Step 5-2: Input the multi-view shape fusion image of the aggregate particles in the training set into an 18-layer deep residual network for training, and use the adaptive feature extraction ability of the deep residual network to establish an aggregate particle shape classifier with high accuracy;

[0062] Step 5-3: Use the aggregate shape classifier to classify the shape of the multi-view shape fusion image of the aggregate particles in the test set and verify the accuracy of the model.

[0063] The shape classification of the same batch of aggregate particles is carried out by using the method for identifying flaky and elongated aggregate particles based on multi-view image fusion proposed by the present invention and the method for identifying flaky and elongated aggregates based on multi-view image analysis under a single camera in the prior art. The 5 classification results of the two methods are as Figure 4 shown. It can be seen from Figure 4 that the method for identifying flaky and elongated aggregate particles based on multi-view image fusion of the present invention can effectively improve the recognition accuracy of flaky and elongated aggregate particles, is applicable to the detection of the content of flaky and elongated aggregates in road construction, and has significant engineering application value.

[0064] The description and application of the present invention here are illustrative and do not intend to limit the scope of the present invention to the above embodiments. The related descriptions of effects or advantages in the specification may not be reflected in actual experimental examples due to uncertainties in specific condition parameters or other factors. The related descriptions of effects or advantages are not used to limit the scope of the invention. The deformations and changes of the disclosed embodiments here are possible, and the substitutions and equivalent various components of the embodiments are well-known to those of ordinary skill in the art. Those skilled in the art should clearly understand that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials, and parts without departing from the spirit or essential characteristics of the present invention. Other deformations and changes can be made to the disclosed embodiments here without departing from the scope and spirit of the present invention.

Claims

1. A method for identifying needle-like and flaky aggregate particles based on multi-view image fusion, characterized in that, Including: Step 1: Collect the aggregate image sequence: The vibrating feeder controls the falling of the aggregate, and the aggregate images during the falling process are collected to obtain the aggregate image sequences from multiple perspectives. Step 2: Image preprocessing and multi-modal data association: Threshold segmentation is performed on the aggregate image sequence, the aggregate particles with incomplete morphology at the upper and lower boundaries of the image are removed, and the morphological data of the same aggregate particle are associated to obtain the multi-perspective morphological data of the aggregate particle. The specific steps include: Step 2-1: Threshold segmentation is performed on the aggregate image sequence using the single-threshold segmentation method to obtain the binary aggregate image. Step 2-2: The particle analysis operator is used to extract the position information of the aggregate particles, and the position information includes the minimum pixel y coordinate Bounding Rect Top and the maximum pixel y coordinate Bounding Rect Bottom. Step 2-3: The aggregate particles with Bounding Rect Top equal to 1 or Bounding Rect Bottom equal to the image pixel height are identified as incomplete particles, and the incomplete particles are removed from the image. Step 2-4: The aggregate particles are tracked according to the movement law of the aggregate particles in the image sequence, and the image sequences of multiple morphologies belonging to the same aggregate particle are associated to obtain the multi-perspective morphological data of the aggregate particle. Step 3: Adjust the morphological angle of the aggregate particle: Calculate the counterclockwise angle between the axis where the minimum moment of inertia of the aggregate particle morphology is located and the horizontal axis, and rotate clockwise by an equal angle so that the axis where the minimum moment of inertia of the aggregate particle is located is in the horizontal state. The calculation formula is as follows: Among them, n is the number of pixels included in the aggregate particle,( x i , y i ) is the set of pixel coordinates of the aggregate particle, x i is the abscissa of the pixel, y i is the ordinate of the pixel; Step 4: Fuse the multi-perspective morphologies of the aggregate particle: Based on the centroid coordinates of the aggregate particle, the multiple morphologies of the aggregate particle are translated to the target position in the same image for pixel value superposition, so that the multi-perspective morphologies of the aggregate particle are fused into one aggregate image, and the pixel value range of the aggregate image is proportionally mapped to [0, 255] to construct the multi-perspective morphology fusion image of the aggregate particle. Step 5: Build a classification model and classify: Based on the deep learning network, an aggregate shape classification model is established and the aggregate shape is classified.

2. The method for identifying needle-like and flaky aggregate particles based on multi-view image fusion according to claim 1, characterized in that In Step 1, an industrial camera is used to collect the aggregate images during the falling process, and the backlight shooting method is adopted, and the industrial camera and the light source are installed on both sides of the aggregate falling position.

3. The method for identifying needle-like and flaky aggregate particles based on multi-view image fusion according to claim 2, wherein The frame rate of the industrial camera is 70fps, and the height of the shooting field of view is 24cm.

4. The method for identifying needle-like and flaky aggregate particles based on multi-view image fusion according to claim 1, wherein The formula for proportionally mapping the pixel value range of the image to [0, 255] in Step 4 is as follows: Among them, p is the pixel value after image superposition, p max is the maximum pixel value after image superposition, p * is the pixel value after pixel mapping.

5. The method for identifying needle-like and flaky aggregate particles based on multi-view image fusion according to claim 1, characterized in that The specific steps for Step 5 to build a classification model and classify include: Step 5-1: Divide the multi-perspective morphology fusion image of the aggregate particle into a training set and a test set. Step 5-2: Input the multi-perspective morphology fusion image of the aggregate particle in the training set into the 18-layer deep residual network for training to establish an aggregate particle shape classifier. Step 5-3: Use the aggregate shape classifier to perform shape classification on the multi-perspective morphology fusion image of the aggregate particle in the test set and verify the accuracy of the model.

Images