Multi-mode intelligent detection method for filling quality of rockfill material

Abstract

The invention discloses a rockfill material filling quality multi-mode intelligent detection method, and belongs to the technical field of civil engineering filling quality detection. According to the method, aiming at the problems that traditional rockfill material grading detection is low in efficiency and insufficient in precision, high-resolution color images, three-dimensional point clouds and near-infrared data of the surface of a rockfill material are synchronously collected through multiple sensors, and multi-source data alignment is achieved through space-time registration; carrying out feature layer deep fusion by adopting an attention-guided multi-mode fusion network to generate a fusion feature map; an adaptive network based on neural architecture search automatically selects an optimal path to realize particle segmentation; reconstructing a particle three-dimensional model by using multi-view geometric constraints and minimizing energy functions of a fusion data item, a smooth item and a luminosity item; and finally, extracting grain composition characteristics to realize quality evaluation. The method is mainly used for rapidly and accurately detecting the filling quality of rockfill materials such as water conservancy projects and highway roadbeds.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of civil engineering filling quality detection. More specifically, the present application relates to a multi-modal intelligent detection method for rockfill filling quality. BACKGROUND

[0002] Rockfill filling, as a key link in the construction of infrastructure such as water conservancy projects and road subgrades, its quality is directly related to the safety and durability of the entire project. The core control index of filling quality is the particle size distribution, i.e. the mass distribution of particles of different sizes. The traditional grading detection method mainly relies on on-site manual sieving sampling and laboratory analysis. This method not only consumes time and effort, but also destroys the structure of the filling body, and the sampling rate is extremely low, making it difficult to fully reflect the true quality of the entire construction surface.

[0003] With the development of visual sensing technology, particle size detection methods based on image analysis have gradually been applied. This method collects rockfill surface images, uses image processing algorithms for particle segmentation and size statistics, and to some extent improves the detection efficiency. However, relying solely on two-dimensional image information has inherent limitations: images are easily disturbed by factors such as lighting conditions, particle overlap, and shadow obstruction, leading to decreased segmentation accuracy; more importantly, two-dimensional images cannot provide three-dimensional morphological information of particles, resulting in significant errors in particle size calculation.

[0004] To address the shortcomings of two-dimensional measurement, researchers have begun to introduce three-dimensional scanning technology, which uses laser or structured light devices to obtain point cloud data of the rockfill surface. Three-dimensional point cloud data can provide more geometric information, which is beneficial to the three-dimensional recognition and segmentation of particles. However, in practical applications, single three-dimensional data also has problems: although point cloud data has high geometric accuracy, it lacks color and texture information, and the segmentation effect in areas where particle boundaries are blurred or particles are severely connected is still not ideal; in addition, point cloud data is easily disturbed by environmental light during collection, resulting in high noise.

[0005] The introduction of new sensing technologies such as near-infrared spectroscopy provides a new way for particle material identification, which helps to distinguish particles with similar appearance but different materials. However, how to effectively fuse multi-source heterogeneous data such as images, three-dimensional point clouds, and near-infrared data has become a new technical challenge. Multi-source data has differences in resolution, coordinate system, and physical meaning, and needs accurate spatio-temporal registration and information fusion to take full advantage of each data source. Existing simple fusion methods often fail to fully consider the complementarity and redundancy between different modal data, resulting in poor fusion results.

[0006] Therefore, developing a new method capable of comprehensively utilizing multi-source sensing information to realize automatic, accurate and non-destructive detection of rockfill particles has become an urgent need in engineering practice. This method needs to solve a series of key technical problems such as high-precision registration of multi-modal data, deep feature fusion and accurate reconstruction of three-dimensional morphology, thereby providing a more reliable technical means for rockfill filling quality evaluation. SUMMARY

[0007] An object of the present application is to solve at least the above problems and provide at least the advantages to be described later.

[0008] Another object of the present application is to provide a multi-modal intelligent detection method for rockfill filling quality, which realizes automatic and accurate identification and three-dimensional reconstruction of rockfill particles, significantly improves the segmentation accuracy in occlusion and adhesion scenes through multi-modal data fusion, and ensures the model reconstruction quality of complex texture particles through adaptive energy function optimization, thereby providing an efficient and reliable non-destructive detection means for filling gradation evaluation.

[0009] To achieve these objects and other advantages in accordance with the present application, a multi-modal intelligent detection method for rockfill filling quality is provided, comprising: S1, synchronously collecting high-resolution color images, three-dimensional structured light point cloud data and near-infrared reflection data on the surface of rockfill through multiple sensors, and realizing spatial alignment and time synchronization of multi-source data through a built-in space-time registration model; S2, inputting the registered multi-source data obtained in step S1 into an attention-guided multi-modal fusion network, and performing deep fusion to form a fusion feature map at a feature layer; S3, constructing an adaptive network based on neural architecture search, automatically selecting an optimal network path for particle segmentation according to the fusion feature map obtained in step S2, and obtaining a two-dimensional image morphology Si of the particles; S4, based on the particle segmentation result obtained in step S3, reconstructing a three-dimensional model of each particle and extracting morphological parameters using multi-view geometric constraints; S5, based on the three-dimensional model of each particle reconstructed in step S4, extracting the gradation features of the rockfill and performing quality evaluation; Wherein, the three-dimensional model of each particle reconstructed in step S4 is realized by minimizing the following energy function: ; E(S i ) is the energy function of the i-th particle; E data (S i ) is the data item energy; ;N views is the number of different camera views used to reconstruct the particle i; To apply the projection matrix of the j-th camera, project the 3D model of the i-th particle onto the 2D imaging plane of the j-th camera; Q i,j The two-dimensional segmentation mask for the i-th particle in the j-th view; Weights for data items; ; ε represents the basic weight of the data item; its value ranges from 1.0 to 2.0; ε is the standard deviation of geometric measurement noise obtained through sensor calibration, in mm; ε0 is the reference noise level, in mm. E smooth (S i ) represents the energy of the smoothing term. (u, v) represents an edge in the reconstructed 3D model mesh; D is the set of edges in the reconstructed 3D model; n u n v c represents the normal vectors of vertices u and v in the reconstructed 3D model. u c v , respectively, represent the curvature of vertices u and v in the reconstructed 3D model; μ is the curvature weighting factor, with a value of 0.2~0.5; β is the weight of the smoothing term. β0 is the basic weight of the smoothing term, ranging from 0.1 to 0.3; η is the adaptive coefficient of the smoothing term, ranging from 0.5 to 1.0; A i Let be the surface area of ​​the i-th particle, mm. 2 A max The maximum particle surface area, in mm 2 ; E photo (S i () represents the photometric uniformity energy. ; p is the current center pixel, Ω is the set of pixels corresponding to the i-th particle in all viewpoints; q is a specific pixel in the neighborhood, N(p) is the set of pixels in the neighborhood of point p; I(p) and I(q) are the grayscale or color values ​​of pixels p and q, respectively. ρ(I p ,I q ) is a Gaussian weighting function based on the similarity between the center point p and the neighborhood point q; γ is the weight of the photometric term. γ0 is the basic weight of the photometric term, with a value ranging from 0.5 to 1.0; C i C0 is the texture richness index calculated by the image variance for the i-th particle, C0 is the texture richness threshold, and k is the photometric term adjustment coefficient, with a value of 0.5 to 3.0.

[0010] Preferably, the multi-sensor in step S1 comprises an RGB camera for capturing high-resolution color images, a 3D structured light camera for capturing 3D structured light point cloud data, and a near-infrared camera for capturing near-infrared reflection data.

[0011] Preferably, the spatio-temporal registration model in step S1 is: ; is a coordinate transformation matrix from the RGB camera to the 3D structured light camera; is a coordinate transformation matrix from the near-infrared camera to the RGB camera; N is the total number of pairs of feature points successfully matched and used to calculate the transformation matrix T in the images obtained by the RGB camera and the 3D structured light camera; is the coordinate of the i-th feature point in the RGB camera; is the coordinate of the i-th feature point in the 3D structured light camera; denotes the action of the transformation matrix T on the point above; R(T) is the regularization term of the transformation matrix, ; is a rotation matrix including the pitch angle θ and the yaw angle Ф, i.e. wherein: ; ; is a translation matrix, ; T is a homogeneous transformation matrix: ; wherein, ; is a calibration error compensation term; h is a regularization coefficient, taking a value of 0.1-10.

[0012] Preferably, step S2 forms a fusion feature map F fusion : ; F fusion is a fusion feature map; A is an attention map, ; σ is a Sigmoid function that compresses the output value to the range of (0, 1); W c、 W d、 Wt and W a are learnable weight matrices; b a is a learnable bias vector; concat is a concatenation operation along the channel dimension; is an element-wise multiplication; F c is a color feature map extracted from a high-resolution color image inputted into a convolutional neural network (CNN) by an RGB camera; F d is a depth feature map extracted from three-dimensional structured light point cloud data inputted into a convolutional neural network (CNN) by a 3D structured light camera; F t is a near-infrared feature map extracted from near-infrared reflection data inputted into a convolutional neural network (CNN) by a near-infrared camera.

[0013] Preferably, the optimal network path is automatically selected in step S3 according to the complexity of the fusion feature map: if G(F fusion ) > τ1, path A is selected; path A: 16 layers of convolution, each layer with 256 channels and containing an attention module; if τ2 ≤ G(F fusion ) ≤ τ1, path B is selected; path B: 8 layers of convolution, each layer with 128 channels; if G(F fusion ) < τ2, path C is selected; path C: 4 layers of convolution, each layer with 64 channels; G(F fusion ) is a complexity measurement function of the fusion feature map, and τ1 and τ2 are complexity threshold parameters, which are set according to the sample size.

[0014] Preferably, the complexity measurement function of the fusion feature map is an average absolute gradient-based measurement function: ; H and L are the height and width of the fusion feature map, respectively, in mm; is the gradient amplitude of the fusion feature map at position (i, j).

[0015] Preferably, the process of extracting the gradation features of the rockfill material and performing quality evaluation in step S5 includes: For the i-th particle, calculate its equivalent particle size: ; Calculate the gradation curve: ; wherein d i is the equivalent particle diameter of the i-th particle, mm; X i is the long axis length of the i-th particle, mm; Y i is the short axis length of the i-th particle, mm; P(d) is the mass percentage of particles with particle size d; I(d i ≤d) is an indicator function; v i is the volume of the i-th particle, mm 3 ; K is the total number of particles.

[0016] The present application at least includes the following beneficial effects: Firstly, the present application synchronously collects high-resolution color images, three-dimensional structured light point cloud data and near-infrared reflection data of the rockfill surface through multiple sensors, and realizes accurate alignment of multi-source data by using a space-time registration model, effectively solving the problems of limited view angle and incomplete information existing in traditional single-sensor data collection; the collaborative use of multi-modal data significantly improves the robustness of data collection under complex working conditions, providing a richer and more accurate data basis for subsequent processing; the fusion collection mode of multi-source heterogeneous data can also adapt to complex lighting conditions and particle distribution conditions on site, ensuring the acquisition of comprehensive and reliable raw data; Secondly, the present application innovatively realizes the deep fusion of feature levels through a multi-modal fusion network based on attention guidance, highlights the superior features of different modal data through an adaptive weighting mechanism, effectively utilizes the texture details of color images, the geometric accuracy of point cloud data and the material distinguishing ability of near-infrared data, generates a fusion feature map with more information, retains the unique advantages of each modal data, and also produces a synergistic enhancement effect, significantly improving the accuracy of subsequent particle segmentation, especially in scenes such as particle edges and adhesion areas where traditional methods are not effective; Thirdly, the adaptive network constructed by the neural architecture search technology can automatically select the optimal processing path according to the complexity of the fusion feature map, realizing the intelligent balance of calculation efficiency and accuracy, using deep networks for complex scenes to ensure accuracy and using lightweight networks for simple scenes to improve speed, effectively solving the efficiency bottleneck problem of traditional fixed structure networks in processing diversified rockfill scenes, so that the system can intelligently adapt to the particle distribution characteristics of different construction sites, significantly improving the processing efficiency while ensuring the accuracy; Fourthly, the application even adopts a multi-view geometric constraint reconstruction algorithm, realizes high-precision reconstruction of the particle three-dimensional model by minimizing an energy function of fusion data items, smoothing items and luminosity items, and an adaptive weight mechanism can dynamically adjust the contribution of each constraint according to particle surface area, texture richness and other characteristics, so that different characteristic particles can obtain optimal reconstruction effect, effectively solves the problems of inaccurate reconstruction of texture missing areas and excessive smoothing of complex texture areas in traditional three-dimensional reconstruction, and provides accurate three-dimensional morphological parameters for subsequent gradation analysis. Fifthly, based on the accurate three-dimensional reconstruction result, the application realizes scientific evaluation of the rockfill material filling quality by calculating the equivalent particle diameter and the gradation curve, uses three-dimensional volume parameters instead of traditional two-dimensional projection parameters for gradation calculation, significantly improves the measurement accuracy of the particle size distribution, realizes automatic processing from data acquisition to quality evaluation, avoids errors caused by manual sampling, and provides comprehensive and continuous filling quality monitoring, thereby providing reliable technical support for engineering quality control.

[0017] Other advantages, objects and features of the application will be partly embodied in the following description, and partly understood by those skilled in the art through research and practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of the rockfill material filling quality multi-modal intelligent detection method described in one technical solution of the application is shown. Figure 2 A specific flowchart of the rockfill material filling quality multi-modal intelligent detection method described in another technical solution of the application is shown. DETAILED DESCRIPTION

[0019] The application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement it according to the description.

[0020] It should be understood that the terms such as "have", "contain" and "include" used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0021] As shown in Figure 1 , 2 The application provides a rockfill material filling quality multi-modal intelligent detection method, which comprises: S1, high-resolution color images, three-dimensional structured light point cloud data and near-infrared reflection data on the surface of the rockfill material are synchronously collected through a plurality of sensors, and spatial alignment and time synchronization of the multi-source data are realized through a built time-space registration model; S2, the registered multi-source data obtained in step S1 are input into an attention-guided multi-modal fusion network, deep fusion is performed on the feature layer to form a fusion feature map; S3. Construct an adaptive network based on neural architecture search, and automatically select the optimal network path for particle segmentation based on the fusion feature map obtained in step S2 to obtain the two-dimensional image morphology Si of the particles. S4. Based on the particle segmentation results obtained in step S3, the three-dimensional model of each particle is reconstructed using multi-view geometric constraints and morphological parameters are extracted. S5. Based on the three-dimensional model of each particle reconstructed in step S4, extract the gradation characteristics of the rockfill and conduct a quality assessment. In step S4, reconstructing the three-dimensional model of each particle is achieved by minimizing the following energy function: ; E(S i Let be the energy function of the i-th particle; E data (S i ) represents the energy of the data item; N views The number of different camera viewpoints used to reconstruct particle i; To apply the projection matrix of the j-th camera, project the 3D model of the i-th particle onto the 2D imaging plane of the j-th camera; Q i,j The two-dimensional segmentation mask for the i-th particle in the j-th view; Weights for data items; ; ε represents the basic weight of the data item; its value ranges from 1.0 to 2.0; ε is the standard deviation of geometric measurement noise obtained through sensor calibration, in mm; ε0 is the reference noise level, in mm. E smooth (S i ) represents the energy of the smoothing term. (u, v) represents an edge in the reconstructed 3D model mesh; D is the set of edges in the reconstructed 3D model; n u n v c represents the normal vectors of vertices u and v in the reconstructed 3D model. u c v , respectively, represent the curvature of vertices u and v in the reconstructed 3D model; μ is the curvature weighting factor, with a value of 0.2~0.5; β is the weight of the smoothing term. β0 is the basic weight of the smoothing term, ranging from 0.1 to 0.3; η is the adaptive coefficient of the smoothing term, ranging from 0.5 to 1.0; A i Let be the surface area of ​​the i-th particle, mm. 2 A max The maximum particle surface area, in mm 2 ; E photo(S i () represents the photometric uniformity energy. ; p is the current center pixel, Ω is the set of pixels corresponding to the i-th particle in all viewpoints; q is a specific pixel in the neighborhood, N(p) is the set of pixels in the neighborhood of point p; I(p) and I(q) are the grayscale or color values ​​of pixels p and q, respectively. ρ(I p ,I q ) is a Gaussian weighting function based on the similarity between the center point p and the neighborhood point q; γ is the weight of the photometric term. γ0 is the basic weight of the photometric term, with a value ranging from 0.5 to 1.0; C i C0 is the texture richness index calculated by the image variance for the i-th particle, C0 is the texture richness threshold, and k is the photometric term adjustment coefficient, with a value of 0.5 to 3.0.

[0022] The multimodal intelligent detection method for riprap filling quality provided by this invention is implemented through a multi-sensor synchronous acquisition system. This system includes at least a visual sensor for acquiring high-resolution color images, a depth sensor for acquiring three-dimensional structured light point cloud data, and a spectral sensor for acquiring near-infrared reflectance data. Each sensor achieves synchronous acquisition through a unified hardware trigger signal, ensuring complete temporal alignment of different modal data and laying the foundation for subsequent spatiotemporal registration. The data acquired by the multi-sensor synchronous acquisition system achieves spatial alignment and temporal synchronization of multi-source data through an established spatiotemporal registration model. The multimodal fusion network adopts an attention-guided network architecture. First, features are extracted from each modal data to generate corresponding feature maps. The weight distribution of each feature map is calculated through an attention mechanism, and an attention map is generated using a Sigmoid activation function. Finally, feature weighted fusion is achieved through element-wise multiplication. During energy function optimization, the reference noise level ε0 is preferably set to 0.5 mm, the basic weight γ0 of the photometric term is preferably 0.8, and the texture richness threshold C0 is adjusted between 0.3 and 0.6 based on the characteristics of the materials on site. The 3D reconstruction process is based on the principle of minimizing the energy function, and is solved iteratively through an optimization algorithm. The energy function consists of three components: a data term, a smoothing term, and a luminosity term, where the data term has a weight. The smoothing term weight β is dynamically adjusted based on the geometric measurement noise standard deviation, while the photometric term weight γ is adjusted according to the texture richness index. During the optimization process, parameters such as the reference noise level ε0 and the texture richness threshold C0 are set according to the actual detection requirements.

[0023] According to the above technical solution, a specific workflow of the multimodal intelligent detection process for riprap filling quality is as follows: Multiple sensors simultaneously acquire multimodal data from the riprap surface. A visual sensor acquires high-resolution color images, a depth sensor obtains 3D point cloud data, and a spectral sensor records near-infrared reflectance characteristics. All data acquisition is completed within milliseconds, ensuring data temporal synchronization. After acquisition, a spatiotemporal registration model transforms the multi-source data to a unified coordinate system, achieving spatial alignment and temporal synchronization. The registered data is input into an attention-guided multimodal fusion network. The network first extracts features from each modality, then calculates feature weights through an attention mechanism, ultimately generating a fused feature map. The resulting fused feature map is input into an adaptive network based on neural architecture search, automatically selecting the optimal network path for particle segmentation, obtaining a 2D image segmentation result for each particle. Based on the segmentation result for each particle, 3D reconstruction is performed using multi-view geometric constraints. For each particle, observation data is extracted from multiple perspectives, a 3D model is constructed, and optimization is performed by minimizing the energy function. During reconstruction, data terms ensure projection consistency, smoothing terms maintain surface smoothness, and photometric terms ensure texture continuity. Finally, the morphological parameters of each particle are extracted, and the gradation characteristics are calculated for quality assessment.

[0024] According to the above technical solution, the multimodal intelligent detection method for riprap filling quality provided by this invention has at least the following technical effects: The multi-sensor system simultaneously utilizes texture information from color images, geometric information from point cloud data, and material information from near-infrared data, overcoming the limitations of a single data source. The attention-guided fusion mechanism effectively highlights the advantageous characteristics of each modality of data, maintaining reliable detection performance even under complex working conditions. The dynamic adjustment of data term weights in the polynomial structure of the energy function adapts to different measurement noise environments; the adaptive change of smoothing term weights takes into account the detailed features of particles of different sizes; and the automatic adjustment of photometric term weights ensures the surface reconstruction quality of particles with different texture characteristics, adapting to diverse field conditions. The entire process from data acquisition to quality assessment is automated, avoiding errors caused by manual intervention. The gradation analysis method based on a three-dimensional model provides more accurate quality assessment results, offering reliable technical support for riprap filling quality control.

[0025] In one technical solution, step S1 involves multiple sensors including an RGB camera for acquiring high-resolution color images, a 3D structured light camera for acquiring three-dimensional structured light point cloud data, and a near-infrared camera for acquiring near-infrared reflection data. This invention provides a preferred layout for a multi-sensor synchronous acquisition system, including specialized cameras: an RGB camera, a 3D structured light camera, and a near-infrared camera. In terms of hardware configuration, the RGB camera uses a high-resolution model capable of capturing detailed texture information of the riprap surface; the 3D structured light camera employs an active projection of structured light patterns, obtaining accurate three-dimensional point cloud data by calculating light spot deformation; and the near-infrared camera is equipped with specific optical filters to specifically capture the reflectivity of materials in the near-infrared band. The layout of the multi-sensor synchronous acquisition system provided by this invention enables simultaneous acquisition of multi-dimensional data, significantly improving the integrity and reliability of data acquisition and overcoming the limitations of single-sensor information. Furthermore, each type of camera fully leverages its professional advantages, forming a complementary data acquisition system that adapts to different field environmental conditions and lays a solid foundation for subsequent multimodal data fusion and analysis, ultimately improving the performance and reliability of the entire riprap quality inspection system.

[0026] In one of the technical solutions, the spatiotemporal registration model for step S1 is as follows: Q ; This is the coordinate transformation matrix from an RGB camera to a 3D structured light camera; This is the coordinate transformation matrix from a near-infrared camera to an RGB camera; N is the total number of feature point pairs successfully matched and used to calculate the transformation matrix T in the images obtained from the RGB camera and the 3D structured light camera. Let be the coordinates of the i-th feature point in the RGB camera; Let be the coordinates of the i-th feature point in the 3D structured light camera; This indicates that the transformation matrix T is applied to the point. superior; R(T) is the regularization term of the transformation matrix. ; It is a rotation matrix that includes the pitch angle θ and the yaw angle Ф, i.e. ,in: ; ; It is a translation matrix. ; T is the homogeneous transformation matrix: ; in, ; This is for calibration error compensation. h is the regularization coefficient, with a value ranging from 0.1 to 10.

[0027] This invention further optimizes the spatiotemporal registration model by establishing precise coordinate transformation relationships to achieve spatial alignment of multi-sensor data. The spatiotemporal registration model provided by this invention includes two transformation matrices: a coordinate transformation matrix from the RGB camera to the 3D structured light camera. And the coordinate transformation matrix from near-infrared camera to RGB camera. The transformation matrix is ​​further solved using an optimization algorithm, where... The solution employs the least squares method with added regularization constraints. The regularization coefficient h ranges from 0.1 to 10, controlling the smoothness of the transformation matrix. (Regularization term) Chinese R T The superscript T denotes the matrix transpose operation, which is not the same concept as the transformation matrix T. The calculation of the transformation matrix is ​​based on the correspondence of feature points, specifically the coordinates of feature points in RGB cameras and 3D structured light cameras. and Obtained through a feature matching algorithm. The rotation matrix R(θ, Ф) is determined by the pitch angle θ and yaw angle Ф, using the basic Euler angle rotation representation. Translation matrix T trans Using homogeneous coordinates, it includes translation components t in three directions. x t y and t z Calibration error compensation term Δ z Used to compensate for systematic errors generated during camera calibration.

[0028] Feature points are detected in each camera and a correspondence is established. A feature matching algorithm is used to find corresponding feature points in the RGB camera and the 3D structured light camera. Using these feature point pairs, an iterative optimization method is employed to solve for the transformation matrix. The goal is to minimize reprojection error while maintaining the regularity of the transformation matrix. During the solution process, the regularization coefficient h is adjusted according to the actual registration accuracy requirements to ensure that the transformation matrix is ​​both accurate and stable.

[0029] For registration from a near-infrared camera to an RGB camera, the rotation matrix R(θ,Ф) and translation matrix T are used. transEstablish coordinate transformation relationships. The rotation matrix is ​​obtained by multiplying two basic rotation matrices, representing rotations around different coordinate axes. The translation matrix adopts a standard homogeneous transformation form, containing three translation parameters. Finally, a composite transformation is used to obtain the complete transformation relationship from near-infrared camera data to the RGB camera coordinate system, achieving spatial alignment of multi-source data.

[0030] The spatiotemporal registration model provided by this invention can effectively achieve accurate spatial alignment of multi-sensor data, ensuring that different modal data have a consistent spatial reference in the same coordinate system. By introducing regularization constraints, the stability and robustness of the transformation matrix solution are improved, avoiding overfitting. The addition of calibration error compensation terms further enhances registration accuracy and reduces the impact of system errors on the registration results. The feature-point-based registration method can adapt to different field environmental conditions, exhibiting good versatility and adaptability. The parameterized representation of rotation and translation matrices gives the registration process a clear physical meaning, facilitating understanding and adjustment. The use of homogeneous coordinates simplifies the calculation process of coordinate transformation and improves registration efficiency. The entire registration model provides an accurate spatial foundation for subsequent multimodal data fusion and processing.

[0031] In one of the technical solutions, step S2 forms the fused feature map F. fusion for: ; F fusion To fuse feature maps; A is the attention map. ; σ is the Sigmoid function, which compresses the output value to the range (0,1); W c、 W d、 W t and W a The weight matrix is ​​a learnable matrix; b a It is a learnable bias vector; concat is a concatenation operation along the channel dimension; This is element-wise multiplication; F c Color feature maps are extracted after high-resolution color images acquired by an RGB camera are input into a convolutional neural network (CNN). F d The depth feature map is extracted by projecting the 3D structured light point cloud data acquired by the 3D structured light camera onto the 2D imaging plane and then inputting it into the convolutional neural network (CNN). F tNear-infrared feature maps are extracted by inputting near-infrared reflection data collected by a near-infrared camera into a convolutional neural network (CNN).

[0032] This invention further optimizes the multi-source data deep fusion method, namely, an intelligent fusion method at the feature level achieved through an attention mechanism. The multimodal fusion network first extracts features from each modality of data separately. Color images acquired by an RGB camera are processed by a convolutional neural network to extract color feature maps F. c Point cloud data acquired by a 3D structured light camera is projected onto a two-dimensional plane and then used to extract depth feature maps F through a convolutional neural network. d The data collected by the near-infrared camera is used to extract near-infrared feature maps F through a convolutional neural network. t The resulting feature maps maintain the same spatial dimensions, facilitating subsequent fusion processing. Attention map generation employs a combination of concatenation and linear transformation, concatenating the three feature maps along the channel dimension, and then applying a learnable weight matrix W. a and bias vector b a A linear transformation is performed, and finally, an attention map A is generated using the sigmoid activation function. At the start of neural network training, the weight matrix W... a and bias vector b a Typically initialized randomly, the attention map is continuously adjusted during training using backpropagation and gradient descent optimizers based on the model's error on the task. Each element in the attention map has a value between 0 and 1, representing the importance weight of the corresponding feature location. The final fused feature map is achieved through element-wise multiplication, applying the attention weights to the weighted sum of the feature maps to form the final fused feature representation.

[0033] The multi-source data deep fusion method provided by this invention achieves intelligent feature selection through an attention mechanism, automatically identifying important feature regions in each modality of data. The attention map is adaptively generated based on the content of the input data, assigning different importance weights to features in different regions, making the fusion process more accurate and effective. The data-driven approach avoids the limitations of manually designed fusion rules, improving the intelligence level of feature fusion. Simultaneously, the weighted fusion of multimodal features fully utilizes the complementarity of each modality of data: color features provide rich texture information, depth features provide geometric structure information, and near-infrared features provide material property information. Furthermore, through the adjustment of learnable weights, the network can adaptively balance the contribution of each modality of features, achieving optimal fusion results in different scenarios. This fusion method significantly improves the performance of subsequent particle segmentation tasks. The attention-guided fusion mechanism also has good interpretability; the attention map intuitively displays the important regions of each modality of features, facilitating understanding of the network's decision-making process. The element-wise multiplication operation ensures the fineness of the fusion process, enabling pixel-level feature optimization. The entire fusion process adopts an end-to-end learning approach, and all parameters can be optimized through training, ensuring optimal fusion results.

[0034] In one of the technical solutions, the automatic selection of the optimal network path in step S3 is based on the complexity of the fused feature map: If G(F) fusion If ) > τ1, then path A is selected; Path A: 16 convolutional layers, each with 256 channels, and includes an attention module; If τ2≤G(F) fusion If )≤τ1, then choose path B; Path B: 8-layer convolution, 128 channels per layer; If G(F) fusion If τ < 2, then choose path C; path C: 4-layer convolution, 64 channels per layer; G(F fusion ) is a metric function for the complexity of the fused feature maps, and τ1 and τ2 are complexity threshold parameters, which are set according to the sample size.

[0035] This invention further automatically selects the optimal network processing path by evaluating the complexity of the fused feature map. First, the complexity of the fused feature map is quantified using a complexity metric function. Then, path selection is performed based on preset complexity threshold parameters τ1 and τ2. These threshold parameters are set according to the statistical characteristics of the actual sample size to ensure adaptability to different detection scenarios and requirements. Generally, based on the complexity values ​​of the fused feature maps in the samples, the high complexity threshold is taken from the 75th to 90th percentile, and the low complexity threshold is taken from the 10th to 25th percentile. This invention provides three different processing paths based on an adaptive network using neural architecture search to adapt to feature maps of different complexities. Path A, for high complexity, uses a 16-layer convolutional structure, each layer containing 256 channels, and integrates an attention module to enhance feature processing capabilities. Path B is suitable for medium complexity scenarios, using an 8-layer convolutional network with 128 channels per layer. Path C processes low complexity feature maps, using a 4-layer convolutional structure with 64 channels per layer; this hierarchical design ensures an optimal balance between processing efficiency and accuracy.

[0036] This invention optimizes the network processing path, ensuring both processing accuracy and computational efficiency. For simple feature patterns, a lightweight network is used for rapid processing, while a more robust network structure is employed for complex feature patterns to ensure processing quality. This differentiated processing approach significantly improves the overall system performance. The optimized network processing path uses complexity-driven path selection, avoiding unnecessary waste of computational resources and maximizing processing efficiency while maintaining accuracy. Furthermore, the configurable threshold parameters allow the system to adapt to different application scenarios and needs, exhibiting excellent flexibility and adaptability. It is particularly suitable for handling large-scale rockfill detection tasks, maintaining stable performance under various working conditions. The multi-path network architecture proposed in this invention also provides the ability to process multiple sample features, ensuring reliable segmentation results under various complex conditions. This reduces reliance on manual parameter tuning, improves the system's automation and practicality, and provides a reliable technical guarantee for achieving efficient and accurate particle segmentation.

[0037] In one of the technical solutions, the metric function for the complexity of the fused feature map is based on the mean absolute gradient: ; H and L are the height and width of the fused feature map, respectively, in mm; This is the gradient magnitude of the fused feature map at position (i,j).

[0038] The aforementioned technical solution further defines the specific representation method of the complexity measurement function of the fused feature map, namely, calculating the complexity of the fused feature map by means of the average absolute gradient. This method has good computational efficiency and practicality. Gradient calculation is a fundamental operation in image processing, with low computational overhead and easy implementation. The gradient magnitude-based calculation method can accurately reflect the richness of edge and texture information in the feature map, which is directly related to the difficulty of subsequent processing tasks. The larger the average absolute gradient value, the more detailed information the feature map contains, requiring a more complex network structure for processing, thus ensuring the scientific and rational nature of path selection decisions.

[0039] In one of the technical solutions, the process of extracting the gradation characteristics of the riprap and conducting quality assessment in step S5 includes: For the i-th particle, calculate its equivalent particle size: ; Calculate the gradation curve: ; Where, d i Let be the equivalent particle size of the i-th particle, in mm; X i Let be the length of the major axis of the i-th particle, in mm; Y i Let be the length of the minor axis of the i-th particle, in mm; P(d) is the percentage of particles whose mass is less than the particle size d; I(d i ≤d) is an indicator function; v i Let be the volume of the i-th particle, in mm. 3 ; K represents the total number of particles.

[0040] This invention further clarifies the gradation characteristics of riprap and the process for quality assessment. First, the three-dimensional morphological parameters of each particle are obtained, including the major axis length X. i minor axis length Y i and volume v i The equivalent particle size d is calculated based on the length values ​​of the major and minor axes. iThis reflects the actual size characteristics of the particles. Calculating the gradation curve requires statistical analysis of the volume data of all particles and cumulative calculations for particles in different size ranges. An indicator function is used to determine whether a particle belongs to a certain size range. By summing the proportion of the volume of particles meeting the criteria to the total volume, the percentage of particles smaller than a certain size is obtained. This percentage value forms a complete gradation curve, visually displaying the particle size distribution. Calculating the equivalent particle size using three-dimensional morphological parameters avoids errors caused by traditional two-dimensional measurements, significantly improving the accuracy of particle size calculation. The volume-based gradation curve calculation method can more realistically reflect the mass distribution of particles because volume parameters are directly related to the mechanical behavior and filling characteristics of particles in actual engineering, providing a more reliable data foundation for quality assessment. Furthermore, the generation process of the gradation curve is fully automated, reducing subjective errors caused by human intervention; the use of indicator functions ensures the accuracy of particle size classification, and the volumetric cumulative calculation guarantees the accuracy of the mass percentage calculation. The entire process can quickly process large amounts of particle data, greatly improving the efficiency and reliability of the testing work and meeting the efficiency requirements of engineering sites.

[0041] The number of devices and processing scale described herein are for simplification of the invention. Applications, modifications, and variations of the multimodal intelligent detection method for rockfill quality of this invention will be readily apparent to those skilled in the art.

[0042] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A multimodal intelligent detection method for the quality of riprap filling, characterized in that, include: S1. High-resolution color images, three-dimensional structured light point cloud data and near-infrared reflection data of the surface of the riprap are collected synchronously by multiple sensors, and spatial alignment and temporal synchronization of multi-source data are achieved through the established spatiotemporal registration model. S2. Input the registered multi-source data obtained in step S1 into the attention-guided multimodal fusion network, and perform deep fusion at the feature layer to form a fusion feature map; S3. Construct an adaptive network based on neural architecture search. Using the fused feature map obtained in step S2, automatically select the optimal network path for particle segmentation to obtain the two-dimensional image morphology S of the particles. i ; S4. Based on the particle segmentation results obtained in step S3, the three-dimensional model of each particle is reconstructed using multi-view geometric constraints and morphological parameters are extracted. S5. Based on the three-dimensional model of each particle reconstructed in step S4, extract the gradation characteristics of the rockfill and conduct a quality assessment. In step S4, reconstructing the three-dimensional model of each particle is achieved by minimizing the following energy function: ； Let be the energy function of the i-th particle; Energy for data items; N views The number of different camera viewpoints used to reconstruct particle i; To apply the projection matrix of the j-th camera, project the 3D model of the i-th particle onto the 2D imaging plane of the j-th camera; Q i,j The two-dimensional segmentation mask for the i-th particle in the j-th view; Weights for data items; ; ε represents the basic weight of the data item; its value ranges from 1.0 to 2.0; ε is the standard deviation of geometric measurement noise obtained through sensor calibration, in mm; ε0 is the reference noise level, in mm. For the energy of the smooth term, ; (u, v) represents an edge in the reconstructed 3D model mesh; D is the set of edges in the reconstructed 3D model; n u n v c represents the normal vectors of vertices u and v in the reconstructed 3D model. u c v , respectively, represent the curvature of vertices u and v in the reconstructed 3D model; μ is the curvature weighting factor, with a value of 0.2~0.5; β is the weight of the smoothing term. β0 is the basic weight of the smoothing term, ranging from 0.1 to 0.3; η is the adaptive coefficient of the smoothing term, ranging from 0.5 to 1.0; A i Let be the surface area of ​​the i-th particle, mm. 2 A max The maximum particle surface area, in mm 2 ; For photometric uniformity energy, ; p is the current center pixel, Ω is the set of pixels corresponding to the i-th particle in all viewpoints; q is a specific pixel in the neighborhood, N(p) is the set of pixels in the neighborhood of point p; I(p) and I(q) are the grayscale or color values ​​of pixels p and q, respectively. ρ(I p ,I q ) is a Gaussian weighting function based on the similarity between the center point p and the neighborhood point q; γ is the weight of the photometric term. γ0 is the basic weight of the photometric term, with a value ranging from 0.5 to 1.0; C i C0 is the texture richness index calculated by the image variance for the i-th particle, C0 is the texture richness threshold, and k is the photometric term adjustment coefficient, with a value of 0.5 to 3.

0.

2. The multimodal intelligent detection method for riprap filling quality as described in claim 1, characterized in that, In step S1, the multiple sensors include an RGB camera that acquires high-resolution color images, a 3D structured light camera that acquires three-dimensional structured light point cloud data, and a near-infrared camera that acquires near-infrared reflection data.

3. The multimodal intelligent detection method for riprap filling quality as described in claim 2, characterized in that, The spatiotemporal registration model for step S1 is as follows: ； This is the coordinate transformation matrix from an RGB camera to a 3D structured light camera; This is the coordinate transformation matrix from a near-infrared camera to an RGB camera; N is the total number of feature point pairs successfully matched and used to calculate the transformation matrix T in the images obtained from the RGB camera and the 3D structured light camera. Let be the coordinates of the i-th feature point in the RGB camera; Let be the coordinates of the i-th feature point in the 3D structured light camera; This indicates that the transformation matrix T is applied to the point. superior; R(T) is the regularization term of the transformation matrix, R(T) = ; It is a rotation matrix that includes the pitch angle θ and the yaw angle Ф, i.e. :,in = ； ； It is a translation matrix. ; T is the homogeneous transformation matrix: T= ； ； This is for calibration error compensation. h is the regularization coefficient, with a value ranging from 0.1 to 10.

4. The multimodal intelligent detection method for riprap filling quality as described in claim 3, characterized in that, Step S2 forms the fused feature map F fusion for: ； F fusion To fuse feature maps; A is the attention map, A= ; σ is the Sigmoid function, which compresses the output value to the range (0,1); W c、 W d、 W t and W a The weight matrix is ​​a learnable matrix; b a It is a learnable bias vector; concat is a concatenation operation along the channel dimension; This is element-wise multiplication; F c Color feature maps are extracted after high-resolution color images acquired by an RGB camera are input into a convolutional neural network (CNN). F d The depth feature map is extracted by projecting the 3D structured light point cloud data acquired by the 3D structured light camera onto the 2D imaging plane and then inputting it into the convolutional neural network (CNN). F t Near-infrared feature maps are extracted by inputting near-infrared reflection data collected by a near-infrared camera into a convolutional neural network (CNN).

5. The multimodal intelligent detection method for riprap filling quality as described in claim 4, characterized in that, The automatic selection of the optimal network path in step S3 is based on the complexity of the fused feature map: If G(F) fusion If ) > τ1, then path A is selected; Path A: 16 convolutional layers, each with 256 channels, and includes an attention module; If τ2≤G(F) fusion If )≤τ1, then choose path B; Path B: 8-layer convolution, 128 channels per layer; If G(F) fusion If τ < 2, then choose path C; path C: 4-layer convolution, 64 channels per layer; G(F fusion ) is a metric function for the complexity of the fused feature maps, and τ1 and τ2 are complexity threshold parameters, which are set according to the sample size.

6. The multimodal intelligent detection method for riprap filling quality as described in claim 5, characterized in that, The complexity metric for the fused feature map is based on the average absolute gradient. ； H and L are the height and width of the fused feature map, respectively, in mm; This is the gradient magnitude of the fused feature map at position (i,j).

7. The multimodal intelligent detection method for riprap filling quality as described in claim 6, characterized in that, Step S5 involves extracting the gradation characteristics of the riprap and conducting a quality assessment, including: For the i-th particle, calculate its equivalent particle size: ； Calculate the gradation curve: ； Where, d i Let be the equivalent particle size of the i-th particle, in mm; X i The length of the major axis of the i-th particle is in mm; Y i Let be the length of the minor axis of the i-th particle, in mm; P(d) is the percentage of particles whose mass is less than the particle size d; I(d i ≤d) is an indicator function; v i Let be the volume of the i-th particle, in mm. 3 ; K represents the total number of particles.

Images