A subgrade compaction quality evaluation method and equipment based on multi-modal double vision
By combining multimodal multiple vision technology with infrared and RGB cameras, and using multiple CNN and ANN models to predict the compaction degree of the subgrade, the problems of destructiveness, time-consuming and labor-intensive and low accuracy of subgrade compaction quality detection in existing technologies are solved, and non-destructive, real-time and low-cost compaction quality evaluation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEBEI UNIV OF TECH
- Filing Date
- 2023-06-08
- Publication Date
- 2026-05-19
AI Technical Summary
Existing methods for testing the compaction quality of roadbeds are characterized by being highly destructive, time-consuming, labor-intensive, inaccurate, and costly, making it difficult to achieve continuous, non-destructive, and efficient quality evaluation.
Multimodal diplopia technology is adopted, which combines infrared and RGB cameras to acquire roadbed images and roller parameters from multiple angles. Real-time evaluation is performed through a multimodal diplopia roadbed compaction prediction network, which includes a combination of multiple CNN and ANN models to predict roadbed compaction.
It achieves non-destructive, real-time, and low-cost evaluation of subgrade compaction quality, is applicable to different construction scenarios, improves the accuracy and robustness of the model, and reduces the impact on subgrade quality.
Smart Images
Figure CN116662931B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of roadbed engineering technology, specifically a method and equipment for evaluating roadbed compaction quality based on multimodal multiple vision, used to evaluate roadbed compaction quality based on multimodal multiple vision information. Background Technology
[0002] A well-constructed roadbed can extend the service life of roads and reduce operation and maintenance costs. Monitoring the compaction quality of the roadbed is particularly important during construction. Roadbed compaction degree is a commonly used indicator for evaluating compaction quality.
[0003] Commonly used testing methods include sand cone method, water cone method, and ring sampler method. These methods are all sampling tests, which have a high degree of randomness. The sampling process is a destructive experiment, which can affect the quality of the already constructed roadbed. The time and labor required are also drawbacks of these methods. Nuclear density gauges effectively increase the frequency of roadbed compaction sampling and are a non-destructive experiment. However, they are radioactive and require proper storage; they are not suitable for long-term use.
[0004] Intelligent compaction systems achieve non-destructive testing for compaction quality evaluation by collecting and feeding back the mechanical parameters of the roller's steel wheel. The high acquisition frequency enables continuous monitoring of the construction strip. The compaction value (CMV) is a dimensionless indirect indicator output by the intelligent compaction system. Conventional intelligent compaction systems directly determine the compaction degree of the roadbed using CMV. This method suffers from poor robustness and low accuracy due to relying on a single indicator. Chinese Patent 202111611914.8 discloses a method for identifying roadbed compaction degree based on ground-penetrating radar images; however, the high cost of the equipment used in this method is its main drawback, hindering its widespread adoption.
[0005] Therefore, this application proposes a method for evaluating the compaction quality of roadbed by combining multimodal multiple vision, which fully integrates multimodal image data and text data during the construction process to provide a more accurate evaluation of the compaction quality of roadbed. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a method and equipment for evaluating the compaction quality of roadbeds based on multimodal multiple vision.
[0007] The technical solution of the present invention to solve the aforementioned technical problem is:
[0008] In a first aspect, the present invention provides a method for evaluating the compaction quality of roadbed based on multimodal multiple vision, the method comprising the following:
[0009] A camera array is installed to continuously acquire images of the roadbed compaction. The camera array includes an infrared camera and at least two RGB cameras. The infrared camera is used to acquire a clear infrared image of the roadbed, one RGB camera is used to acquire a low-level detailed image of the roadbed taken vertically, and the remaining RGB cameras are used to acquire a high-level macro image of the roadbed taken at an angle.
[0010] The acquired infrared images are converted into infrared frequency domain images.
[0011] The image dataset is composed of low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images. The mechanical parameters of the road roller and the parameters of the subgrade filling, as well as the corresponding subgrade compaction degree, are obtained at the image acquisition locations. The mechanical parameters of the road roller and the parameters of the subgrade filling constitute the parameter dataset.
[0012] A multimodal multi-view roadbed compaction prediction network is constructed. This network includes a first network part for converting two-dimensional information into one-dimensional features and a second network part for processing multiple input one-dimensional information. The number of the first network parts is the total number of RGB cameras plus the number of infrared cameras * 2. The input to the first network parts is one of low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images. The input to the second network parts is the output of all the first network parts and features from the parameter dataset. The output of the second network parts is the roadbed compaction.
[0013] A multimodal multiview subgrade compaction prediction network was trained using image datasets, parameter datasets, and corresponding subgrade compaction values. The trained multimodal multiview subgrade compaction prediction network was then used for subgrade compaction quality evaluation.
[0014] The multimodal multiview roadbed compaction prediction network can be a multiCNN-ANN model. The multiCNN-ANN model takes multimodal, multi-view, and multi-domain images, roller mechanical parameters, and roadbed filling parameters as inputs and outputs the roadbed compaction degree. The CNN network constitutes the first network part, and multiple CNN networks are connected in parallel and then connected to the ANN network, which constitutes the second network part. The input of the ANN network also includes roller mechanical parameters and roadbed filling parameters, and the output of the ANN network is the roadbed compaction degree.
[0015] The multimodal, multi-view, and multi-domain images include low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images.
[0016] The first network part can also adopt network structures such as DenseNet or ResNet series, while the second network part can adopt network structures such as BP or XGBoost.
[0017] The infrared camera and all the RGB cameras are installed at the same height, but at different shooting angles. The infrared camera uses oblique shooting to obtain a relatively large range of infrared images.
[0018] The RGB camera has a vertical shooting height of 50-60cm, preferably 50cm, and a shooting angle of 90°; the RGB camera has an oblique shooting height of 50-60cm, preferably 50cm, and a shooting angle of 30°-60°; the infrared camera has a shooting height of 50-60cm, preferably 50cm, and a shooting angle of 30°-60°.
[0019] The mechanical parameters of the road roller include excitation force, travel speed, number of compaction passes, and vibration frequency. The parameters of the subgrade fill include P10. The number of input nodes in the second network part is the number of feature types in the parameter dataset + the total number of RGB cameras + the number of infrared cameras * 2. P10 is the mass percentage of particles larger than 10mm. Therefore, the features of the parameter dataset are excitation force, travel speed, vibration frequency, number of compaction passes, and P10, with a total of 5 feature types. Subgrade fill property parameters such as moisture content, liquid limit, and plastic limit can also be added to the feature types in the parameter dataset. The specific feature types in the parameter dataset are obtained through sensitivity analysis based on field experimental data; the essential features are excitation force, travel speed, vibration frequency, number of compaction passes, and P10.
[0020] Secondly, the present invention provides a roadbed compaction quality evaluation device based on multimodal multiple vision, the device comprising:
[0021] An infrared camera used to acquire clear infrared images of the roadbed;
[0022] An RGB camera used to acquire low-level detailed images of the roadbed taken vertically;
[0023] An RGB camera used to acquire high-level macroscopic images of the roadbed taken from a squint perspective;
[0024] A wireless transmission device used to transmit captured images back to the terminal in real time;
[0025] A multimodal complex view subgrade compaction degree prediction network is used to predict subgrade compaction degree.
[0026] The terminal is used to perform frequency domain conversion on infrared images to obtain infrared frequency domain images, and is used for training and saving of the multimodal diopter compaction prediction network.
[0027] The intelligent compaction system is used to obtain the mechanical parameters of the road roller and the parameters of the roadbed filling.
[0028] The database is used to store low-level microscopic images, high-level macroscopic images, infrared images and infrared frequency domain images, road roller mechanical parameters and subgrade filling parameters, as well as the corresponding subgrade compaction degree.
[0029] An infrared camera and at least two RGB cameras constitute a camera group, which is installed on a road roller. The database is integrated in the terminal, the intelligent compaction system is connected to the terminal, and the database is connected to a multimodal diffraction roadbed compaction degree prediction network.
[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0031] This invention presents a novel non-destructive testing method with advantages of real-time performance and robustness. It allows for continuous evaluation of compaction quality during construction without affecting the quality of already compacted roadbeds and is applicable to various construction scenarios. This invention utilizes commercially available, low-cost cameras, facilitating widespread adoption. Based on the imaging principle of insect compound eyes, this invention acquires multimodal and multi-angle image information features and performs frequency domain transformation on infrared images. It comprehensively considers image feature changes caused by variations in compaction quality during construction. The multi-parameter input method reduces the instability issues of models with single inputs, improving model accuracy and robustness, and making it applicable to various construction scenarios. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating an embodiment of the roadbed compaction quality evaluation method based on multimodal multiple vision according to the present invention.
[0033] Figure 2 This is a schematic diagram of the multimodal multiple vision subgrade compaction degree prediction network structure, which is an embodiment of the subgrade compaction quality evaluation method based on multimodal multiple vision of the present invention.
[0034] Figure 3 This is a schematic diagram of the CNN network structure in this invention.
[0035] Figure 4 This is a schematic diagram of the ANN network structure in this invention. Detailed Implementation
[0036] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention and do not limit the scope of protection of this application.
[0037] This invention presents a low-cost method for evaluating the compaction quality of roadbeds based on multimodal multiple-view imaging. "Multimodal" refers to different information sources, including RGB cameras, infrared cameras, images after frequency domain conversion, and information collected from intelligent compaction systems. Data from all RGB cameras constitute one modality. "Multimodal" refers to multi-angle shooting. This application ensures at least vertical and oblique angle shooting. The number of oblique angle shots can be one or more, with the oblique angle ranging from 30° to 60°. "Low-level" refers to a relatively small shooting range covering less image information; "high-level" refers to a relatively large shooting range covering more image information. In this embodiment, images obtained from vertically shooting RGB cameras are defined as low-level microscopic images, and images obtained from obliquely shooting RGB cameras are defined as high-level macroscopic images. Multiple high-level macroscopic images are possible.
[0038] This embodiment of the roadbed compaction quality evaluation method based on multimodal multiple vision includes:
[0039] (1) Image and parameter information acquisition and image processing;
[0040] (2) Training of multiCNN-ANN models;
[0041] (3) Save the model weight file and use the trained multiCNN-ANN model to evaluate the subgrade compaction in real time.
[0042] The specific process for image and parameter information acquisition and image processing is as follows: Before the initial construction begins, workers install the camera array bracket on the front bumper of the road roller, using strong magnets for fixation and support. The camera array is fixed to the camera bracket via a mechanical pan-tilt head, preferably 50cm above the ground. The camera array consists of three cameras: two motion cameras (Camera A and Camera B) are RGB cameras, and one infrared camera (24 frames / s). Camera A's lens is vertically downward, while the lenses of Camera B and the infrared camera are both raised at 45°. Parameter datasets are acquired through an intelligent compaction system.
[0043] The road roller begins to move, and the camera array starts working, extracting frames of images based on the roller's speed. Camera A captures low-level detailed images, camera B captures high-level macroscopic images, and the infrared camera captures infrared images. Simultaneously, the intelligent compaction system collects parameter data, including at least: excitation force, travel speed, vibration frequency, number of compaction passes, and P10.
[0044] Images captured by the camera array are transmitted back to the terminal in real time via a wireless transmission device. The terminal performs frequency domain conversion on the infrared images, specifically using Fourier transform, to obtain infrared frequency domain images. The low-resolution detailed images, high-resolution macroscopic images, infrared images, and infrared frequency domain images constitute the image dataset.
[0045] The parameter data collected by the intelligent compaction system is also transmitted back to the terminal via a wireless transmission device, forming a parameter dataset.
[0046] The multiCNN-ANN model is used to fuse multimodal, multi-domain image features with roller mechanical parameters and subgrade fill parameters to predict compaction quality. The multiCNN-ANN model consists of a multiCNN part and an ANN part, and its structure is as follows: Figure 2 As shown, multiCNN extracts deep features from each image and compresses them into one-dimensional features. The ANN part takes the image dataset as input and the one-dimensional features and parameter dataset obtained by the multiCNN part as input, and outputs the subgrade compaction degree. A collection point in the subgrade includes images collected by different cameras and infrared frequency domain images after frequency domain conversion, roller mechanical parameters, subgrade fill parameters and the corresponding subgrade compaction degree.
[0047] The specific content is as follows:
[0048] (1) The multiCNN part contains four independent and parallel CNN networks, and the structure of the CNN networks is as follows: Figure 3 As shown; one CNN network is the first network part. In this embodiment, there are four CNN networks. The inputs of the four CNN networks are low-level fine image, high-level macro image, infrared image, and infrared frequency domain image, respectively; the outputs are one-dimensional features of the four images.
[0049] (2) Each independent CNN network contains an input layer, a convolutional layer, a pooling layer, and a fully connected layer. The CNN network receives one of the above four types of images (including low-level fine images, high-level macro images, infrared images, and infrared frequency domain images), and the images are uniformly sized at 100 pixels * 100 pixels. The input images are passed through multiple convolution and pooling operations and then passed into the fully connected layer to obtain one-dimensional features, which are used as the four inputs of the ANN part for training and prediction of the ANN model.
[0050] (3) The ANN part is the second network part, which contains an ANN structure, the specific structure of which is as follows: Figure 4 As shown, it includes an input layer, a hidden layer, and an output layer. The input layer has 9 nodes, and the output layer has 1 node. The input consists of one-dimensional features of four types of images (including: one-dimensional features of low-level microscopic images, one-dimensional features of high-level macroscopic images, one-dimensional features of infrared images, and one-dimensional features of infrared frequency domain images), mechanical parameters of the road roller (including: excitation force, travel speed, vibration frequency, and number of compaction passes), and subgrade filling parameters (including: P10). The output is the subgrade compaction degree.
[0051] Training of multiCNN-ANN models:
[0052] The image dataset and parameter dataset were fed into a multiCNN-ANN model for training. The image dataset was connected to four CNN networks, and the parameter dataset was connected to an ANN network. Learning rate, number of iterations, batch size, and dropout were set for each CNN network, and learning rate, number of neurons, number of iterations, batch size, and dropout were set for the ANN network. The model hyperparameters were continuously adjusted until R... 2 The multiCNN-ANN model training is complete once the weights reach 0.8 or higher. The multiCNN-ANN weight file is then saved. In subsequent construction processes, it is unnecessary to retrain the multiCNN-ANN model; the saved weight file can be directly used for prediction to evaluate subgrade compaction.
[0053] Real-time evaluation of roadbed compaction:
[0054] Before construction begins, the terminal loads the weight file of the multiCNN-ANN model. As the road roller moves, a camera array captures images in real time and uploads them to the terminal via wireless transmission. The terminal receives the image dataset, performs frequency domain conversion on the infrared images, and generates infrared frequency domain images. The terminal's multiCNN-ANN model, inputting the image dataset and parameter dataset, predicts the roadbed compaction degree to guide construction.
[0055] The working principle and workflow of this invention are as follows:
[0056] I. Working Principle
[0057] This invention employs a multimodal compound vision method for real-time evaluation of subgrade compaction during subgrade construction. Specifically, based on the imaging principle of insect compound eyes, a camera array is installed on the front bumper of the road roller to capture multimodal, multi-angle images of subgrade compaction. An intelligent compaction system collects various mechanical parameters of the road roller and subgrade fill parameters. The collected images and parameter information are transmitted back to the terminal in real time via a wireless transmission device. The terminal performs frequency domain conversion on the infrared images to generate infrared frequency domain images. The image and parameter datasets collected during the initial construction process are used to train a multiCNN-ANN model, outputting the subgrade compaction degree for that area. The model weight file is saved after training. During subsequent daily construction, the terminal imports the model weight file and receives the image and parameter datasets transmitted in real time from the road roller. After receiving the image dataset, the terminal first performs frequency domain conversion on the infrared images, then inputs the image and parameter datasets into the multiCNN-ANN model for prediction, outputting the subgrade compaction degree for the corresponding area.
[0058] II. Workflow (see) Figure 1 )
[0059] 1. Initial construction
[0060] (1) Construction personnel install equipment, including: camera group and camera bracket, intelligent compaction system; three mechanical pan-tilt heads are 50cm off the ground, camera A lens is vertically downward, camera B lens is raised 45°, and infrared camera lens is raised 45°.
[0061] (2) As the road roller moves, the camera group and the intelligent compaction system collect images and parameter information respectively, and transmit them back to the terminal through a wireless transmission device.
[0062] (3) The terminal receives the image dataset and parameter dataset to realize the acquisition of image and parameter information; performs frequency domain conversion on the infrared image in the image dataset to generate an infrared frequency domain image, realizes the image processing process, and updates the image dataset.
[0063] (4) Train the multiCNN-ANN model using the image dataset and parameter dataset. Set the learning rate, number of iterations, batch size, and dropout for the CNN, and set the learning rate, number of neurons, number of iterations, batch size, and dropout for the ANN structure. Repeatedly adjust the hyperparameters until R... 2 Once the value reaches 0.8, save the model weight file.
[0064] 2. Routine construction
[0065] (1) Construction personnel install equipment, including: camera group and camera bracket, intelligent compaction system; three mechanical gimbals are 50cm off the ground, camera A lens is vertically downward, camera B lens is raised 45°, infrared camera lens is raised 45°; the terminal loads the trained model weight file;
[0066] (2) As the road roller moves, the camera group and intelligent compaction system collect images and parameter information, which are then transmitted back to the terminal via a wireless transmission device.
[0067] (3) The terminal receives the image dataset and parameter dataset to realize the acquisition of image and parameter information; it performs frequency domain conversion on the infrared image in the image dataset to generate an infrared frequency domain image, thus realizing the image processing process.
[0068] The updated image dataset includes all image data acquired by the three cameras and the frequency-domain converted infrared frequency domain images;
[0069] (4) Input the real-time image dataset and parameter dataset into the multiCNN-ANN model to predict the compaction degree of the roadbed in the area.
[0070] This invention uses ordinary cameras and infrared cameras to predict the compaction degree of roadbed using multimodal multiple-view multi-domain images and parameter datasets. The cameras are low-cost and have relatively low configuration requirements. The method of this invention not only monitors the mechanical parameters of the road roller, but also monitors the roadbed filling parameters and roadbed compaction images. The monitoring methods are richer and more complete, and have better application prospects.
[0071] The multimodal approach mentioned in this invention is not only for the fusion of image and text data, but also includes multimodal approaches for different perspectives (diplopia) and different types of images (ordinary RGB and infrared images). It can collect image information features from different scales and multiple sources, and has higher accuracy and robustness compared to a single image.
[0072] Any aspects not covered in this invention are applicable to existing technologies.
Claims
1. A method for evaluating the compaction quality of roadbed based on multimodal multiple vision, characterized in that, The method Includes the following: A camera array is installed to continuously acquire images of the roadbed compaction. The camera array includes an infrared camera and at least two RGB cameras. The infrared camera is used to acquire a clear infrared image of the roadbed, one RGB camera is used to acquire a low-level detailed image of the roadbed taken vertically, and the remaining RGB cameras are used to acquire a high-level macro image of the roadbed taken at an angle. The acquired infrared images are converted into infrared frequency domain images. The image dataset is composed of low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images. The mechanical parameters of the road roller and the parameters of the subgrade filling, as well as the corresponding subgrade compaction degree, are obtained at the image acquisition locations. The mechanical parameters of the road roller and the parameters of the subgrade filling constitute the parameter dataset. A multimodal multi-view roadbed compaction prediction network is constructed. This network includes a first network part for converting two-dimensional information into one-dimensional features and a second network part for processing multiple input one-dimensional information. The number of the first network parts is the total number of RGB cameras plus the number of infrared cameras * 2. The input to the first network parts is one of low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images. The input to the second network parts is the output of all the first network parts and features from the parameter dataset. The output of the second network parts is the roadbed compaction. A multimodal multiview subgrade compaction prediction network was trained using image datasets, parameter datasets, and corresponding subgrade compaction values. The trained multimodal multiview subgrade compaction prediction network was then used for subgrade compaction quality evaluation.
2. The method for evaluating the compaction quality of roadbed based on multimodal multiple vision according to claim 1, characterized in that, The multimodal multiview roadbed compaction prediction network is a multiCNN-ANN model. The multiCNN-ANN model takes multimodal, multi-view, and multi-domain images, roller mechanical parameters, and roadbed filling parameters as inputs and outputs the roadbed compaction degree. The CNN network constitutes the first network part, and multiple CNN networks are connected in parallel and then connected to the ANN network, which constitutes the second network part. The input of the ANN network also includes roller mechanical parameters and roadbed filling parameters, and the output of the ANN network is the roadbed compaction degree. The multimodal, multi-view, and multi-domain images include low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images.
3. The method for evaluating the compaction quality of roadbed based on multimodal multiple vision according to claim 1, characterized in that, The first network part adopts the DenseNet or ResNet series network structure, and the second network part adopts the BP or XGBoost network structure.
4. The method for evaluating the compaction quality of roadbed based on multimodal multiple vision according to claim 1, characterized in that, The infrared camera and all the RGB cameras are installed at the same height, but at different shooting angles; the infrared camera uses oblique shooting.
5. The method for evaluating the compaction quality of roadbed based on multimodal multiple vision according to claim 1, characterized in that, The RGB camera has a vertical shooting height of 50-60cm and a shooting angle of 90°; the RGB camera has an oblique shooting height of 50-60cm and a shooting angle of 30°-60°; the infrared camera has a shooting height of 50-60cm and a shooting angle of 30°-60°.
6. The method for evaluating the compaction quality of roadbed based on multimodal multiple vision according to claim 1, characterized in that, The mechanical parameters of the road roller include excitation force, travel speed, number of compaction passes and vibration frequency. The subgrade filling parameters include P10. The number of input nodes in the second network part is the number of feature types in the parameter dataset + the total number of RGB cameras + the number of infrared cameras * 2.
7. A roadbed compaction quality evaluation equipment based on multimodal multiple vision, characterized in that, The equipment includes: An infrared camera used to acquire clear infrared images of the roadbed; An RGB camera used to acquire low-level detailed images of the roadbed taken vertically; An RGB camera used to acquire high-level macroscopic images of the roadbed taken from a squint perspective; A wireless transmission device used to transmit captured images back to the terminal in real time; A multimodal complex view subgrade compaction degree prediction network is used to predict subgrade compaction degree. The terminal is used to perform frequency domain conversion on infrared images to obtain infrared frequency domain images, and is used for training and saving of the multimodal diopter compaction prediction network. The intelligent compaction system is used to obtain the mechanical parameters of the road roller and the parameters of the roadbed filling. The database is used to store low-level microscopic images, high-level macroscopic images, infrared images and infrared frequency domain images, road roller mechanical parameters and subgrade filling parameters, as well as the corresponding subgrade compaction degree. An infrared camera and at least two RGB cameras constitute a camera group, which is mounted on a road roller. The database is integrated into a terminal, and the intelligent compaction system is connected to the terminal. The database is connected to a multimodal, multi-view roadbed compaction degree prediction network. The multimodal, multi-view roadbed compaction degree prediction network includes a first network part for converting two-dimensional information into one-dimensional features and a second network part for processing one-dimensional information from multiple inputs. The number of the first network parts is the total number of RGB cameras plus the number of infrared cameras * 2. The input of the first network part is one of low-level microscopic images, high-level macroscopic images, infrared images, and infrared frequency domain images. The input of the second network part is the output of all the first network parts and the features in the parameter dataset. The output of the second network part is the roadbed compaction degree.