A method and system for visual detection of early road damage based on feature collaboration
Through the coordinated analysis of roadbed and road surface images collected by ground penetrating radar and industrial cameras in road disease detection, the image correspondence relationship is maintained by using encoder and GPS system, image alignment and feature fusion are carried out, and the roadbed and road surface in the existing technology are solved. The lack of synergy problem of image analysis improves the sensitivity and robustness of detection and ensures the safety and reliability of the road.
Patent Information
- Application Number
- CN202410260827.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-07
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-03-07
AI Technical Summary
Existing road disease detection technology is difficult to achieve coordinated analysis of roadbed and road surface images, resulting in low comprehensive utilization of detection data and difficult to maintain detection performance under different environmental factors. Early diseases are not obvious on the road surface but problems in the roadbed are difficult. Discover in time.
The ground penetration radar and industrial cameras collect the pathway disease images, use encoder, GPS system and significance detection algorithm to maintain the space-time correspondence of the images, image alignment and feature fusion, and use a shared encoding network to extract the pathway and road surface features. Through interactive adaptive related operations, the interactive coordination and information complementarity between the roadbed and the road surface image features are achieved.
It improves the sensitivity and robustness of early road disease detection, can identify potential problems earlier, extend road life, reduce maintenance costs, ensure road safety and reliability, and enhances the comprehensive performance of the detection system in complex environments.
Smart Images

Figure CN118038384B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual detection of early road damage, and in particular to a method and system for visual detection of early road damage based on feature collaboration. Background Art
[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.
[0003] Road damage detection has become the core business of highway management departments, but the existing detection theories and key technologies are still difficult to meet the needs of refined detection. In response to the problems of high cost, low efficiency, poor safety, and strong subjectivity of traditional manual detection, researchers have proposed a variety of non-destructive testing methods, including ultrasonic testing, laser testing, ground penetrating radar testing, infrared testing, etc. Due to its advantages in visualization and traceability, road damage detection based on computer vision has made certain progress in both theoretical research and practical application.
[0004] However, current early road disease detection and research are mainly focused on visual inspection of the road surface. However, many diseases will not be directly displayed on the road surface. The cracks and deformations formed in the early stage are more hidden inside the roadbed, which causes these potential problems to seriously affect the structural health of the road before they appear on the road surface. For example, in the scenario of highway reconstruction and expansion, the consolidation degree of the foundation under the new and old roadbeds is different. Affected by factors such as the dead weight of the roadbed filler, it is inevitable that the new and old roadbeds will settle differently, causing cracks in the road surface structure. These potential diseases are difficult to monitor and identify through conventional road surface images. Therefore, it is necessary to combine the roadbed ground penetrating radar images and the road surface industrial camera images to comprehensively and accurately evaluate the road disease condition.
[0005] In the current road disease visual detection system, the roadbed and pavement image analysis systems are used to complete the detection tasks of roadbed and pavement diseases respectively, but the corresponding image acquisition and analysis systems lack coordination and interaction, and the detection tasks are mostly carried out independently. Most of the existing road disease visual detection systems analyze roadbed and pavement disease images separately, with low comprehensive data utilization rate, and it is difficult to achieve information complementarity of different modal data, resulting in large differences in the detection performance of existing detection technologies under different environmental factors. The detection model fragmentation problem is serious, and it is difficult to explore the correlation between different detection tasks in terms of feature morphology, occurrence mechanism, etc., resulting in the need to improve the comprehensive performance of the detection system. How to fully explore the complementary attributes between roadbed and pavement images and make full use of their advantages is an effective way to solve the above problems.
[0006] Due to various factors, the position of road defects on the roadbed and pavement may deviate. For example, foundation problems under the roadbed, groundwater levels, climate change, etc. may cause uneven sinking or uplift of the pavement and roadbed, which may cause the position of road defects on the roadbed and pavement to deviate. How to perform image preprocessing to ensure the smooth fusion of roadbed image and pavement image features is also a key issue in road defect detection.
[0007] The invention patent with application number 201710430770.3 and the invention name "A multifunctional road CT comprehensive inspection vehicle" provides a multifunctional road CT comprehensive inspection vehicle that can integrate road surface visual inspection, laser inspection, ground penetrating radar inspection, illumination inspection, and friction coefficient inspection to achieve non-destructive, efficient, and continuous inspection. However, this road disease detection method does not comprehensively consider the conditions of the road surface and the interior of the road, and lacks coordination and interaction between data. Summary of the invention
[0008] In order to solve the shortcomings of the prior art, the present invention provides a method and system for visual detection of early road diseases based on feature collaboration; the method uses ground penetrating radar and industrial camera to respectively collect roadbed and pavement disease images, and through encoders, GPS systems and saliency detection algorithms, the images collected by the ground penetrating radar and the industrial camera maintain a temporal and spatial correspondence; image alignment technology is used to ensure that road diseases in roadbed images and pavement images are in the same position in the images; a shared coding network is used to simultaneously extract roadbed and pavement disease image features, and the roadbed features and pavement features are collaboratively used through interactive adaptive correlation operations to achieve interactive collaboration and information complementarity of roadbed and pavement image features, and obtain complementary fusion features; the fusion features are input into a feature decoding network, and the road disease detection results are output.
[0009] On the one hand, a visual detection method for early road damage based on feature collaboration is provided;
[0010] A method for visual detection of early road damage based on feature collaboration, comprising:
[0011] Acquire a road surface image to be detected and a roadbed image to be detected; the roadbed image to be detected is a roadbed image at a position where the road surface image to be detected is collected;
[0012] Performing image alignment processing on the road surface image to be detected and the roadbed image to be detected to ensure that the spatial positions of the road defects in the road surface image to be detected and the roadbed image to be detected are corresponding;
[0013] Input the aligned road surface image to be detected and the aligned roadbed image to be detected into the trained road early disease visual detection model, and output the road early disease classification label;
[0014] Among them, the trained road early disease visual detection model is used to extract features from the road surface image to be detected and the roadbed image to be detected respectively, and the extracted road surface features and roadbed features are fused to obtain fused features; the fused features are classified to obtain classification labels.
[0015] On the other hand, a visual detection system for early road damage based on feature collaboration is provided;
[0016] A road early disease visual detection system based on feature collaboration, comprising:
[0017] An acquisition module is configured to: acquire a road surface image to be detected and a roadbed image to be detected; the roadbed image to be detected is a roadbed image at a position where the road surface image to be detected is collected;
[0018] An alignment module is configured to: perform image alignment processing on the road surface image to be detected and the roadbed image to be detected to ensure that the spatial positions of the road defects in the road surface image to be detected and the roadbed image to be detected are corresponding;
[0019] An output module is configured to: input the aligned road surface image to be detected and the aligned roadbed image to be detected into the trained road early disease visual detection model, and output a road early disease classification label;
[0020] Among them, the trained road early disease visual detection model is used to extract features from the road surface image to be detected and the roadbed image to be detected respectively, and the extracted road surface features and roadbed features are fused to obtain fused features; the fused features are classified to obtain classification labels.
[0021] On the other hand, there is also provided an electronic device, comprising:
[0022] a memory for non-transitory storage of computer-readable instructions; and
[0023] a processor for executing the computer readable instructions,
[0024] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.
[0025] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.
[0026] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.
[0027] The above technical solution has the following advantages or beneficial effects:
[0028] (1) Many early road diseases are not obvious on the road surface, but will first appear in the roadbed. If the problems in the roadbed are not discovered and solved in time, it may lead to more serious consequences, such as roadbed subsidence, potholes on the road surface, and may even cause traffic accidents. The roadbed and pavement image feature fusion method proposed in the present invention can increase the sensitivity and robustness of road disease detection, which is of great significance and value for solving the problem that early road diseases are not obvious on the road surface. Such fusion analysis can help road maintenance personnel identify and solve potential problems earlier, help to take preventive maintenance measures in advance, extend the life of the road, reduce maintenance costs, and ensure the safety and reliability of the road.
[0029] (2) The present invention proposes for the first time the idea of identifying road defects by fusing the features of the road surface RGB image and the time-space corresponding roadbed ground penetrating radar image, and provides a possible specific implementation method.
[0030] (3) The present invention proposes a complete set of image preprocessing methods to address the situation that may occur during the image feature fusion process, that is, the situation where the positions of road defects on the roadbed and the road surface do not correspond. First, a saliency detection algorithm is used to crop the continuous ground penetrating radar image to obtain a roadbed image, and the image can be guaranteed to contain the same road defects as the road surface image. Then, through the image alignment method, the roadbed and road surface images containing the same defect are made as consistent as possible (the image size is consistent and the location of the defect in the image is consistent). BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0032] Figure 1 It is a schematic diagram of alignment of a roadbed image and a road surface image in Embodiment 1;
[0033] Figure 2 This is a flow chart of a shared coding network according to Embodiment 1;
[0034] Figure 3 Schematic diagram of the internal structure of VGG16 in Example 1. DETAILED DESCRIPTION
[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0036] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0037] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.
[0038] Terminology explanation:
[0039] 1. Image alignment: Image alignment refers to geometrically transforming different images or images of the same scene from different perspectives so that they are aligned or aligned in space. Image alignment is commonly used in the fields of computer vision, image processing, and pattern recognition to solve problems such as object recognition, target tracking, image stitching, and stereo matching.
[0040] 2. Salient Object Detection: Salient Object Detection is a technology in the field of computer vision and image processing that aims to identify the most attractive or salient areas in an image. In an image, a salient object usually refers to an area that is significantly different from the background or attracts attention, such as a face, an object, or an area with a significant difference in texture from the surrounding environment.
[0041] 3. Siamese network: Siamese network is a dual-branch (or dual-network) neural network structure. Siamese network mainly consists of two identical sub-networks that share the same weights and structure. Its purpose is to compare the similarity of inputs by mapping two different inputs into a shared feature space.
[0042] 4. VGG-16: VGG-16 is a deep convolutional neural network developed by the Visual Geometry Group at the University of Oxford. The VGG-16 model consists of 16 convolutional layers and fully connected layers, which can learn very complex feature representations and can be fine-tuned and transferred in many different computer vision tasks.
[0043] 5. Shared coding network: Shared coding network is usually used to process multiple input data, and the parameters of its encoding part are shared, so that multiple different types of input data can be mapped to the same feature space. This method can effectively learn the common features of each type of data, so that different types of data can be directly compared and jointly learned in the shared feature space.
[0044] 6. Feature decoding network: Feature decoding network is usually used to decode high-dimensional feature maps into visualization or classification results. For road damage recognition tasks, the fused features are input into a decoding module for classification to obtain the prediction results of road damage categories. This structure is called a feature decoding network, which decodes high-dimensional fused features into prediction results of road damage categories.
[0045] The present invention provides a method for visual detection of early road diseases based on feature collaboration, which helps to improve the accurate identification of early road diseases by collaboratively using the features of roadbed and pavement disease images. The present invention is suitable for early road diseases that are not very obvious on the pavement but more obvious on the roadbed, and comprehensively considers the conditions of the diseases on the road surface and inside the road. The method uses an encoder to control the imaging frequency of an industrial camera, and uses a GPS system to record the position of the collected image. The ground penetrating radar image is cropped by a saliency detection algorithm to obtain the roadbed image of the corresponding position, so that the roadbed image collected by the ground penetrating radar and the pavement image collected by the industrial camera maintain a synchronous corresponding relationship. Image alignment related technology is used to ensure that the road diseases maintain spatial position correspondence on the roadbed and pavement images.
[0046] The shared coding network is used to extract the image features of roadbed and pavement diseases at the same time, and the image features of roadbed and pavement diseases are used collaboratively through interactive adaptive correlation operations to obtain fusion features with stronger complementary capabilities. The fusion features are used as input and combined with the labeled road image dataset to train a model that can identify road diseases. The collaborative use of roadbed and pavement disease image features helps to explore the correlation between roadbed diseases and pavement diseases in terms of characteristic morphology and occurrence mechanism, and realize the feature interaction between roadbed and pavement images. Comprehensively considering the road surface condition and the internal condition of the road can improve the detection capability of the road early disease visual detection system in complex operating environments, providing an important reference for road maintenance and repair.
[0047] Embodiment 1
[0048] This embodiment provides a method for visual detection of early road damage based on feature collaboration;
[0049] A method for visual detection of early road damage based on feature collaboration, comprising:
[0050] S101: Acquire a road surface image to be detected and a roadbed image to be detected; the roadbed image to be detected is a roadbed image at a position where the road surface image to be detected is collected;
[0051] S102: performing image alignment processing on the road surface image to be detected and the roadbed image to be detected to ensure that the spatial positions of the road defects in the road surface image to be detected and the roadbed image to be detected are corresponding;
[0052] S103: inputting the aligned road surface image to be detected and the aligned roadbed image to be detected into the trained road early disease visual detection model, and outputting a road early disease classification label;
[0053] Among them, the trained road early disease visual detection model is used to extract features from the road surface image to be detected and the roadbed image to be detected respectively, and the extracted road surface features and roadbed features are fused to obtain fused features; the fused features are classified to obtain classification labels.
[0054] Furthermore, the S101: obtains the image of the road surface to be detected and the image of the roadbed to be detected, wherein the image of the road surface to be detected is collected by an industrial camera, and the industrial camera is controlled by a rotary encoder. While the image of the road surface to be detected is collected by the industrial camera, a GPS system is used to record the location where the image of the road surface to be detected is collected, and at the location where the image of the road surface to be detected is collected, a three-dimensional ground penetrating radar is used to collect the image of the roadbed to be detected.
[0055] It should be understood that the present invention synchronously collects images of the roadbed and pavement. The rotary encoder sends a control signal to the industrial camera. After receiving the signal sent by the encoder, the industrial camera collects the road surface image and records the collection position through the GPS system. The position recorded by the GPS system and the significance detection are used to crop the ground penetrating radar image to ensure that the roadbed image contains the road disease corresponding to the pavement image when the roadbed and pavement disease may not completely correspond to the disease position.
[0056] Furthermore, before the S102: performing image alignment processing on the road surface image to be detected and the roadbed image to be detected, the process further includes:
[0057] The roadbed image to be detected is processed by using a saliency detection algorithm, the area with a saliency greater than a set threshold is retained, and the area with a saliency less than or equal to the set threshold is deleted to obtain a saliency area;
[0058] The center point of the salient area is used as the center point of the image to be cropped, and the cropping size of the image to be cropped is set;
[0059] The image to be cropped is cropped from the roadbed image to be detected according to the cropping size to obtain an updated roadbed image.
[0060] It should be understood that, considering that the locations of roadbed and pavement defects may not completely correspond, in order to facilitate subsequent feature fusion, the roadbed and pavement images are aligned before image feature extraction to ensure that the spatial locations of road defects in the roadbed and pavement images remain corresponding.
[0061] Exemplarily, an industrial camera, a three-dimensional ground penetrating radar and a GPS system are all installed on a mobile detection carrier. The industrial camera is triggered by a rotary encoder arranged on the vehicle body.
[0062] When the mobile detection carrier moves forward, the rotary encoder triggers a pulse at every fixed distance (such as 5mm), and the pulse signal sent triggers the industrial camera to collect road images. The industrial camera obtains a picture with W1×H1 pixels. When collecting road images, the GPS system is used to determine the location information of the image.
[0063] Starting from the industrial camera collecting road surface images, the 3D ground penetrating radar starts to collect roadbed images at the same time and records the collected track position information. The roadbed images collected by the ground penetrating radar on the measuring track are continuous. Among them, the roadbed image required for image feature fusion is the horizontal plane image generated by the C scan obtained by the 3D ground penetrating radar.
[0064] The RGB images of the road surface with road diseases are screened, and the roadbed images at the corresponding positions are found in the continuous images obtained by the ground penetrating radar according to the GPS location information when the encoder triggers the industrial camera to collect the road surface images. It is hoped that the corresponding roadbed images of similar sizes can be obtained through cropping operations for subsequent image fusion.
[0065] Due to various factors, the location of road defects on the roadbed and pavement may deviate. For example, foundation problems under the roadbed, groundwater levels, climate change, etc. may cause uneven sinking or uplift of the pavement and roadbed, which may cause the location of road defects on the roadbed and pavement to deviate.
[0066] Specifically, the defects (such as cracks) at the same position on the roadbed and pavement may not be on the same vertical line, but have a certain deviation distance. For example, if a vertical line is drawn along the shape of the pavement defect and mapped onto the roadbed, the resulting mapping cannot completely overlap with the actual defect on the roadbed, so it is necessary to process the roadbed and pavement images for subsequent image feature fusion.
[0067] Taking into account the positional deviation of the defects, an image of the same size as the pavement image cannot be captured from the roadbed image, because this may result in the road defects contained in the pavement image not being able to find the corresponding road defects in the roadbed image of the same size. Therefore, a larger roadbed image should be captured at the corresponding position to ensure that the road defects captured in the pavement image can be found in the roadbed image.
[0068] According to the GPS location information, in order to ensure that the road defects captured in the pavement image can be found in the roadbed image, it is necessary to crop the roadbed image containing the road defects in the ground penetrating radar image at the corresponding location (within a larger range). The saliency detection algorithm can help identify the most attractive or meaningful areas in the image, which can be used to crop the roadbed image to ensure that the image contains road defects.
[0069] Select a texture-based saliency detection algorithm to perform saliency detection on the ground penetrating radar image. According to the area with large texture changes, determine the saliency value of the road disease area.
[0070] According to the significance value, a threshold can be set to determine the significant area, and an appropriate threshold can be selected to determine the significance boundary to distinguish which areas contain road diseases.
[0071] Cropping image: Based on the set threshold, the ground penetrating radar roadbed image of W2×H2 pixels containing road defects is cropped (W2>W1, H2>H1). The image is cropped by performing saliency detection on the roadbed image to ensure that the roadbed image containing road defects can be captured relatively accurately even if there is a positional deviation of the road defects, which is conducive to the subsequent image alignment operation to further ensure the correspondence between the roadbed and road surface images.
[0072] Furthermore, the S102: performing image alignment processing on the road surface image to be detected and the roadbed image to be detected includes:
[0073] Perform image edge detection on the road surface image to be detected to obtain the edge points of the road surface image;
[0074] Performing image edge detection on the updated roadbed image to obtain the roadbed image edge points;
[0075] For each pair of edge points, calculate the Euclidean distance between the edge point of the updated roadbed image and the edge point of the road surface image to be detected;
[0076] For each roadbed image edge point b i (x i ,y i ), through the Euclidean distance D ij Find the nearest edge point a of the road surface image to be detected j (x j ,y j ), Euclidean distance D ij Calculated as:
[0077]
[0078] By minimizing the sum of the Euclidean distances between all corresponding edge points, image alignment processing is performed on the updated roadbed image; the image alignment processing includes: image scaling, rotation and translation;
[0079] The roadbed image after image alignment is cropped again to obtain an optimized roadbed image, so that the size of the optimized roadbed image is consistent with that of the road surface image to be detected.
[0080] For example, Figure 1 As shown, the details of image alignment processing of the road surface image to be detected and the roadbed image to be detected are as follows:
[0081] Due to various factors, the location of road defects on the roadbed and road surface may deviate. For example, foundation problems under the roadbed, groundwater levels, climate change, etc. may cause uneven sinking or uplift of the road surface and roadbed, which may cause the location of road defects on the roadbed and road to deviate.
[0082] Specifically, the defects (such as cracks) at the same position on the roadbed and pavement may not be on the same vertical line, but have a certain deviation distance. For example, the roadbed defect may appear on one side, while the pavement defect may appear on the other side.
[0083] Although the operation of cropping the roadbed image using salient object detection has taken this issue into account, the cropped image only ensures that the roadbed image contains the corresponding road damage, but still cannot ensure that the road damage in the roadbed and pavement image is in the same position in the image, and the image size is still different. Specifically, the road damage may be located in the upper left corner of the pavement image captured by the industrial camera, but in the lower right corner of the roadbed image after preliminary processing.
[0084] This situation will cause certain difficulties for the automated road disease detection and identification system, because if the positions of road diseases on the roadbed and road surface do not correspond, additional processing and alignment are required to ensure the accuracy of fusion. In order to ensure that the positions of road diseases in the roadbed image and the road surface image can correspond, and facilitate the subsequent image feature fusion, geometric transformation is now used to achieve the alignment of the roadbed image and the road surface image.
[0085] Match the key information (such as edges and textures) in the roadbed and pavement images, and calculate the Euclidean distance, cosine similarity, etc. between the matching point pairs.
[0086] Using the key information of the matching, the geometric transformation between images can be estimated, and the sum of the Euclidean distances between all corresponding edge points can be minimized according to the least squares fitting method to achieve image alignment of the updated roadbed image. Image alignment can include operations such as rotation and translation.
[0087] Rotation: Calculate the required rotation angle based on the positional relationship of the edge points, and then perform the rotation operation.
[0088] Translation: Calculate the required translation distance based on the positional relationship of the edge points, and then perform the translation operation.
[0089] Determine the translation, rotation and scaling parameters, and adjust the roadbed image through these transformations to make the position and size of the road disease in the roadbed image as consistent as possible with the road surface image, and ensure that it is aligned with the road surface RGB image as much as possible under a certain metric. Finally, according to the position and size of the aligned roadbed image, the roadbed image is cropped again to make it exactly the same size as the road surface image.
[0090] Furthermore, in S103, the aligned road surface image to be detected and the aligned roadbed image to be detected are input into the trained road early disease visual detection model, and the road early disease classification label is output, wherein the training process of the trained road early disease visual detection model includes:
[0091] Construct a data set, and divide the data set into: a training set, a validation set, and a test set according to proportion;
[0092] The data set includes: pavement images and roadbed images with known pavement disease labels;
[0093] The training set is input into the road early disease visual detection model for training. When the loss function value of the model no longer decreases, or the number of iterations exceeds the set number, the training is stopped to obtain the trained road early disease visual detection model.
[0094] Inputting the validation set into the trained road early disease visual detection model for verification to verify whether the index value of the model reaches the set threshold;
[0095] The test set is input into the trained road early disease visual detection model for testing, and the model whose test accuracy meets the set threshold is output as the final trained model.
[0096] Preferably, the training details of the trained road early disease visual detection model are as follows:
[0097] Prepare a labeled road image dataset to ensure that each image is labeled with the correct road damage category label (such as cracks, potholes, etc.).
[0098] The dataset is divided into training set, validation set and test set, 70% of the data is used for training, 10% of the data is used for validation, and 20% of the data is used for testing.
[0099] Use the data and labels of the training set, input fusion features and labels for model training.
[0100] Choose to use the common deep learning framework TensorFlow to build and train the model. Define the model architecture, select the appropriate loss function and optimizer, and train the model.
[0101] During the model training process, the model's hyperparameters can be adjusted through techniques such as cross-validation and hyperparameter search to achieve better performance.
[0102] The trained model is evaluated using the data from the validation set to measure the model’s accuracy, precision, recall and other indicators in the road damage identification task.
[0103] Tune the model based on the evaluation results, such as adjusting the model architecture and optimizing hyperparameters.
[0104] Finally, the final model is evaluated using the test set to ensure that the performance of the model meets expectations.
[0105] When the model performs well, it is deployed in an actual road monitoring system for automated road damage identification tasks.
[0106] It should be understood that based on the roadbed horizontal plane image generated by the three-dimensional scanning of the three-dimensional ground penetrating radar and the time-space corresponding road surface image taken by the industrial camera, a shared encoding network is trained to simultaneously extract the roadbed image features and the road surface image features and keep them in spatial correspondence, thereby realizing cross-modal complementary information integration and obtaining fusion features with complementary expression capabilities.
[0107] Define the correct road damage labels for the preprocessed road image dataset, input the obtained fusion features and the collected labeled road damage data, and train the early road damage visual detection model. Tune the model to ensure its accurate recognition of road damage. Use the trained model to predict and detect new road data. Apply the model to identify and locate road damage.
[0108] Furthermore, the trained road early disease visual detection model has a network structure including:
[0109] The shared encoding network and feature decoding network are connected in sequence;
[0110] The feature decoding network includes a decoding module and a classifier connected in sequence, wherein the decoding module includes: a first fully connected layer and a convolutional layer connected in sequence; the classifier includes: a second fully connected layer and a Softmax activation function layer connected in sequence; the convolutional layer is connected to the second fully connected layer.
[0111] The feature vector fused by the shared coding network is used as the input of the feature decoding network and decoded by the decoding module of the feature decoding network to obtain the feature representation of the road disease category; the feature representation is input into the classifier of the feature decoding network for category prediction.
[0112] The decoding module uses the fully connected layer and convolutional layer provided by TensorFlow for feature mapping, which is used to map the high-dimensional image features back to the original data space to retain information useful for predicting road damage categories during the decoding process; the classifier is connected to a fully connected layer and a Softmax activation function after the decoding module to map the decoded feature representation to a specific road damage category.
[0113] Furthermore, the shared feature encoding network includes: a first deep convolutional neural network and a second deep convolutional neural network;
[0114] The input end of the first deep convolutional neural network is used to input the road surface image to be detected, and the output end of the first deep convolutional neural network outputs a feature map of the road surface image to be detected;
[0115] The input end of the second deep convolutional neural network is used to input the optimized roadbed image, and the output end of the second deep convolutional neural network outputs a feature map of the optimized roadbed image;
[0116] The output end of the first deep convolutional neural network and the output end of the second deep convolutional neural network are both connected to the input end of the feature fusion network;
[0117] The feature fusion network splices the feature map of the road surface image to be detected with the feature map of the optimized roadbed image to obtain a first splicing result, and inputs the first splicing result into the first convolution layer for convolution operation to obtain a first convolution operation result;
[0118] Performing RG correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain a first interactive correlation feature; performing GR correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain a second interactive correlation feature; splicing the first interactive correlation feature and the second interactive correlation feature to obtain a second splicing result, and inputting the second splicing result into the second convolution layer to perform convolution operation to obtain a second convolution operation result;
[0119] The first convolution operation result is concatenated with the second convolution operation result to obtain a fusion feature;
[0120] The fused features are input into the feature decoding network and the classification results are output.
[0121] Furthermore, the first deep convolutional neural network and the second deep convolutional neural network are both implemented using a VGG-16 network.
[0122] Furthermore, the feature extraction of the road surface image to be detected and the roadbed image to be detected is performed respectively by a first deep convolutional neural network and a second deep convolutional neural network, wherein the first deep convolutional neural network performs feature extraction on the road surface image to be detected; and the second deep convolutional neural network performs feature extraction on the roadbed image to be detected.
[0123] Furthermore, the extracted pavement features and roadbed features are subjected to feature fusion to obtain fused features, which is achieved through a feature fusion network. The feature fusion can be achieved by serial fusion, parallel fusion or weighted fusion.
[0124] Furthermore, the fused features are classified to obtain classification labels, which is achieved through a feature decoding network.
[0125] like Figure 2 As shown in the figure, a feature fusion encoding network is proposed. The network consists of two components: a Siamese network for extracting features and a feature fusion module.
[0126] A Siamese network structure (with shared weights) is used as the encoder network to extract features from road surface RGB images and roadbed radar images.
[0127] The encoder is implemented by VGG-16. Since the input channel of the VGG-16 network is three-channel, the single-channel ground penetrating radar image is converted into a three-channel image.
[0128] like Figure 3 As shown in the figure, the first 13 convolutional layers of VGG-16 are used to accurately extract the features of the image and form a large number of image feature parameters.
[0129] Each convolutional layer network can obtain RGB image features and ground penetrating radar image features. The RGB image features are expressed as The ground penetrating radar (GPR) image feature is expressed as
[0130] In VGG-16, the convolutional layers use a smaller kernel size (usually 3x3) and a smaller stride (usually 1), which enables the convolution operation to capture the spatial relationship between adjacent pixels in the input image.
[0131] VGG-16 uses pooling operations (such as maximum pooling) in the convolutional layer. The pooling operation can make the image features have a certain spatial continuity in space.
[0132] Perform feature fusion and and It is connected to a convolutional layer with a 3×3 kernel size and 512 channels, and interactively adaptive correlation operations are performed simultaneously.
[0133] It is processed by two correlation operations, namely "RG Correlation" and "GR Correlation". Here, the correlation operation usually refers to the correlation operation, which can calculate the correlation between two feature maps.
[0134] When calculating the correlation score between two feature maps, a cross-correlation operation is used to measure the similarity or correlation between them. The cross-correlation operation actually slides a window on one feature map and then calculates the sum of the products of the pixel values of the window and the corresponding area of the other feature map.
[0135] Further, the step of performing R-Gcorrelation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain the first interactive correlation feature; and performing GR correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain the second interactive correlation feature specifically includes:
[0136] For RG correlation processing, the road surface image feature map As a filter, then in the roadbed image feature map Perform a sliding window operation on the window. During the sliding window process, at each position in the window, calculate and The correlation score between them is obtained to obtain a new feature map. Each pixel value in the new feature map represents the value at the corresponding position. and The correlation score between .
[0137] Two feature maps and The size of is N×M. Specifically, for the pixel at each position (i, j) of the new feature map, the RG correlation score is calculated using the following formula:
[0138]
[0139] Among them, C R-G (i,j) represents the relevance score at position (i,j), Representation feature map The pixel value at position (i+m,j+n), Representation feature map The pixel value at position (i,j).
[0140] By calculating the correlation score, a new feature map is obtained, in which each pixel value is represented at the corresponding position. and The correlation score between R-G (i,j).
[0141] Similarly, the GR correlation operation is to convert As a filter, Perform the same operation on and The correlation between them.
[0142] By performing these two interactive correlation operations, the model can learn and and obtain a more comprehensive and rich feature representation.
[0143] Both RGB features and radar image features can undergo these two interactive correlation operations to avoid the loss of relevant features. The interactively correlated features are then concatenated together and adaptively integrated through a convolutional layer with a kernel size of 3×3 and 512 channels.
[0144] Finally, the features obtained by the concatenated convolution operation and the correlated convolution operation are concatenated in parallel to generate complementary features (MC), i.e., f MC Complementary feature ratio and Have stronger complementary expression ability.
[0145] Embodiment 2
[0146] This embodiment provides a road early disease visual detection system based on feature collaboration;
[0147] A road early disease visual detection system based on feature collaboration, comprising:
[0148] An acquisition module is configured to: acquire a road surface image to be detected and a roadbed image to be detected; the roadbed image to be detected is a roadbed image at a position where the road surface image to be detected is collected;
[0149] An alignment module is configured to: perform image alignment processing on the road surface image to be detected and the roadbed image to be detected to ensure that the spatial positions of the road defects in the road surface image to be detected and the roadbed image to be detected are corresponding;
[0150] An output module is configured to: input the aligned road surface image to be detected and the aligned roadbed image to be detected into the trained road early disease visual detection model, and output a road early disease classification label;
[0151] Among them, the trained road early disease visual detection model is used to extract features from the road surface image to be detected and the roadbed image to be detected respectively, and the extracted road surface features and roadbed features are fused to obtain fused features; the fused features are classified to obtain classification labels.
[0152] It should be noted that the acquisition module, alignment module and output module described above correspond to steps S101 to S103 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.
[0153] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0154] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.
[0155] Embodiment 3
[0156] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.
[0157] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0158] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software.
[0159] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.
[0160] Embodiment 4
[0161] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.
[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for visually detecting early road damage based on feature collaboration, characterized in that: include: Acquire a road surface image to be detected and a roadbed image to be detected; The roadbed image to be detected is a roadbed image at a position where the road surface image to be detected is collected; Performing image alignment processing on the road surface image to be detected and the roadbed image to be detected to ensure that the spatial positions of the road defects in the road surface image to be detected and the roadbed image to be detected are corresponding; The aligned road surface image to be detected and the aligned roadbed image to be detected are input into the trained road early disease visual detection model, and the road early disease classification label is output; wherein the trained road early disease visual detection model is used to extract features of the road surface image to be detected and the roadbed image to be detected respectively, and to fuse the extracted road surface features and roadbed features to obtain fused features; and the fused features are classified to obtain classification labels; The trained road early disease visual detection model includes: A shared encoding network and a feature decoding network connected in sequence; the shared encoding network comprises: a first deep convolutional neural network and a second deep convolutional neural network; wherein the input end of the first deep convolutional neural network is used to input a road surface image to be detected, and the output end of the first deep convolutional neural network outputs a feature map of the road surface image to be detected; The input end of the second deep convolutional neural network is used to input the optimized roadbed image, and the output end of the second deep convolutional neural network outputs the feature map of the optimized roadbed image; the output end of the first deep convolutional neural network and the output end of the second deep convolutional neural network are both connected to the input end of the feature fusion network; The feature fusion network splices the feature map of the road surface image to be detected with the feature map of the optimized roadbed image to obtain a first splicing result, and inputs the first splicing result into the first convolution layer for convolution operation to obtain a first convolution operation result; Performing R-Gcorrelation correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain a first interactive correlation feature; performing GR correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain a second interactive correlation feature; splicing the first interactive correlation feature and the second interactive correlation feature to obtain a second splicing result, and inputting the second splicing result into a second convolutional layer to perform a convolution operation to obtain a second convolution operation result; The first convolution operation result is concatenated with the second convolution operation result to obtain a fusion feature; Input the fused features into the feature decoding network and output the classification results; The step of performing R-Gcorrelation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain the first interactive correlation feature; and performing GR correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain the second interactive correlation feature specifically includes: For RG correlation processing, the road surface image feature map As a filter, then in the roadbed image feature map Perform a sliding window operation on the window. During the sliding window process, at each position in the window, calculate and The correlation score between them is obtained to obtain a new feature map. Each pixel value in the new feature map represents the value at the corresponding position. and The correlation score between Two feature maps and The size of is N×M. Specifically, for the pixel at each position (i, j) of the new feature map, the RG correlation score is calculated using the following formula: Among them, C R-G (i, j) represents the relevance score at position (i, j), Representation feature map The pixel value at position (i+m, j+n), Representation feature map The pixel value at position (i, j); By calculating the correlation score, a new feature map is obtained, in which each pixel value is represented at the corresponding position. and The correlation score between R-G (i,j); Similarly, the GR correlation operation is to convert As a filter, Perform the same operation on and The correlation between them.
2. The method for visually detecting early road damage based on feature collaboration as claimed in claim 1, characterized in that: Before the image alignment processing of the road surface image to be detected and the roadbed image to be detected is performed, the following steps are also included: The roadbed image to be detected is processed by using a saliency detection algorithm, the area with a saliency greater than a set threshold is retained, and the area with a saliency less than or equal to the set threshold is deleted to obtain a saliency area; The center point of the salient area is used as the center point of the image to be cropped, and the cropping size of the image to be cropped is set; The image to be cropped is cropped from the roadbed image to be detected according to the cropping size to obtain an updated roadbed image.
3. The method for visually detecting early road damage based on feature collaboration as claimed in claim 2, characterized in that: Performing image alignment processing on the road surface image to be detected and the roadbed image to be detected, including: Perform image edge detection on the road surface image to be detected to obtain the edge points of the road surface image; Performing image edge detection on the updated roadbed image to obtain the roadbed image edge points; Calculate the Euclidean distance between the edge points of the road surface image and the edge points of the roadbed image; Based on the Euclidean distance, the updated roadbed image is subjected to image alignment processing so that the edge line of the updated roadbed image is consistent in size and direction with the edge line of the road surface image to be detected; the image alignment processing includes: image scaling, rotation and translation; The roadbed image after image alignment is cropped again to obtain an optimized roadbed image, so that the size of the optimized roadbed image is consistent with that of the road surface image to be detected.
4. The method for visually detecting early road damage based on feature collaboration as claimed in claim 1, characterized in that: The aligned road surface image to be detected and the aligned roadbed image to be detected are input into the trained road early disease visual detection model, and the road early disease classification label is output. The training process of the trained road early disease visual detection model includes: Construct a data set, and divide the data set into: a training set, a validation set, and a test set according to proportion; The data set includes: pavement images and roadbed images with known pavement disease labels; The training set is input into the road early disease visual detection model for training. When the loss function value of the model no longer decreases, or the number of iterations exceeds the set number, the training is stopped to obtain the trained road early disease visual detection model. Inputting the validation set into the trained road early disease visual detection model for verification to verify whether the index value of the model reaches the set threshold; The test set is input into the trained road early disease visual detection model for testing, and the model whose test accuracy meets the set threshold is output as the final trained model.
5. A road early disease visual detection system based on feature collaboration, characterized in that: include: An acquisition module is configured to: acquire an image of a road surface to be detected and an image of a roadbed to be detected; The roadbed image to be detected is a roadbed image at a position where the road surface image to be detected is collected; An alignment module is configured to: perform image alignment processing on the road surface image to be detected and the roadbed image to be detected to ensure that the spatial positions of the road defects in the road surface image to be detected and the roadbed image to be detected are corresponding; An output module is configured to: input the aligned road surface image to be detected and the aligned roadbed image to be detected into the trained road early disease visual detection model, and output a road early disease classification label; Among them, the trained road early disease visual detection model is used to extract features from the road surface image to be detected and the roadbed image to be detected respectively, and the extracted road surface features and roadbed features are fused to obtain fused features; the fused features are classified to obtain classification labels; The trained road early disease visual detection model includes: A shared encoding network and a feature decoding network connected in sequence; the shared encoding network comprises: a first deep convolutional neural network and a second deep convolutional neural network; wherein the input end of the first deep convolutional neural network is used to input a road surface image to be detected, and the output end of the first deep convolutional neural network outputs a feature map of the road surface image to be detected; The input end of the second deep convolutional neural network is used to input the optimized roadbed image, and the output end of the second deep convolutional neural network outputs the feature map of the optimized roadbed image; the output end of the first deep convolutional neural network and the output end of the second deep convolutional neural network are both connected to the input end of the feature fusion network; The feature fusion network splices the feature map of the road surface image to be detected with the feature map of the optimized roadbed image to obtain a first splicing result, and inputs the first splicing result into the first convolution layer for convolution operation to obtain a first convolution operation result; Performing R-Gcorrelation correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain a first interactive correlation feature; performing GR correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain a second interactive correlation feature; splicing the first interactive correlation feature and the second interactive correlation feature to obtain a second splicing result, and inputting the second splicing result into a second convolutional layer to perform a convolution operation to obtain a second convolution operation result; The first convolution operation result is concatenated with the second convolution operation result to obtain a fusion feature; Input the fused features into the feature decoding network and output the classification results; The step of performing R-Gcorrelation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain the first interactive correlation feature; and performing GR correlation operation on the feature map of the road surface image to be detected and the feature map of the optimized roadbed image to obtain the second interactive correlation feature specifically includes: For RG correlation processing, the road surface image feature map As a filter, then in the roadbed image feature map Perform a sliding window operation on the window. During the sliding window process, at each position in the window, calculate and The correlation score between them is obtained to obtain a new feature map. Each pixel value in the new feature map represents the value at the corresponding position. and The correlation score between Two feature maps and The size of is N×M. Specifically, for the pixel at each position (i, j) of the new feature map, the RG correlation score is calculated using the following formula: Among them, C R-G (i,j) represents the relevance score at position (i,j), Representation feature map The pixel value at position (i+m,j+n), Representation feature map The pixel value at position (i, j); By calculating the correlation score, a new feature map is obtained, in which each pixel value is represented at the corresponding position. and The correlation score C R-G (i,j); Similarly, the GR correlation operation is to convert As a filter, Perform the same operation on and The correlation between them.
6. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 4 is executed.
7. A storage medium characterized by being non-temporary The computer-readable instructions are stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 4 are executed.
8. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 4 when running on one or more processors.
