Road surface defect detection methods, devices, equipment and storage media

Abstract

This application relates to the field of image processing technology and discloses a method, apparatus, device, and storage medium for road surface defect detection. The method extracts feature maps from a road surface image by retrieving features from each hidden layer of a convolutional neural network. Each feature map in the feature map set has corresponding label information for each road surface feature, indicating the degree of defect. Grad-CAM is used to convert each feature map set into a heatmap that only displays road surface features reaching a preset defect level. The defect authenticity of the road surface features displayed in each heatmap is evaluated, and based on the evaluation results, a feature map set that meets a preset accuracy is selected to determine the corresponding target hidden layer. The convolutional neural network is reconstructed based on the target hidden layer to obtain the target convolutional neural network. Features are then re-extracted from the road surface image using the target convolutional neural network, and road surface defects are detected based on the extracted features. This addresses the problem of excessive reliance on labeled data in existing technologies.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for detecting road surface defects. Background Technology

[0002] With rapid societal development, cities are becoming increasingly interconnected, and ground road transportation systems are becoming more and more indispensable in daily life. In my country, highways generally use asphalt concrete pavements with semi-rigid bases. Due to years of exposure to wind and sun, rain erosion, and repeated compaction by heavy vehicles, structural damage over time affects road durability. Aging inevitably leads to a decline in structural performance, thus impacting driving safety.

[0003] With the development of deep learning methods, deep learning based on convolutional neural networks (CNN) and semantic segmentation to detect continuous defects such as cracks is currently the most advanced application. However, this method requires a large amount of data to be collected in advance for training to ensure a certain level of accuracy. In other words, it generally suffers from high dependence on labeled data, insufficient interpretability, or limited adaptability. Summary of the Invention

[0004] This application provides a method, apparatus, equipment, and storage medium for detecting road defects, in order to solve the problem that existing detection methods rely too heavily on labeled data.

[0005] The first aspect of this application provides a method for detecting road surface defects. The method includes: inputting an acquired road surface image into a convolutional neural network for feature extraction, and extracting feature maps extracted from each hidden layer of the convolutional neural network, wherein each feature map in the feature map has corresponding label information for each road surface feature, and the label information is used to indicate the degree of defect of the road surface feature; converting each feature map into a heatmap using Grad-CAM, wherein the heatmap only displays road surface features that reach a preset defect degree; performing a defect authenticity assessment on the road surface features displayed in each heatmap, and selecting a feature map that meets a preset accuracy based on the assessment result, and determining the corresponding target hidden layer; reconstructing the convolutional neural network based on the target hidden layer to obtain a target convolutional neural network; re-extracting features from the road surface image using the target convolutional neural network, and detecting road surface defects based on the extracted features.

[0006] In one feasible implementation, the step of converting each feature map into a heatmap using Grad-CAM includes: extracting label information from the feature maps in each hidden layer using Grad-CAM, and extracting the features of interest from the feature maps based on the relationship between the label information and the degree of defects to obtain the road defect interest region; performing visualization processing on the road defect interest region, and generating a heatmap of the corresponding hidden layer based on the result of the visualization processing.

[0007] In one feasible implementation, the visualization processing of the pavement defect area of ​​interest includes: calculating the mean of multiple feature maps corresponding to the pavement defect area of ​​interest using a visualization processing formula to obtain the visualization result of the pavement defect area of ​​interest.

[0008] In one feasible implementation, generating a heatmap of the corresponding hidden layer based on the result of visualization processing includes: compressing multiple feature maps of each of the visualized hidden layers and calculating the average feature value; and generating a heatmap of the corresponding hidden layer based on the average feature value using heatmap generation rules.

[0009] In one feasible implementation, the step of generating a heatmap of the corresponding hidden layer based on the average feature value using heatmap generation rules includes: using the ReLU activation function to remove features that do not contribute to defect identification or have a negative impact on all average feature values, and normalizing the removed feature map; determining the target color corresponding to the normalized feature map based on a preset color mapping rule, and generating a heatmap of the corresponding hidden layer.

[0010] In one feasible implementation, the step of evaluating the authenticity of road surface features displayed in each heatmap and selecting feature maps that meet a preset accuracy based on the evaluation results to determine the corresponding target hidden layer includes: using a preset scoring formula to calculate the distance between the road surface features displayed in each heatmap and the features in the real defect area to obtain a score; using a selective layer attention network to select target feature maps whose scores reach a preset similarity value, and determining the target hidden layer corresponding to the target feature map based on the correspondence between the feature map and the hidden layer.

[0011] In one feasible implementation, reconstructing the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network includes: retaining all target hidden layers in the convolutional neural network and loading the pre-trained weights of each target hidden layer for configuration to obtain the target convolutional neural network.

[0012] A second aspect of this application provides a road surface defect detection device, comprising: an extraction module, configured to input an acquired road surface image into a convolutional neural network for feature extraction, and extract feature maps extracted from each hidden layer of the convolutional neural network, wherein each feature map in the feature map set has corresponding label information for each road surface feature, and the label information is used to indicate the defect degree of the road surface feature; a conversion module, configured to convert each feature map set into a heatmap using Grad-CAM, wherein the heatmap only displays road surface features that reach a preset defect degree; an evaluation module, configured to evaluate the defect authenticity of the road surface features displayed in each heatmap, and select a feature map set that meets a preset accuracy based on the evaluation result, and determine the corresponding target hidden layer; a reconstruction module, configured to reconstruct the convolutional neural network based on the target hidden layer to obtain a target convolutional neural network; and a detection module, configured to re-extract features from the road surface image using the target convolutional neural network, and detect road surface defects based on the extracted features.

[0013] A third aspect of this application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to perform the above-described road surface defect detection method.

[0014] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned road surface defect detection method.

[0015] The technical solution provided in this application involves extracting feature maps from road images by various hidden layers in a convolutional neural network. Each feature map in the feature map set has corresponding label information for each road feature, which indicates the degree of defect of the road feature. Grad-CAM is used to convert each feature map set into a heatmap that only displays road features that reach a preset defect level. The defect authenticity of the road features displayed in each heatmap is evaluated, and based on the evaluation results, a feature map set that meets a preset accuracy is selected to determine the corresponding target hidden layer. The convolutional neural network is reconstructed based on the target hidden layer to obtain the target convolutional neural network. The target convolutional neural network is used to re-extract features from the road image, and road defects are detected based on the extracted features. In this application, the features extracted from each layer of the convolutional neural network are visualized using Grad-CAM. Then, the features extracted from each layer are evaluated for their defect authenticity to select the target hidden layer with high similarity to the real defect region. Finally, the convolutional neural network is reconstructed based on the target hidden layer. This method adjusts the feature extraction layer in the convolutional neural network so that it can select the feature layer with the most information content and accuracy. It enables the localization of the salient region of the defect even without a large amount of labeled data, reduces the dependence on labeled data, improves the detection effect, and reduces the consumption of computing resources. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of one embodiment of the road surface defect detection method in this application;

[0017] Figure 2 This is a schematic diagram of another embodiment of the road surface defect detection method in this application;

[0018] Figure 3 This refers to the feature map set of each hidden layer represented by Grad-CAM in the embodiments of this application;

[0019] Figure 4 This is the calculation process for defect authenticity assessment in the embodiments of this application;

[0020] Figure 5 This is a schematic diagram of the selective attention module in an embodiment of this application;

[0021] Figure 6 This is the maximum pooling process in the embodiments of this application;

[0022] Figure 7 This is a schematic diagram of one embodiment of the road surface defect detection device in this application;

[0023] Figure 8 This is a schematic diagram of another embodiment of the road surface defect detection device in this application;

[0024] Figure 9 This is a schematic diagram of one embodiment of the electronic device described in this application. Detailed Implementation

[0025] This application provides a method, apparatus, equipment, and storage medium for detecting road surface defects.

[0026] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It is understood that the executing entity of this application can be a road surface defect detection device, a terminal, or a server. The specific implementation is not limited here. This application will use a server as the executing entity for illustration.

[0028] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the road surface defect detection method in this application includes:

[0029] 101. Input the acquired road surface image into a convolutional neural network for feature extraction, and extract the feature map set extracted from each hidden layer in the convolutional neural network. Each feature map in the feature map set has corresponding label information for each road surface feature. The label information is used to indicate the degree of defect of the road surface feature.

[0030] It should be noted that the road surface image refers to a road surface image after preprocessing, such as image cleaning or image enhancement.

[0031] This convolutional neural network is a model for analyzing road surface defect features or road surface information obtained by pre-training using training data. The model consists of multiple hidden layers, and each hidden layer analyzes the road surface image, extracts and outputs road surface features to obtain the corresponding feature map set.

[0032] That is, the acquired road surface image is input into the model, the hidden layers of the model extract features, output the corresponding feature map set, and then the road surface defect is identified and detected for each feature map set. Based on the detection results, a label information is generated; at the same time, the label information is used to identify the corresponding feature as part of the feature map set.

[0033] In practical applications, the convolutional neural network also includes a network structure for identifying road surface defects. After extracting features from each hidden layer, the features are input into the identification network structure to complete the identification of whether it is a defect and generate a label information, which can be color, mark, symbol, etc.

[0034] 102. Use Grad-CAM to convert each feature map set into a heat map, which only displays road surface features that reach the preset defect level.

[0035] In this embodiment, when using Grad-CAM for heatmap conversion, Grad-CAM extracts features from each feature map set that meet the defect identification criteria and performs visualization processing to obtain heatmaps for each feature map set. Specifically, Grad-CAM performs an unweighted average calculation on each feature map set and compares the calculated result with a category-specific Grad-CAM heatmap to filter out features that meet the defect identification criteria and then visualizes them to generate heatmaps. It should be noted that this visualization can be implemented using a color representation method, that is, matching the corresponding color according to the defect identification level, and then replacing the color of the location of the feature in the feature map set with the corresponding color to achieve visualization.

[0036] 103. Evaluate the authenticity of the road surface features shown in each heatmap, and select a feature map set that meets the preset accuracy based on the evaluation results, and determine the corresponding target hidden layer.

[0037] Specifically, the matching degree between the road surface features shown in the feature map and the actual road surface defects is calculated. This matching degree includes the distance between two adjacent road surface features and the similarity of the features themselves. Then, a weighting coefficient is introduced to generate a score.

[0038] The system selects a superior feature map set based on the score, and then matches the target hidden layer corresponding to the superior feature map set based on the output relationship between the selected feature map set and the hidden layer. The selection method can be based on the similarity comparison.

[0039] 104. Reconstruct the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network.

[0040] In this embodiment, the reconstruction can be achieved through elimination and masking. That is, the hidden layers other than the target hidden layer in the original convolutional neural network are eliminated to generate a new convolutional neural network, or a masking label is set for the hidden layers other than the target hidden layer in the original convolutional neural network. After feature extraction is performed using the convolutional neural network again, the features output by each hidden layer are filtered according to the masking label, or the hidden layers with the masking label do not output features.

[0041] 105. Use a target convolutional neural network to re-extract features from the road surface image, and detect road surface defects based on the extracted features.

[0042] Understandably, when this target convolutional neural network extracts road surface images, its output features are only the output features of the target hidden layer, and then only the feature map set output by the target hidden layer is used to detect road surface defects.

[0043] In this embodiment, the features extracted from each layer of the convolutional neural network are visualized using Grad-CAM. Then, the features extracted from each layer are evaluated for their defect authenticity to select a target hidden layer with high similarity to the real defect area. Finally, the convolutional neural network is reconstructed based on the target hidden layer. This method adjusts the feature extraction layer in the convolutional neural network so that it can select the feature layer with the most information and accuracy. It enables the localization of the salient region of the defect even without a large amount of labeled data, reducing the dependence on labeled data.

[0044] Please see Figure 2 Another embodiment of the pavement defect detection method in this application includes:

[0045] 201. Input the acquired road surface image into a convolutional neural network for feature extraction, and extract the feature map set extracted from each hidden layer in the convolutional neural network. Each feature map in the feature map set has corresponding label information for each road surface feature. The label information is used to indicate the degree of defect of the road surface feature.

[0046] In this embodiment, road surface images can be acquired by using devices such as drones or road inspection robots. These devices can efficiently and comprehensively cover large areas of the inspection area, especially for hard-to-reach areas (such as overpasses, tunnels, or remote roads).

[0047] The acquired road surface images are input into a convolutional neural network (CNN). The CNN extracts road surface features from the images and identifies features that indicate road defects. It's important to note that the CNN consists of multiple hidden layers, and each hidden layer extracts potential defective road surface features from the image, outputting a feature map. This feature map can be based on the road surface image, with potentially defective features annotated or outlined within it.

[0048] Specifically, feature maps are extracted from all hidden layers. These feature maps from different layers are generated by performing convolution operations on the corresponding hidden layers in the convolutional neural network. Each feature map has multiple channels, which represent the positions of different important features in the image. These feature maps help to reveal key areas of defects such as cracks and pits.

[0049] 202. Grad-CAM is used to extract the label information of the feature map in each hidden layer, and the features of interest in the feature map are extracted based on the relationship between the label information and the degree of defect, so as to obtain the area of ​​interest for road defects.

[0050] Understandably, the visualization result of the pavement defect area of ​​interest is obtained by averaging multiple feature maps corresponding to the pavement defect area of ​​interest using a visualization processing formula, wherein the visualization processing formula is: in, It is the feature set of the hidden layer, F k This is achieved by compressing the feature map set into a two-dimensional feature map.

[0051] Specifically, by comparing the unweighted average feature map set with the category-specific Grad-CAM heatmap, it can be seen whether the model's focus when detecting road defects is reasonable. The average feature map set generated by the visualization processing formula can highlight these non-category-specific areas as potential defect areas, helping to identify minor defects such as surface unevenness or slight cracks.

[0052] 203. Visualize the areas of interest for road surface defects and generate heatmaps of the corresponding hidden layers based on the results of the visualization.

[0053] In this embodiment, multiple feature maps of each of the visualized hidden layers are compressed, and the average feature value is calculated. Using heatmap generation rules, a heatmap of the corresponding hidden layer is generated based on the average feature value. In other words, Grad-CAM uses heatmaps to display these salient regions, helping us understand the model's attention distribution when processing road structure images, thereby supporting the localization and identification of road defects. Grad-CAM uses a heatmap method to visualize feature maps.

[0054] Furthermore, the ReLU activation function is used to remove features that do not contribute to or have a negative impact on defect identification from all average feature values, and the removed feature map is normalized. Based on the preset color mapping rules, the target color corresponding to the normalized feature map is determined, and the heatmap of the corresponding hidden layer is generated.

[0055] In practical applications, this method for generating heatmaps involves squeezing and averaging the feature values ​​of N feature maps in the k-th hidden layer. Then, ReLU activation is performed using the first formula, setting all negative values ​​to 0 and retaining only positive values. Regions that contribute nothing or have a negative impact on the target category are removed, ensuring that subsequent visualizations only include regions with a positive impact on the target category. The first formula is: x ij It is the feature map value in the i-th row and j-th column of the k-th hidden layer.

[0056] By using the second formula for normalization, the two-dimensional vector-heatmap element (x) is transformed. i x j Normalization to values ​​between 0 and 1 enhances image contrast and ensures a more even distribution of color mapping across activation regions of varying intensities, making high-activation areas more prominent. After normalization, color mapping is applied to map the numerical range to a color range, ensuring different activation intensities correspond to different colors, thus obtaining the heatmap. The second formula is: x′ ij These are the normalized eigenvalues, representing the values ​​after normalization; x ij These are the original eigenvalues, representing the unnormalized eigenvalues; max(x) ij ) represents the original eigenvalue x ij The maximum value in the range is used for normalization operations.

[0057] Figure 3 To obtain heatmaps through the aforementioned transformation process, the feature maps of different hidden layers differ in size and heatmap distribution. Therefore, the feature representations obtained through heatmap visualization also vary. Quantitative analysis of the heatmaps from each layer reveals that the feature maps generated by intermediate layers are more complex than those of the last layer in the CNN model. This indicates that earlier layers can capture subtle local defects on the road surface, such as cracks and small potholes, while the network focuses more on overall structural features closer to the last layer, helping to identify larger-scale defect areas. This hierarchical analysis allows for a deeper understanding of how the network processes detailed information in road images at each layer, thereby improving the accuracy and reliability of road structural defect detection and providing more precise data support for maintenance and repair.

[0058] 204. Evaluate the authenticity of the road surface features shown in each heatmap, and select feature maps that meet the preset accuracy based on the evaluation results to determine the corresponding target hidden layer.

[0059] Based on the evaluation results, the feature layer that most closely resembles the actual defect area is selected. This layer contains the region of interest that the model most accurately focuses on regarding the defect area. Based on the evaluation results, the convolutional layer that most accurately reflects the defect information can be selected, while irrelevant or inaccurate layers are excluded.

[0060] By employing the Selective Layer Attention Network (SAN) method, the layers with the highest scores are selected from all hidden layers. A new, simplified model is then constructed, retaining only these selected layers. The reconstructed model is more efficient, with a significant reduction in computational cost, avoiding unnecessary waste of computational resources.

[0061] In this embodiment, a preset scoring formula is used to calculate the distance between the road surface features displayed in each heatmap and the features in the real defect area to obtain a score; a selective layer attention network is used to select the target feature map whose score reaches a preset similarity value, and the target hidden layer corresponding to the target feature map is determined based on the correspondence between the feature map and the hidden layer.

[0062] It should be noted that the evaluation result is actually a quantitative indicator representing the similarity between the hidden layer feature maps and the ground truth labels. Specifically, it is calculated using this scoring formula to estimate the perceptual accuracy of feature extraction, while taking into account the size of each layer's feature map. The scoring formula is S(L... k The expression is as follows:

[0063]

[0064] Where S(L) k E is the feature map score of the k-th hidden layer; i,j It is the absolute error in calculating the distance between the active region extracted from the hidden layer and the labeled target defect region; E i,j Defined by the following formula.

[0065] E i,j =|x' x,j -t i,j |,

[0066] x'∈(0,1),

[0067] t∈{0,1},

[0068] Where, x′ i,j It is a normalized heatmap (V k ) elements; t i,jThese are elements of the true label mask (Mk) with binary conversion defect labels. The true label mask consists of 0s and 1s, while the heatmap values ​​are between 0 and 1. The accuracy of the heatmap is quantitatively estimated by calculating the difference between the heatmap vector values ​​and the target values. As the feature vector (x... i,j Approaching the target vector (t) i,j Consistency increases. Therefore, the sum of each consistency value divided by the total number of feature vectors serves as an indicator for extracting the layer with the highest similarity to the expected localization. The calculation process is as follows: Figure 4 As shown.

[0069] 205. Reconstruct the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network.

[0070] 206. Use a target convolutional neural network to re-extract features from the road surface image, and detect road surface defects based on the extracted features.

[0071] Specifically, when reconstructing a convolutional neural network, the target convolutional neural network is obtained by retaining all the target hidden layers in the convolutional neural network and loading the pre-trained weights of each target hidden layer.

[0072] By loading pre-trained weights, you can select layers and load them into a new, simplified model without retraining the entire network. This helps maintain the performance of existing models while significantly reducing computational resource consumption.

[0073] Furthermore, the step of reconstructing the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network further includes: introducing an attention mechanism into each target hidden layer in the target convolutional neural network, wherein the attention mechanism is implemented by a combination of max unpooling and average pooling.

[0074] Max Unpooling and Average Pooling: Max Unpooling is applied to the selected convolutional layer to recover the spatial information lost during Max Pooling. To prevent further loss of spatial information, an average pooling operation is also added, which can further improve the integrity of the feature map.

[0075] Enhance key features: Use an attention mechanism to strengthen the focus on key features. The attention module improves the accuracy of defect detection by automatically emphasizing key features (such as defect location and shape) during training, thereby increasing the precision of the final feature map.

[0076] In practical applications, depending on the selected layer, the convolutional network is reconstructed into a scaled-down convolutional network, where the model loads pre-trained weights only from the input layer to the selected layer (L). c ), while excluding the remaining layers.

[0077] Then, the proposed attention module is combined with the selected layer to provide enhanced predictive feature map information, more efficiently focusing on pavement defect areas and enhancing the ability to capture details such as cracks and potholes, providing a more accurate and efficient solution for pavement structural defect detection tasks. The attention module emphasizes the important features of the selected layer. The proposed attention mechanism extracts, enhances, and aligns feature information through progressive processing, thereby improving the model's focus on key areas, such as... Figure 5 As shown.

[0078] This selective layer attention module fully utilizes the salient features extracted by the pooling layer and enhances the expressive power of key information by combining max unpooling and upsampling operations. Specific steps:

[0079] Max pooling records salient feature indices: First, salient regions of the feature map are extracted from the max pooling layer, and the position index of the maximum value in each pooling window (i.e., the pooling index) is recorded. These indices are used to indicate the original spatial position of the activated features before pooling, ensuring that the position of salient features can be correctly restored during subsequent depooling.

[0080] Max unpooling restores salient features: Using the saved pooling index, the pooled activation features are restored to their positions in the original feature map, generating a sparse feature map. This sparse feature map retains key information (i.e., maximizes) of salient activation regions, while non-salient regions are set to zero.

[0081] Upsampling projects salient features: By upsampling, the sparse feature map is expanded to a higher resolution, allowing salient features to cover a larger spatial range. The upsampled feature map is shown below. Figure 6 As shown, the largest number of activation features were projected and enhanced.

[0082] Secondary upsampling enhances unpooled features: The feature map after max unpooling is further upsampled to refine the features and enhance information in key regions. This process ensures that unpooled features are fully represented in the high-resolution feature map.

[0083] Supplementing Spatial Information: Average Pooling: To prevent over-concentration on activation features and neglect of other important spatial information during pooling and upsampling, the module introduces an average pooling operation. Average pooling smoothly aggregates global spatial information in the feature map, supplementing details that may be missed in max pooling and upsampling.

[0084] Feature map fusion for attention weights: The feature maps generated by max pooling and average pooling are fused to combine salient activation regions and global spatial information, generating an attention weight map. This attention weight map is used to weight the original feature maps, enhancing the model's focus on key features.

[0085] In this embodiment, the features extracted from each layer of the convolutional neural network are visualized using Grad-CAM. Then, the authenticity of the features extracted from each layer is evaluated to select the target hidden layer with high similarity to the real defect area. Finally, the convolutional neural network is reconstructed based on the target hidden layer. This method adjusts the feature extraction layer in the convolutional neural network so that it can select the feature layer with the most information and accuracy. It enables the localization of the salient area of ​​the defect even without a large amount of labeled data, reduces the dependence on labeled data, improves the detection effect, and reduces the consumption of computing resources.

[0086] The road surface defect detection method in the embodiments of this application has been described above. The road surface defect detection device in the embodiments of this application is described below. Please refer to Figures 8 and 9. An embodiment of the road surface defect detection device in the embodiments of this application includes:

[0087] The extraction module 710 is used to input the acquired road surface image into a convolutional neural network for feature extraction, and to extract the feature map set extracted by each hidden layer in the convolutional neural network. Each feature map in the feature map set has corresponding label information for each road surface feature, and the label information is used to indicate the degree of defect of the road surface feature.

[0088] The conversion module 720 is used to convert each of the feature maps into a heat map using Grad-CAM, wherein the heat map only shows road surface features that reach a preset defect level;

[0089] The evaluation module 730 is used to evaluate the authenticity of the road surface features displayed in each of the heat maps, and select a feature map set that meets the preset accuracy based on the evaluation results to determine the corresponding target hidden layer.

[0090] Reconstruction module 740 is used to reconstruct the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network;

[0091] The detection module 750 is used to re-extract features from the road surface image using the target convolutional neural network and to detect road surface defects based on the extracted features.

[0092] Optionally, the conversion module 720 is specifically used for:

[0093] Grad-CAM is used to extract the label information of the feature maps in each hidden layer, and the features of interest in the feature maps are extracted based on the relationship between the label information and the degree of defects, so as to obtain the road defect interest area;

[0094] The road surface defect area of ​​interest is visualized, and a heatmap of the corresponding hidden layer is generated based on the visualization results.

[0095] Optionally, the conversion module 720 is specifically used for:

[0096] The mean value of multiple feature maps corresponding to the area of ​​interest in the road defect is calculated using a visualization processing formula to obtain the visualization result of the area of ​​interest in the road defect.

[0097] Optionally, the conversion module 720 is specifically used for:

[0098] The multiple feature maps of each of the visualized hidden layers are squeezed together, and the average feature value is calculated.

[0099] Using heatmap generation rules, heatmaps for the corresponding hidden layers are generated based on the average feature values.

[0100] Optionally, the conversion module 720 is specifically used for:

[0101] The ReLU activation function is used to remove features from all average eigenvalues ​​that do not contribute to or have a negative impact on defect identification, and the feature map after removal is normalized.

[0102] Based on the preset color mapping rules, the target color corresponding to the normalized feature map is determined, and the heatmap of the corresponding hidden layer is generated.

[0103] Optionally, the evaluation module 730 includes:

[0104] The calculation unit 731 is used to calculate the distance between the road surface features displayed in each of the heat maps and the features in the actual defect area using a preset scoring formula, and obtain a score.

[0105] Evaluation unit 732 is used to select target feature maps with scores that reach a preset similarity value using a selective layer attention network, and to determine the target hidden layer corresponding to the target feature map based on the correspondence between the feature map and the hidden layer.

[0106] Optionally, the reconstruction module 740 is specifically used for:

[0107] All target hidden layers in the convolutional neural network are retained, and the pre-trained weights of each target hidden layer are loaded and configured to obtain the target convolutional neural network.

[0108] In this embodiment, the features extracted from each layer of the convolutional neural network are visualized using Grad-CAM. Then, the authenticity of the features extracted from each layer is evaluated to select the target hidden layer with high similarity to the real defect area. Finally, the convolutional neural network is reconstructed based on the target hidden layer. This method adjusts the feature extraction layer in the convolutional neural network so that it can select the feature layer with the most information and accuracy. It enables the localization of the salient area of ​​the defect even without a large amount of labeled data, reduces the dependence on labeled data, improves the detection effect, and reduces the consumption of computing resources.

[0109] above Figure 7 and Figure 8 The road surface defect detection device in this application embodiment is described in detail from the perspective of modular functional entities. The electronic equipment in this application embodiment is described in detail from the perspective of hardware processing.

[0110] See Figure 9 As shown, the electronic device includes a processor 900 and a memory 901. The memory 901 stores machine-executable instructions that can be executed by the processor 900. The processor 900 executes the machine-executable instructions to implement the above-described road surface defect detection method.

[0111] Furthermore, Figure 9 The electronic device shown also includes a bus 902 and a communication interface 903. The processor 900, the communication interface 903, and the memory 901 are connected via the bus 902.

[0112] The memory 901 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 903 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 902 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0113] The processor 900 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 900 or by instructions in software form. The processor 900 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 901. Processor 900 reads the information in memory 901 and, in conjunction with its hardware, completes the method steps of the aforementioned embodiment.

[0114] This application also provides an electronic device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the road surface defect detection method in the above embodiments.

[0115] This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the road surface defect detection method provided above.

[0116] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting road surface defects, characterized in that, The method includes: The acquired road surface image is input into a convolutional neural network for feature extraction, and feature maps extracted from each hidden layer of the convolutional neural network are extracted. Each feature map in the feature map set has corresponding label information for each road surface feature, and the label information is used to indicate the degree of defect of the road surface feature. Grad-CAM is used to convert each of the feature maps into a heat map, wherein the heat map only shows road surface features that reach a preset defect level; The authenticity of the road surface features displayed in each heat map is evaluated, and based on the evaluation results, a feature map set that meets the preset accuracy is selected to determine the corresponding target hidden layer. The convolutional neural network is reconstructed based on the target hidden layer to obtain the target convolutional neural network; The target convolutional neural network is used to re-extract features from the road surface image, and road surface defects are detected based on the extracted features; The step of evaluating the authenticity of road surface features displayed in each heatmap, and selecting feature maps that meet a preset accuracy based on the evaluation results to determine the corresponding target hidden layer includes: Using a preset scoring formula, the distance between the road surface features shown in each heat map and the features in the actual defect area is calculated to obtain a score; A selective layer attention network is used to select target feature maps that achieve a preset similarity value, and the target hidden layer corresponding to the target feature map is determined based on the correspondence between the feature map and the hidden layer. Wherein, the scoring formula The expression is as follows: ， in, It is the feature map score of the k-th hidden layer; It is the absolute error in calculating the distance between the active region extracted from the hidden layer and the labeled target defect region; Defined by the following formula: ， ， ， It is a normalized heatmap Element; These are elements of the real label mask with binary conversion defect labels; the real label mask consists of 0s and 1s, while the heatmap values ​​are between 0 and 1; the accuracy of the heatmap is quantitatively estimated by calculating the difference between the heatmap vector values ​​and the target values; as the feature vector... Approaching the target vector Consistency increases.

2. The road surface defect detection method according to claim 1, characterized in that, The process of converting each feature map into a heatmap using Grad-CAM includes: Grad-CAM is used to extract the label information of the feature maps in each hidden layer, and the features of interest in the feature maps are extracted based on the relationship between the label information and the degree of defects, so as to obtain the road defect interest area; The road surface defect area of ​​interest is visualized, and a heatmap of the corresponding hidden layer is generated based on the visualization results.

3. The road surface defect detection method according to claim 2, characterized in that, The visualization processing of the road surface defect area of ​​interest includes: The mean value of multiple feature maps corresponding to the area of ​​interest in the road defect is calculated using a visualization processing formula to obtain the visualization result of the area of ​​interest in the road defect.

4. The road surface defect detection method according to claim 2, characterized in that, The generation of heatmaps corresponding to hidden layers based on the results of visualization processing includes: The multiple feature maps of each of the visualized hidden layers are squeezed together, and the average feature value is calculated. Using heatmap generation rules, heatmaps for the corresponding hidden layers are generated based on the average feature values.

5. The road surface defect detection method according to claim 4, characterized in that, The step of generating a heatmap of the corresponding hidden layer based on the average feature value using heatmap generation rules includes: The ReLU activation function is used to remove features from all average eigenvalues ​​that do not contribute to or have a negative impact on defect identification, and the feature map after removal is normalized. Based on the preset color mapping rules, the target color corresponding to the normalized feature map is determined, and the heatmap of the corresponding hidden layer is generated.

6. The method for detecting road surface defects according to claim 1, characterized in that, The process of reconstructing the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network includes: All target hidden layers in the convolutional neural network are retained, and the pre-trained weights of each target hidden layer are loaded and configured to obtain the target convolutional neural network.

7. A road surface defect detection device, characterized in that, The device includes: The extraction module is used to input the acquired road surface image into a convolutional neural network for feature extraction, and to extract the feature map set extracted by each hidden layer in the convolutional neural network. Each feature map in the feature map set has corresponding label information for each road surface feature, and the label information is used to indicate the degree of defect of the road surface feature. A conversion module is used to convert each of the feature maps into a heat map using Grad-CAM, wherein the heat map only displays road surface features that reach a preset defect level; The evaluation module is used to evaluate the authenticity of the road surface features displayed in each heat map, and select a feature map set that meets the preset accuracy based on the evaluation results to determine the corresponding target hidden layer. The reconstruction module is used to reconstruct the convolutional neural network based on the target hidden layer to obtain the target convolutional neural network; The detection module is used to re-extract features from the road surface image using the target convolutional neural network, and to detect road surface defects based on the extracted features; The step of evaluating the authenticity of road surface features displayed in each heatmap, and selecting feature maps that meet a preset accuracy based on the evaluation results to determine the corresponding target hidden layer includes: Using a preset scoring formula, the distance between the road surface features shown in each heat map and the features in the actual defect area is calculated to obtain a score; A selective layer attention network is used to select target feature maps that achieve a preset similarity value, and the target hidden layer corresponding to the target feature map is determined based on the correspondence between the feature map and the hidden layer. Wherein, the scoring formula The expression is as follows: ， in, It is the feature map score of the k-th hidden layer; It is the absolute error in calculating the distance between the active region extracted from the hidden layer and the labeled target defect region; Defined by the following formula: ， ， ， It is a normalized heatmap Element; These are elements of the real label mask with binary conversion defect labels; the real label mask consists of 0s and 1s, while the heatmap values ​​are between 0 and 1; the accuracy of the heatmap is quantitatively estimated by calculating the difference between the heatmap vector values ​​and the target values; as the feature vector... Approaching the target vector Consistency increases.

8. An electronic device, characterized in that, The electronic device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the electronic device to perform the road surface defect detection method as described in any one of claims 1-6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the road surface defect detection method as described in any one of claims 1-6.

Images