Deep learning integrated asphalt pavement damage visual analysis method and system

Abstract

The present application relates to the field of image processing, in particular to an asphalt pavement damage visual analysis method and system integrating deep learning. It aims to solve the problem of misidentifying the asphalt pavement repair area as a crack in the prior art. An RGB image, a thermal image, a location and an environmental parameter are obtained by a vehicle-mounted device; local asphalt material properties are queried; an anti-interference fusion feature map is generated by fusing the RGB and thermal image features, combining the attention mechanism and the environmental parameter; an enhanced feature map is generated by adjusting the convolution kernel using the material properties through the MAML algorithm; and the damage is identified through the domain adaptive classifier and the adversarial training. The enhanced repair area and crack area differentiation ability reduces the misjudgment caused by environmental and regional material differences, and improves the identification accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and system for visual analysis of asphalt pavement damage integrated with deep learning. Background Art

[0002] Asphalt pavement cracks are a typical road hazard. They not only affect the pavement's aesthetics and driving comfort, but can also exacerbate structural damage. Cracks allow rainwater and snow to seep into the base layer and roadbed, softening and weakening the base layer, leading to more serious problems like subsidence and potholes. Accurate crack identification can provide timely insight into the damage status of the pavement structure, providing a basis for targeted repairs and effectively preventing further damage.

[0003] Traditionally, crack detection in asphalt pavements relies on manual inspections and counting, but this is inefficient and results in a high rate of missed detections. Thanks to the rapid development of vision and image processing technologies, methods are emerging that combine vision with deep learning models to analyze and identify asphalt pavement cracks. However, due to the high similarity in visual features such as texture, color, and shape between asphalt pavement repair areas and cracks in RGB images, this often leads to model misclassification, compromising the accuracy of disease identification.

[0004] The disadvantage of the existing technology is that there is a problem of misidentifying the repaired area of ​​the asphalt pavement as a crack in the asphalt pavement. Summary of the Invention

[0005] The present invention provides a method and system for visual analysis of asphalt pavement damage that integrates deep learning. The technical problem to be solved is the problem of misidentifying repaired areas of asphalt pavement as cracks in the asphalt pavement.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] First, a deep learning-based visual analysis method for asphalt pavement damage is developed, which includes the following steps:

[0008] Step S1: Acquire RGB image data and thermal image data of the road surface through the vehicle-mounted multispectral camera; acquire location data through the GPS module; and acquire real-time environmental parameters through the environmental sensor.

[0009] Step S2: querying a pre-built material knowledge graph library based on the location data to obtain local asphalt material properties, wherein the local asphalt material properties include expansion coefficient and elastic modulus; the material knowledge graph library stores a mapping relationship between regional coordinates and the asphalt material properties of the region.

[0010] Step S3: RGB image feature map Thermal image characteristic diagram The fused feature map is spliced ​​along the channel direction, and local features are extracted from the spliced ​​fused feature map using convolution to capture local spatial correlation across modalities. The association map output by the convolution is converted into an attention weight map Att using the Sigmoid activation function.

[0011] Step S4: construct a thermal image weight expression, calculate the thermal image weight and the RGB image weight based on the environmental parameters using the thermal image weight expression, and obtain the fusion feature map based on the constructed fusion feature map expression. ;

[0012] Step S5: Use the elastic modulus E and thermal expansion coefficient αT obtained from the local asphalt material properties to splice into a conditional vector [E, αT]; based on the fusion feature map and the conditional vector, the convolution kernel parameters are adaptively adjusted through the MAML algorithm to generate a material enhancement feature map with enhanced material features. .

[0013] Step S6: Enhance the material feature map The input is sent to a domain adaptive classifier for damage identification. The classifier combines the correlation between material properties and image features constrained by adversarial training to output the discrimination results between repair areas and cracks.

[0014] Further, specifically, according to the light intensity scalar Adjust HDR parameters, when The three-level exposure synthesis is enabled when the image is in the same scene; Mask R-CNN is used to segment the road area, generate ROI masks, filter out non-road background noise, and preprocess the RGB image data.

[0015] Furthermore, the pixels of the RGB image and the thermal image are normalized between [0, 1] to obtain the RGB image feature map , Thermal image characteristic diagram ;

[0016] The expression of the attention weight map is: ;in, is the Sigmoid function, and Conv represents convolution.

[0017] Furthermore, the fusion feature map expression is: =( ·Att)· +(1- ·Att)· ; Combined with thermal image weight and cross-modal attention maps The multiplication and superposition control dynamically allocates the contribution weights of RGB and thermal image data in the spatial dimension, and finally generates a fusion feature map that is resistant to environmental interference. .

[0018] Furthermore, the rainfall intensity, light intensity, and surface temperature are normalized to 0-1 scalars respectively; thermal image weight ; Where P is the rainfall intensity scalar, L is the light intensity scalar, α, β, γ are preset coefficients; after calibration experiments, the optimized coefficients α=0.5, β=2.0, γ=0.3.

[0019] Furthermore, the conditional vector is mapped to the convolution kernel offset through a two-layer fully connected network, ΔK=FC2(ReLU(FC1([E,αT]))); where FC1 and FC2 are the first fully connected layer and the second connected layer respectively; ReLU is the activation function.

[0020] The first layer of convolution kernel is: = +ΔK; For highly elastic materials, the weight of the horizontal texture convolution kernel is enhanced in ΔK, the vertical edge response is suppressed, and the deformation misjudgment of the repair area is reduced. Among them, ΔK is the convolution kernel offset, is the basic convolution kernel, which is obtained through pre-training.

[0021] Material Enhancement Feature Map =Conv( , ); Conv means convolution.

[0022] Furthermore, step S6 further includes the following steps:

[0023] Directly used as the first convolution kernel of S4-ResNet;

[0024] Material Enhancement Feature Map Input the ResNet-34 backbone network to extract multi-scale feature maps;

[0025] The extracted feature map is then connected to a gradient reversal layer to decouple the feature distribution from the material properties;

[0026] Two-branch adversarial training:

[0027] Branch 1, crack classifier, outputs the probability distribution of crack / repair / normal;

[0028] Branch 2, the material property discriminator, constrains the repair features in the highly elastic material region to not be classified as cracks.

[0029] Secondly, the asphalt pavement damage visual analysis system integrated with deep learning includes a raw data acquisition module, a local material property query module, a module for capturing cross-modal correlations, a fusion feature map module, a material enhancement feature map module, and a discrimination module.

[0030] An original data acquisition module acquires road surface RGB image data and thermal image data through a vehicle-mounted multi-spectral camera, acquires position data through a GPS module, and acquires real-time environmental parameters through an environmental sensor.

[0031] A local material attribute query module is configured to query a pre-built material knowledge graph database according to the position data to obtain local asphalt material attributes, including an expansion coefficient and an elastic modulus; and the material knowledge graph database stores a mapping relationship between a region coordinate and asphalt material attributes in the region.

[0032] A cross-modal correlation capturing module is configured to concatenate an RGB image feature map and a thermal image feature map along a channel direction to a fusion feature map, perform local feature extraction on the concatenated fusion feature map by convolution, capture cross-modal local spatial correlation, and form an attention weight map Att by using a Sigmoid activation function on a correlation map output by the convolution.

[0033] A fusion feature map module is configured to construct a thermal image weight expression , calculate thermal image weights and RGB image weights based on the environmental parameters by using the thermal image weight expression, and construct a fusion feature map .

[0034] A material enhancement feature map module is configured to concatenate the elastic modulus E and the thermal expansion coefficient αT in the obtained local asphalt material attributes into a conditional vector [E, αT]; and to generate a material enhancement feature map with enhanced material features by adaptively adjusting convolution kernel parameters based on the fusion feature map and the conditional vector by using a MAML algorithm .

[0035] A discrimination module is configured to input the material enhancement feature map to a domain adaptive classifier to perform damage recognition, the classifier combines an adversarial training to constrain the correlation between material attributes and image features, and outputs a discrimination result of a repair area and a crack.

[0036] Further, in the cross-modal correlation capturing module, specifically, the pixels of the RGB image and the pixels of the thermal image are normalized between 0 and 1, respectively, to obtain an RGB image feature map and a thermal image feature map .

[0037] The expression of the attention weight map is: ; wherein, is a Sigmoid function, and Conv represents convolution.

[0038] Further, in the fusion feature map module, the rainfall intensity, the illumination intensity, and the ground temperature are normalized to 0-1 scalars, respectively; the thermal image weight ;Where P is the rainfall intensity scalar, L is the light intensity scalar, α, β, γ are preset coefficients, coefficients α=0.5, β=2.0, γ=0.3;

[0039] =( ·Att)· +(1- ·Att)· ; By combining thermal image weights and cross-modal attention maps The multiplication and superposition control dynamically allocates the contribution weights of RGB and thermal image data in the spatial dimension, strengthens the temperature difference characteristics in the reliable area of ​​the thermal image to suppress the misjudgment of the repair area, and finally generates a fusion feature map that is resistant to environmental interference. .

[0040] The beneficial effects of the present invention are:

[0041] 1. By fusing RGB image data with thermal image data and using an attention mechanism (such as a sigmoid activation function to generate an attention weight map) to capture cross-modal local spatial correlations (for example, the temperature difference characteristics between the repair area and the crack), the model's ability to distinguish between repair areas and cracks is enhanced, thereby reducing misjudgments and improving damage identification accuracy.

[0042] 2. By dynamically calculating the thermal image weight based on environmental parameters (such as rainfall intensity and light intensity) (for example, forcibly reducing the thermal image weight in heavy rain and relying mainly on RGB data), and constructing a fusion feature map that is resistant to environmental interference, it is possible to suppress noise interference in thermal images in severe weather, enhance model robustness, and support all-weather applications.

[0043] 3. By querying local asphalt material properties (such as elastic modulus and thermal expansion coefficient), and combining these properties with meta-learning algorithms (such as MAML) to adaptively adjust convolution kernel parameters (for example, enhancing transverse texture weights and suppressing longitudinal edge responses for highly elastic materials), it is possible to reduce cross-regional misjudgments caused by regional material differences (such as "creep lines" in Southeast Asian patching areas). BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a logical diagram of the asphalt pavement damage visual analysis method integrating deep learning in the present invention. DETAILED DESCRIPTION

[0045] In order to make the contents of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0046] As shown in the attached figure, the asphalt pavement damage visual analysis method integrating deep learning includes the following steps:

[0047] Step S1: Acquire RGB image data and thermal image data of the road surface through the vehicle-mounted multispectral camera; acquire location data through the GPS module; and acquire real-time environmental parameters through the environmental sensor.

[0048] The multispectral camera's multi-band acquisition capability is the hardware foundation for its simultaneous acquisition of RGB image data and thermal imaging data. RGB images capture visible light characteristics of asphalt pavement, such as texture and color, while thermal imaging data records the temperature distribution of the asphalt pavement. Environmental parameters include rainfall intensity, light intensity, and surface temperature. The corresponding environmental sensors are a rain gauge, a light intensity sensor, and a non-contact infrared thermometer, used to assess the degree of environmental interference. Location data provides a spatial positioning basis for subsequent queries of regional material properties. For example, a road under cloudy weather with a light intensity of 0.4 (normalized value) and GPS coordinates of [60.128, 18.643] was captured. The thermal image showed an average temperature of 28°C in the repair area and 21°C at the crack.

[0049] Specifically, according to the light intensity scalar Adjust HDR parameters, when The three-level exposure synthesis is enabled when the image is in the same scene; Mask R-CNN is used to segment the road area, generate ROI masks, filter out non-road background noise, and preprocess the RGB image data.

[0050] Step S2: querying a pre-built material knowledge graph library based on the location data to obtain local asphalt material properties, wherein the local asphalt material properties include expansion coefficient and elastic modulus; the material knowledge graph library stores a mapping relationship between regional coordinates and the asphalt material properties of the region.

[0051] Among other things, acquiring local asphalt material properties aims to reduce the probability of patch areas (i.e., early repair areas for asphalt pavement cracks) being misidentified as cracks. This is because asphalt material properties vary significantly across regions due to differences in climate, traffic demands, and construction standards. Northern Europe primarily uses modified asphalt, which, to adapt to low-temperature environments, has high hardness and strong crack resistance, resulting in stable physical properties in the patch area. Southeast Asia, on the other hand, uses rubber asphalt, which is highly elastic and ductile in response to high temperatures and high rainfall. The patch area is prone to developing "creep marks" due to temperature changes, and its thermal expansion coefficient differs significantly from that of Nordic materials. The deformation patterns formed by the highly elastic and easily deformed rubber asphalt in Southeast Asia at high temperatures conflict with the patterns learned by existing domestic models, making creep marks in the patch area easily misidentified as cracks.

[0052] Step S3: RGB image feature map Thermal imaging characteristics The fused feature map is spliced ​​along the channel direction, and local features are extracted from the spliced ​​fused feature map using convolution to capture local spatial correlation across modalities. The association map output by the convolution is converted into an attention weight map Att using the Sigmoid activation function.

[0053] Specifically, the pixels of the RGB image and the thermal image are normalized between [0, 1] to obtain the RGB image feature map. , Thermal image characteristic diagram ;

[0054] The expression of the attention weight map is: ;in, is the Sigmoid function, and Conv represents convolution.

[0055] Step S4: construct a thermal image weight expression, calculate the thermal image weight and the RGB image weight based on the environmental parameters using the thermal image weight expression, and construct a fusion feature map :

[0056] =( · ) +(1- · )· ; By combining the thermal image weight ( ) and the cross-modal attention map ( )'s multiplication and superposition control ( ), dynamically allocates the contribution weights of RGB and thermal image data in the spatial dimension, strengthens the temperature difference characteristics in the reliable thermal image area to suppress the misjudgment of the repair area, and weakens the environmental noise interference by reducing the thermal image weight in bad weather, and finally generates a fusion feature map that is resistant to environmental interference , providing high-reliability input for subsequent classification and reducing the misjudgment rate from the source.

[0057] The preset threshold is set based on the rainfall intensity detected by the sensor (e.g., when the rainfall intensity is > 0.6, the thermal image distortion rate is > 40%), and the dynamic weight balances the modal contribution under different environments. Example: When the rainfall intensity scalar P = 0.7 (heavy rain), the forced thermal image weight is =0.2, to avoid the temperature noise caused by rain evaporation interfering with classification.

[0058] Specifically, rainfall intensity, light intensity, and surface temperature are normalized to 0-1 scalars respectively; thermal image weight expression ; Where P is the rainfall intensity scalar, L is the light intensity scalar, α, β, γ are preset coefficients; for example, when P>0.6 (rainstorm threshold), it is forced to set [0.1, 0.3]; the RGB image and thermal image data are weighted and fused to generate fusion data resistant to environmental interference. This step quantifies environmental factors. The coefficients a = 0.5, b = 2.0, and g = 0.3 are optimized through calibration experiments, and the rainstorm threshold 0.6 corresponds to the heavy rain level defined by the meteorological bureau. Example: when P = 0.2 (light rain) and L = 0.8 (noon), the calculation = 0.52, the fusion data retains effective thermal features; when P = 0.7, the calculation = 0.1, mainly relies on RGB data.

[0059] Step S5, the obtained elastic modulus E and thermal expansion coefficient aT in the local asphalt material properties are spliced as a conditional vector [E, aT]; based on the fusion feature map and the conditional vector, the MAML algorithm is used to adaptively adjust the convolution kernel parameters, and a material feature enhanced material enhanced feature map is generated.

[0060] Specifically, the conditional vector is mapped to the convolution kernel offset amount DK through a two-layer fully connected network; DK = FC2(ReLU(FC1([E, aT]))); where FC1 and FC2 are the first fully connected layer and the second connected layer, respectively; and ReLU is an activation function.

[0061] The first layer convolution kernel is: = + DK; for high-elasticity materials (E < 1.0 GPa), the transverse texture convolution kernel weight is enhanced in DK, the longitudinal edge response is suppressed, and the deformation misjudgment of the repair area is reduced. Where DK is the convolution kernel offset amount, is the basic convolution kernel, which is obtained through pre-training. The specific training process is as follows: an RGB and thermal image fusion dataset containing diversified environments (different light, temperature, and rainfall) and regions (different material properties) is used; a basic convolution network is trained to perform a crack / repair / normal region classification task. After training, the first layer convolution kernel weight of the network is directly extracted as .

[0062] = Conv( , ); where Conv represents convolution.

[0063] Step S6, the material enhanced feature map is input into a domain adaptive classifier for damage recognition, and the classifier combines the association between material properties and image features through adversarial training, and outputs the discrimination results of the repair area and the crack.

[0064] Further, step S6 further includes the following steps:

[0065] Directly used as the first convolution kernel of S4-ResNet;

[0066] Material Enhancement Feature Map Input the ResNet-34 backbone network to extract multi-scale feature maps;

[0067] The extracted feature map is then connected to a gradient inversion layer to decouple the feature distribution from the material properties;

[0068] Two-branch adversarial training:

[0069] Branch 1, crack classifier, outputs the probability distribution of crack / repair / normal;

[0070] Branch 2, material property discriminator, constrains the repair features of high elastic material areas not to be classified as cracks;

[0071] Furthermore, the loss function L= +λ , which decays dynamically with the training rounds; specifically, the initial weight λ is set to 1.0, and the crack classification branch is trained synchronously ( ) and the material identification branch ( ), forcing the feature extractor to strip away material correlations (such as the texture features of highly elastic asphalt in Southeast Asia); multiplying λ by 0.9 every 10 training rounds, gradually reducing the adversarial loss weight (λ≈0.4 after 100 rounds), so that the model can focus on improving damage classification accuracy in the later stage; retaining material features for discriminator calculations in the forward propagation through the gradient reversal layer (GRL) , inverting the sign of the gradient during backpropagation, forcing the feature distribution to be decoupled from the material.

[0072] In summary, by fusing RGB and thermal image data and utilizing an attention mechanism (e.g., a sigmoid activation function to generate an attention weight map) to capture cross-modal local spatial correlations (e.g., temperature differences between repaired areas and cracks), the model's ability to distinguish between repaired areas and cracks is enhanced, thereby reducing misclassifications and improving damage identification accuracy. By dynamically calculating thermal image weights based on environmental parameters (e.g., rainfall intensity and light intensity) (e.g., forcibly reducing thermal image weights during heavy rain and relying primarily on RGB data), and constructing a fused feature map that is resilient to environmental interference, the model suppresses noise interference from thermal images in inclement weather, enhancing model robustness and supporting all-weather applications. By querying local asphalt material properties (e.g., elastic modulus and thermal expansion coefficient) and incorporating these properties to adaptively adjust convolution kernel parameters (e.g., increasing transverse texture weights and suppressing longitudinal edge responses for highly elastic materials), the model mitigates cross-regional misclassifications caused by regional material differences (e.g., "creep marks" in Southeast Asian repair areas).

[0073] Based on the same inventive concept as the aforementioned asphalt pavement damage visual analysis method integrated with deep learning, the present embodiment provides an asphalt pavement damage visual analysis system integrated with deep learning, including:

[0074] The raw data acquisition module obtains road surface RGB image data and thermal image data through the on-board multispectral camera; obtains location data through the GPS module; and obtains real-time environmental parameters through environmental sensors.

[0075] The multispectral camera's multi-band acquisition capability is the hardware foundation for its simultaneous acquisition of RGB image data and thermal imaging data. RGB images capture visible light characteristics of asphalt pavement, such as texture and color, while thermal imaging data records the temperature distribution of the asphalt pavement. Environmental parameters include rainfall intensity, light intensity, and surface temperature. The corresponding environmental sensors are a rain gauge, a light intensity sensor, and a non-contact infrared thermometer, used to assess the degree of environmental interference. Location data provides a spatial positioning basis for subsequent queries of regional material properties. For example, a road under cloudy weather with a light intensity of 0.4 (normalized value) and GPS coordinates of [60.128, 18.643] was captured. The thermal image showed an average temperature of 28°C in the repair area and 21°C at the crack.

[0076] Specifically, according to the light intensity scalar Adjust HDR parameters, when The three-level exposure synthesis is enabled when the image is in the same scene; Mask R-CNN is used to segment the road area, generate ROI masks, filter out non-road background noise, and preprocess the RGB image data.

[0077] The local material property query module is used to query the pre-built material knowledge graph library based on the location data to obtain local asphalt material properties, wherein the local asphalt material properties include expansion coefficient and elastic modulus; the material knowledge graph library stores the mapping relationship between regional coordinates and the asphalt material properties of the region.

[0078] Among other things, acquiring local asphalt material properties aims to reduce the probability of patch areas (i.e., early repair areas for asphalt pavement cracks) being misidentified as cracks. This is because asphalt material properties vary significantly across regions due to differences in climate, traffic demands, and construction standards. Northern Europe primarily uses modified asphalt, which, to adapt to low-temperature environments, has high hardness and strong crack resistance, resulting in stable physical properties in the patch area. Southeast Asia, on the other hand, uses rubber asphalt, which is highly elastic and ductile in response to high temperatures and high rainfall. The patch area is prone to developing "creep marks" due to temperature changes, and its thermal expansion coefficient differs significantly from that of Nordic materials. The deformation patterns formed by the highly elastic and easily deformed rubber asphalt in Southeast Asia at high temperatures conflict with the patterns learned by existing domestic models, making creep marks in the patch area easily misidentified as cracks.

[0079] Capturing cross-modal correlation module, transforming RGB image feature map Thermal imaging characteristics The fused feature map is spliced ​​along the channel direction, and local features are extracted from the spliced ​​fused feature map using convolution to capture local spatial correlation across modalities. The association map output by the convolution is converted into an attention weight map Att using the Sigmoid activation function.

[0080] Specifically, the pixels of the RGB image and the thermal image are normalized between [0, 1] to obtain the RGB image feature map. , Thermal image characteristic diagram ;

[0081] The expression of the attention weight map is: ;in, is the Sigmoid function, and Conv represents convolution.

[0082] Fusion feature map module to construct thermal image weight expression , based on the environmental parameters, the thermal image weight is calculated using the thermal image weight expression and RGB image weights, and construct fusion feature maps :

[0083] =( ·Att)· +(1- ·Att)· ; By combining the thermal image weight ( ) and the cross-modal attention map ( )'s multiplication and superposition control ( ), dynamically allocates the contribution weights of RGB and thermal image data in the spatial dimension, strengthens the temperature difference characteristics in the reliable thermal image area to suppress the misjudgment of the repair area, and weakens the environmental noise interference by reducing the thermal image weight in bad weather, and finally generates a fusion feature map that is resistant to environmental interference , providing high-reliability input for subsequent classification and reducing the misjudgment rate from the source.

[0084] The preset threshold is set based on the rainfall intensity detected by the sensor (e.g., when the rainfall intensity is > 0.6, the thermal image distortion rate is > 40%), and the dynamic weight balances the modal contribution under different environments. Example: When the rainfall intensity scalar P = 0.7 (heavy rain), the forced thermal image weight is =0.2, to avoid the temperature noise caused by rain evaporation interfering with classification.

[0085] Specifically, the rainfall intensity, light intensity, and surface temperature are normalized to 0-1 scalars respectively; thermal image weight ; wherein P is a rainfall intensity scalar, L is an illumination intensity scalar, a, b, g are preset coefficients; for example, when P>0.6 (rainstorm threshold), the RGB image is forcibly set as ∈[0.1,0.3]; the RGB image and the thermal image data are fused by weights and 1- to generate fusion data resistant to environmental interference. This step quantifies environmental factors. The coefficients a=0.5, b=2.0, g=0.3 are optimized through calibration experiments, and the rainstorm threshold 0.6 corresponds to the heavy rain level defined by the meteorological bureau. Example: when P=0.2 (light rain), L=0.8 (noon), calculate =0.52, the fusion data retains effective thermal features; when P=0.7, forcibly =0.1, mainly relying on RGB data.

[0086] The material enhancement feature map module uses the obtained elastic modulus E and thermal expansion coefficient aT in the local asphalt material properties to splice as a condition vector [E, aT]; based on the fusion feature map and the condition vector, the MAML algorithm is used to adaptively adjust the convolution kernel parameters to generate a material feature enhancement material enhancement feature map .

[0087] Specifically, the condition vector is mapped to the convolution kernel offset amount DK through a two-layer fully connected network; DK=FC2(ReLU(FC1([E, aT]))); wherein FC1 and FC2 are the first fully connected layer and the second connected layer respectively; ReLU is an activation function.

[0088] The first layer convolution kernel is: = +DK; for high-elasticity materials (E<1.0GPa), the transverse texture convolution kernel weight is enhanced in DK, the longitudinal edge response is suppressed, and the deformation misjudgment of the repair area is reduced. Wherein, DK is the convolution kernel offset amount, is the basic convolution kernel, which is obtained through pre-training.

[0089] =Conv( , ); wherein Conv represents convolution.

[0090] The discrimination module inputs the material enhancement feature map to the domain adaptive classifier for damage recognition, the classifier combines the correlation between material properties and image features through adversarial training, and outputs the discrimination results of the repair area and the crack.

[0091] Further, step S6 further includes the following steps:

[0092] Directly as the first layer of S4-ResNet convolution kernel

[0093] Material enhanced feature map Input ResNet-34 backbone network, extract multi-scale feature map;

[0094] The extracted feature map is connected to the gradient reversal layer, which decouples the feature distribution from the material attribute;

[0095] Dual-branch adversarial training:

[0096] Branch 1, crack classifier, output crack / repair / normal probability distribution;

[0097] Branch 2, material attribute discriminator, constrain the repair features of high-elasticity material region not to be classified as cracks;

[0098] Further, the loss function L= +λ , where λ is dynamically attenuated with training rounds; Specifically, set the initial weight λ=1.0, train the crack classification branch ( ) and the material discrimination branch ( ) synchronously, force the feature extractor to strip the material correlation (such as the texture features of Southeast Asian high-elasticity asphalt); Multiply λ by 0.9 every 10 training rounds, gradually reduce the adversarial loss weight (λ≈0.4 after 100 rounds), so that the model focuses on improving the damage classification accuracy in the later stage; Through the gradient reversal layer (GRL), the material features are preserved for discriminator calculation in forward propagation , and the gradient sign is reversed during back propagation, forcing the feature distribution to be decoupled from the material.

[0099] In summary, by fusing RGB image data and thermal image data, and using attention mechanism (such as Sigmoid activation function to generate attention weight map) to capture cross-modal local spatial correlation (for example, the temperature difference features of repair area and cracks), the enhanced model can distinguish between repair areas and cracks, reducing misjudgment and improving damage recognition accuracy. By dynamically calculating the thermal image weight based on environmental parameters (such as rainfall intensity, light intensity) (for example, forcing to reduce the thermal image weight in heavy rain and mainly relying on RGB data), and constructing an anti-environmental interference fusion feature map, the noise interference of thermal images in bad weather is suppressed, the model robustness is enhanced, and the full-weather application effect is supported. By querying the local asphalt material properties (such as elastic modulus and thermal expansion coefficient), and combining these properties to adaptively adjust the convolution kernel parameters (for example, increasing the horizontal texture weight for high-elasticity materials and suppressing the vertical edge response), the cross-regional misjudgment effect caused by regional material differences (such as Southeast Asian repair area "creep lines") is reduced.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A visual analysis method for asphalt pavement damage integrating deep learning, characterized by: The following steps are involved: Step S1, obtaining RGB image data and thermal image data of the road surface through the vehicle-mounted multispectral camera; obtaining location data through the GPS module; and obtaining real-time environmental parameters through the environmental sensor; Step S2: querying a pre-built material knowledge graph based on the location data to obtain local asphalt material properties, wherein the local asphalt material properties include expansion coefficient and elastic modulus; the material knowledge graph stores a mapping relationship between regional coordinates and the asphalt material properties of the region; Step S3: RGB image feature map Thermal image characteristic diagram The fused feature map is obtained by splicing along the channel direction, and local features are extracted from the spliced ​​fused feature map using convolution to capture the local spatial correlation across modalities. The association map output by the convolution is converted into an attention weight map Att using the Sigmoid activation function; Step S4: construct a thermal image weight expression, calculate the thermal image weight and the RGB image weight based on the environmental parameters using the thermal image weight expression, and obtain the fusion feature map based on the constructed fusion feature map expression. ; Step S5, using the elastic modulus E and thermal expansion coefficient αT obtained from the local asphalt material properties to form a conditional vector [E, αT]; Based on the fusion feature map and conditional vector, the convolution kernel parameters are adaptively adjusted through the MAML algorithm to generate a material enhancement feature map with enhanced material features. ; Step S6: Enhance the material feature map The input is fed into a domain adaptive classifier for damage identification. The classifier combines the correlation between material properties and image features constrained by adversarial training to output the discrimination results between repair areas and cracks. The conditional vector is mapped to the convolution kernel offset through a two-layer fully connected network, ΔK=FC2(ReLU(FC1([E,αT]))); FC1 and FC2 are the first fully connected layer and the second connected layer respectively; ReLU is the activation function; The first convolution kernel is: = +ΔK; For highly elastic materials, the weight of the horizontal texture convolution kernel is enhanced in ΔK, the vertical edge response is suppressed, and the deformation misjudgment of the repair area is reduced; where ΔK is the convolution kernel offset, is the basic convolution kernel, which is obtained through pre-training; Material Enhancement Feature Map =Conv( , ); Conv means convolution.

2. The asphalt pavement damage visual analysis method integrated with deep learning according to claim 1 is characterized in that: Specifically, according to the light intensity scalar Adjust HDR parameters, when The three-level exposure synthesis is enabled when the image is in the same scene; Mask R-CNN is used to segment the road area, generate ROI masks, filter out non-road background noise, and preprocess the RGB image data.

3. The asphalt pavement damage visual analysis method integrated with deep learning according to claim 1, wherein step S3 is characterized in that: Normalize the pixels of RGB image and thermal image to [0,1] respectively, and then obtain the RGB image feature map , Thermal image characteristic diagram ; The expression of the attention weight map is: ;in, is the Sigmoid function, and Conv means convolution.

4. The asphalt pavement damage visual analysis method integrated with deep learning according to claim 1 is characterized in that: The fusion feature map expression is: =( ·Att)· +(1- ·Att)· ; Combined with thermal image weight and cross-modal attention maps The multiplication and superposition control dynamically allocates the contribution weights of RGB and thermal image data in the spatial dimension, and finally generates a fusion feature map that is resistant to environmental interference. .

5. The asphalt pavement damage visual analysis method integrated with deep learning according to claim 4 is characterized in that: Normalize rainfall intensity, light intensity, and surface temperature to 0-1 scalars respectively; thermal image weight ; Where P is the rainfall intensity scalar, L is the light intensity scalar, α, β, γ are preset coefficients; after calibration experiments, the optimized coefficients α=0.5, β=2.0, γ=0.

3.

6. The asphalt pavement damage visual analysis method integrated with deep learning according to claim 1 is characterized in that: Step S6 further includes the following steps: Directly used as the first layer convolution kernel of S4-ResNet; Material Enhancement Feature Map Input the ResNet-34 backbone network to extract multi-scale feature maps; The extracted feature map is then connected to a gradient reversal layer to decouple the feature distribution from the material properties; Two-branch adversarial training: Branch 1, crack classifier, outputs the probability distribution of crack / repair / normal; Branch 2, the material property discriminator, constrains the repair features of highly elastic material regions to not be classified as cracks.

7. Asphalt pavement damage visual analysis system integrating deep learning, characterized by: It includes raw data acquisition module, local material property query module, cross-modal correlation capture module, fusion feature map module, material enhancement feature map module, and discrimination module; The raw data acquisition module acquires RGB image data and thermal image data of the road surface through the vehicle-mounted multispectral camera; obtains location data through the GPS module; and obtains real-time environmental parameters through environmental sensors; A local material property query module is used to query a pre-built material knowledge graph based on the location data to obtain local asphalt material properties, wherein the local asphalt material properties include expansion coefficient and elastic modulus; the material knowledge graph stores the mapping relationship between regional coordinates and the asphalt material properties of the region; Capturing cross-modal correlation module, transforming RGB image feature map Thermal image characteristic diagram The fused feature map is obtained by splicing along the channel direction, and local features are extracted from the spliced ​​fused feature map using convolution to capture the local spatial correlation across modalities. The association map output by the convolution is converted into an attention weight map Att using the Sigmoid activation function; Fusion feature map module to construct thermal image weight expression , based on the environmental parameters, the thermal image weight and the RGB image weight are calculated using the thermal image weight expression, and a fusion feature map is constructed ; The material enhancement feature map module uses the elastic modulus E and thermal expansion coefficient αT obtained from the local asphalt material properties to form a conditional vector [E, αT]; Based on the fusion feature map and conditional vector, the convolution kernel parameters are adaptively adjusted through the MAML algorithm to generate a material enhancement feature map with enhanced material features. ; The discrimination module enhances the material feature map The input is fed into a domain adaptive classifier for damage identification. The classifier combines the correlation between material properties and image features constrained by adversarial training to output the discrimination results between repair areas and cracks. The conditional vector is mapped to the convolution kernel offset through a two-layer fully connected network, ΔK=FC2(ReLU(FC1([E,αT]))); FC1 and FC2 are the first fully connected layer and the second connected layer respectively; ReLU is the activation function; The first convolution kernel is: = +ΔK; For highly elastic materials, the weight of the horizontal texture convolution kernel is enhanced in ΔK, the vertical edge response is suppressed, and the deformation misjudgment of the repair area is reduced; where ΔK is the convolution kernel offset, is the basic convolution kernel, which is obtained through pre-training; Material Enhancement Feature Map =Conv( , ); Conv means convolution.

8. The asphalt pavement damage visual analysis system integrated with deep learning according to claim 7, characterized in that: In the module for capturing cross-modal correlation, specifically, the pixels of the RGB image and the thermal image are normalized between [0, 1] to obtain the RGB image feature map. , Thermal image characteristic diagram ; The expression of the attention weight map is: ;in, is the Sigmoid function, and Conv means convolution.

9. The asphalt pavement damage visual analysis system integrated with deep learning according to claim 7, characterized in that: In the fusion feature map module, rainfall intensity, light intensity, and surface temperature are normalized to 0-1 scalars respectively; thermal image weight ;Where P is the rainfall intensity scalar, L is the light intensity scalar, α, β, γ are preset coefficients, coefficients α=0.5, β=2.0, γ=0.3; =( ·Att)· +(1- ·Att)· ; By combining thermal image weights and cross-modal attention maps The multiplication and superposition control dynamically allocates the contribution weights of RGB and thermal image data in the spatial dimension, strengthens the temperature difference characteristics in the reliable area of ​​the thermal image to suppress the misjudgment of the repair area, and finally generates a fusion feature map that is resistant to environmental interference. .

Images