Road crack danger grade evaluation method and system based on graph neural network large model

Abstract

The application discloses a road crack danger level evaluation method and system based on a graph neural network large model, and relates to the technical field of artificial intelligence. The method comprises the following steps: inputting historical fusion features into a graph neural network model for training to obtain an optimal graph neural network model; acquiring real-time fusion features, inputting the real-time fusion features into the optimal graph neural network model to obtain a road crack detection result of the real-time fusion features, and obtaining discrete point clouds in a road crack region based on the road crack detection result; regarding the discrete point clouds as a group of discrete sampling points, constructing a continuous surface model through a triangulation algorithm, calculating geometric parameters of road cracks based on the continuous surface model, calculating a danger index of the road cracks based on the geometric parameters, and obtaining a road crack danger level evaluation result based on the graph neural network model based on the danger index, which helps to solve the problem that the prior art cannot efficiently and accurately evaluate the road crack danger level.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a road crack danger level evaluation method and system based on a graph neural network large model. BACKGROUND

[0002] With the acceleration of urbanization and the continuous growth of traffic flow, the safety and maintenance of road infrastructure have attracted increasing attention. If road cracks are not treated in time, they will lead to the deterioration of the road structure, seriously threaten road safety, and greatly increase maintenance costs. Therefore, efficiently and accurately evaluating the danger level of road cracks is crucial to ensuring road safety and reducing maintenance costs.

[0003] However, traditional convolutional neural networks (CNN) still have significant bottlenecks in practical applications. On the one hand, most detection algorithms rely on a single RGB image, which is easily disturbed by complex environmental factors such as changes in light, rain and fog, and low light at night. The detection accuracy can decrease by 30%-50% under extreme light, and the false detection rate exceeds 40% in rain and fog. Moreover, the detection effect varies greatly in different scenarios, and the universality is insufficient. On the other hand, for fine cracks, traditional CNNs have difficulty achieving high recall rate and low false detection rate due to fixed convolution kernels, network structure limitations, and training data sample bias.

[0004] In addition, mainstream detection models such as Faster R-CNN and YOLOv4 have large parameter quantities and high computational costs, making it difficult to deploy on edge devices such as vehicles and drones, and unable to meet real-time detection requirements.

[0005] In terms of multi-modal data fusion, although integrating infrared thermal imaging and laser radar point cloud data can improve detection performance, different modal data has problems such as temporal and spatial inconsistency and large feature representation differences. Existing fixed weight allocation strategies cannot adapt to dynamic environmental changes, affecting detection effectiveness. At the same time, existing detection methods are mostly limited to two-dimensional image analysis, lacking the ability to quantitatively evaluate the three-dimensional structural features of cracks, and risk assessment relies heavily on manual judgment, making it difficult to achieve automation and refined management.

[0006] Therefore, there is an urgent need for a method that can efficiently and accurately evaluate the danger level of road cracks. SUMMARY

[0007] Therefore, the present application proposes a road crack danger level evaluation method and system based on a graph neural network large model, which can realize data security isolation and privacy protection, and ensure the safety and compliance of data in the entire process.

[0008] To achieve the above-mentioned purposes, the present application provides the following technical solutions:

[0009] A road crack danger level evaluation method based on a graph neural network large model, comprising:

[0010] Collecting multi-modal historical data, the multi-modal historical data comprising RGB modal data, infrared modal data and laser modal data;

[0011] Performing data processing on the multi-modal historical data to obtain preprocessed data, performing spatio-temporal alignment on the preprocessed data, inputting the spatio-temporal aligned preprocessed data into a self-attention mechanism to assign weights to different modal data, and performing weighted splicing on different modal data according to the weights to obtain historical fusion features;

[0012] Constructing a graph neural network model, the graph neural network model comprising an input layer, a dynamic graph convolution module, a dynamic edge weight learning module, a three-layer graph convolution module, a lightweight optimization module and an output layer, the input layer being used to receive the historical fusion features, the dynamic graph convolution module being used to construct a graph structure based on the historical fusion features, the graph structure comprising nodes and edges, the dynamic edge weight learning module being used to dynamically adjust the weights of edges in the graph structure in a training process by using a learnable similarity measurement function, the three-layer graph convolution module being used to perform feature fusion on adjacent nodes in the graph structure, the lightweight optimization module being used to perform lightweight processing on the graph neural network model by using channel pruning and weight binarization, and the output layer being used to output a road crack detection result;

[0013] Inputting the historical fusion features into the graph neural network model for training to obtain an optimal graph neural network model;

[0014] Obtaining real-time fusion features, inputting the real-time fusion features into the optimal graph neural network model to obtain a road crack detection result of the real-time fusion features, and obtaining discrete point clouds in a road crack region based on the road crack detection result;

[0015] Regarding the discrete point clouds as a group of discrete sampling points, constructing a continuous surface model by using a triangulation algorithm, calculating geometric parameters of road cracks based on the continuous surface model, calculating a danger index of the road cracks based on the geometric parameters, and obtaining a road crack danger level evaluation result based on the graph neural network model based on the danger index.

[0016] On the basis of the above technical solution, the application can also be improved as follows:

[0017] Optionally, the spatio-temporal alignment of the preprocessed data and the input of the spatio-temporally aligned preprocessed data into the self-attention mechanism to assign weights to different modal data comprises:

[0018] The weight coefficient of the infrared modal data is calculated by formula (1);

[0019] w IR = Sigmoid(k1 I light +k2 H humidity ) Formula (1);

[0020] In the formula, w IR is a weight coefficient of infrared modal data, Sigmoid is an activation function, k1 is a learnable parameter, I light is an illumination intensity, k2 is a learnable parameter, and H humidity is humidity.

[0021] The signal-to-noise ratio of the RGB modal data is calculated by formula (2);

[0022]

[0023] In the formula, SNR RGB is the signal-to-noise ratio of the RGB modal data, μ texture is the mean of the texture features in the RGB modal data, and σ noise is the standard deviation of the noise in the RGB modal data.

[0024] Optionally, the dynamic edge weight learning module dynamically adjusts the weight of the edge in the graph structure in the training process by using a learnable similarity measurement function, including:

[0025] The weight of the edge between node i and node j in the graph structure is calculated by formula (3);

[0026]

[0027] In the formula, A ij is the weight of the edge between node i and node j in the graph structure, Softmax is an activation function, φ is an independent linear transformation layer, ψ is an independent linear transformation layer, v i is the feature vector of node i in V j is the feature vector of node j, and d is the dimension of the feature vector after linear transformation.

[0028] Optionally, the three-layer graph convolution module is used for feature fusion of adjacent nodes in the graph structure, including:

[0029] The node feature matrix output by the l+1th layer of the graph convolution network is calculated by formula (4);

[0030]

[0031] In the formula, H (l+1) is the node feature matrix output by the l+1th layer of the graph convolution network, LayerNorm is a layer normalization operation, σ is a Swish activation function, A is an adjacency matrix, and H(l) is a node feature matrix of the l-th layer graph convolution network, W (l) is a trainable weight matrix of the l-th layer graph convolution network.

[0032] Optionally, the lightweight optimization module adopts channel pruning and weight binarization to perform lightweight processing on the graph neural network model, comprising:

[0033] The loss function of structured pruning is calculated by formula (5);

[0034]

[0035] In the formula, is the loss function of structured pruning, λ is a hyperparameter, W c is the weight corresponding to the c-th channel, C is the number of channels, ||W c ||1 is the L1 norm of the c-th channel weight W c ;

[0036] The binarized weight parameter is calculated by formula (6);

[0037]

[0038] In the formula, W binary is the binarized weight parameter, Sign is the sign function, W float is the weight parameter originally stored in the network in the form of a floating-point number, is the loss function, is the partial derivative, is the small change amount of the loss function L.

[0039] Optionally, the history fusion feature is input into the graph neural network model for training to obtain an optimal graph neural network model, comprising:

[0040] The parameters of the teacher network are calculated by formula (7);

[0041] θ teacher = m·θ teacher +(1-m)·θ student Formula (7);

[0042] In the formula, θ teacher is the parameter of the teacher network, m is the momentum coefficient, θ student is the parameter of the student network, m is the momentum coefficient;

[0043] The loss function of contrastive learning is calculated by formula (8);

[0044]

[0045] In the formula, is the loss function of the contrastive learning, s p is the cosine similarity of the positive sample pair, s n is the cosine similarity of the negative sample pair, τ is the temperature parameter, and N is the number of negative sample pairs.

[0046] The loss function of the elastic weight solidification method is calculated by formula (9);

[0047]

[0048] In the formula, is the loss function of the elastic weight solidification method, is the index of the model parameter, F i is the value of the corresponding parameter i in the Fisher information matrix, θ i is the value of parameter i in the current model training process, θ i，old is the value of parameter i at the end of the previous task training, i is the index of the model parameter.

[0049] Optionally, the geometric parameters of the road crack are calculated based on the continuous surface model, and the road crack evaluation result is obtained based on the geometric parameters, comprising:

[0050] The total volume of the crack is calculated by formula (10);

[0051]

[0052] In the formula, V is the total volume of the crack, N tri is the total number of triangular facets in the crack area, A t is the area of the tth triangular facet, is the average depth of the three vertices of the tth triangular facet;

[0053] The risk index is calculated by formula (11);

[0054]

[0055] In the formula, R is the risk index, L is the length of the crack, L max is the length threshold parameter, W max is the maximum width of the crack, W crit is the width threshold parameter, D is the depth of the crack, D crit is the depth threshold parameter.

[0056] A road crack risk level evaluation system based on a graph neural network large model, comprising:

[0057] The acquisition module is configured to acquire multi-modal historical data, the multi-modal historical data including RGB modal data, infrared modal data, and laser modal data.

[0058] a data processing module, configured to perform data processing on the multi-modal historical data to obtain preprocessed data, perform spatio-temporal alignment on the preprocessed data, input the spatio-temporally aligned preprocessed data into a self-attention mechanism to assign weights to different modal data, and perform weighted splicing on the different modal data according to the weights to obtain historical fusion features;

[0059] a model construction module, configured to construct a graph neural network model, the graph neural network model comprising an input layer, a dynamic graph convolution module, a dynamic edge weight learning module, a three-layer graph convolution module, a lightweight optimization module and an output layer, the input layer being configured to receive the historical fusion features, the dynamic graph convolution module being configured to construct a graph structure based on the historical fusion features, the graph structure comprising nodes and edges, the dynamic edge weight learning module being configured to dynamically adjust the weights of the edges in the graph structure in a training process by using a learnable similarity measurement function, the three-layer graph convolution module being configured to perform feature fusion on adjacent nodes in the graph structure, the lightweight optimization module being configured to perform lightweight processing on the graph neural network model by using channel pruning and weight binarization, and the output layer being configured to output a road crack detection result;

[0060] a training module, configured to input the historical fusion features into the graph neural network model for training to obtain an optimal graph neural network model;

[0061] a road crack detection module, configured to obtain real-time fusion features, input the real-time fusion features into the optimal graph neural network model to obtain a road crack detection result of the real-time fusion features, and obtain discrete point clouds in a road crack region based on the road crack detection result;

[0062] a road crack evaluation module, configured to regard the discrete point clouds as a group of discrete sampling points, construct a continuous surface model by using a triangulation algorithm, calculate a geometric parameter of a road crack based on the continuous surface model, calculate a danger index of the road crack based on the geometric parameter, and obtain a road crack danger level evaluation result based on the graph neural network model based on the danger index.

[0063] An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor implements the steps of the method when executing the computer program.

[0064] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method.

[0065] The present application has the following advantages:

[0066] The road crack danger level evaluation method based on the graph neural network large model in the application collects multi-modal historical data such as RGB modal data, infrared modal data and laser modal data, and through data processing, space-time alignment and self-attention mechanism weighted splicing, fully fuses the advantages of different modal data, effectively overcomes the problem that a single RGB image is disturbed by environmental factors such as light and weather, and can also stably obtain crack features in complex environments such as strong light, rain and fog, night and the like, and significantly improves the adaptability and robustness of the detection system to complex environments.

[0067] The road crack danger level evaluation method based on the graph neural network large model in the application constructs a graph neural network model, a dynamic graph convolution module can construct a graph structure based on fused features, a dynamic edge weight learning module can dynamically adjust edge weights, and three-layer graph convolution modules are combined to realize efficient fusion of adjacent node features, compared with a traditional convolutional neural network, the road crack danger level evaluation method based on the graph neural network large model can better capture diversified crack forms, especially in the detection of fine cracks and irregular forms, a lightweight optimization module is used for channel pruning and weight binarization, which greatly reduces the model parameter quantity and calculation complexity, reduces the demand for computing resources, enables the model to be quickly deployed on edge devices such as vehicles and unmanned aerial vehicles, meets the real-time detection and continuous monitoring requirements, reduces hardware investment and energy consumption costs, and improves the economy and practicability of the system.

[0068] The road crack danger level evaluation method based on the graph neural network large model in the application constructs a continuous surface model for crack area discrete point clouds through a triangulation algorithm, and then calculates geometric parameters such as crack depth, width and volume, combines the geometric parameter calculation with the danger index to determine the danger level, realizes the leap from two-dimensional plane to three-dimensional space, compared with traditional two-dimensional detection and artificial experience judgment, the road crack danger level evaluation method based on the graph neural network large model can more comprehensively and accurately evaluate crack risks, provides a scientific and quantitative basis for road maintenance decision-making, and promotes the intelligentization and automation of road safety management. BRIEF DESCRIPTION OF DRAWINGS

[0069] For the purpose of illustration and not limitation, the application will now be described in conjunction with embodiments and drawings in which:

[0070] Figure 1 A flowchart of the road crack danger level evaluation method based on the graph neural network large model in the embodiments of the application is shown;

[0071] Figure 2 A schematic diagram of main components of the road crack danger level evaluation system based on the graph neural network large model in the embodiments of the application is shown;

[0072] Figure 3 An electronic device entity structure schematic diagram provided by the application is shown. DETAILED DESCRIPTION

[0073] In order for those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0074] It should be noted that the terms "first", "second", and the like in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0075] It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict. The embodiments of the present application will be described in detail below in conjunction with the drawings.

[0076] Figure 1 The flowchart of the road crack danger level evaluation method based on the large model of the graph neural network in the embodiments of the present application is shown in FIG. 1, and the road crack danger level evaluation method based on the large model of the graph neural network provided by the embodiments of the present application includes the following steps S101 to S106. Figure 1

[0077] S101, collect multi-modal historical data, the multi-modal historical data including RGB modal data, infrared modal data and laser modal data.

[0078] The RGB modal data is collected by a high-definition camera, which can intuitively present the two-dimensional morphology of the road surface texture and cracks, but is easily disturbed by light, weather and other factors, and may appear image overexposure, blur and other problems in complex environments.

[0079] The infrared modal data is obtained by an infrared thermal imager, which is based on the principle of object thermal radiation imaging and is not limited by light conditions. In the night or in the rain and fog weather, the temperature difference can be used to clearly display the crack profile, providing supplementary information for detection.

[0080] The laser modal data is collected by a laser radar, which can obtain high-precision three-dimensional point cloud, accurately describe the crack depth and spatial distribution, and make up for the shortcomings of two-dimensional images. ​

[0081] By collecting these three modal data, the respective advantages can be fully played, and mutual complementation is achieved, thereby providing a comprehensive and reliable data basis for subsequent crack detection and risk level assessment.

[0082] In S102, the multi-modal historical data is subjected to data processing to obtain preprocessed data, the preprocessed data is subjected to spatio-temporal alignment, and the preprocessed data subjected to spatio-temporal alignment is input into a self-attention mechanism to assign weights to different modal data, the different modal data is subjected to weighted splicing according to the weights, and historical fusion features are obtained.

[0083] Firstly, the multi-modal historical data is subjected to denoising, normalization and preliminary feature extraction to obtain respective low-level semantic information.

[0084] Next, the spatio-temporal alignment module is used to accurately match different modal data in space and time through a dynamic registration algorithm or a real-time calibration method based on SLAM, so as to eliminate the deviation caused by factors such as sensor installation angle, sampling frequency and data transmission delay. Subsequently, the aligned data enters the attention fusion module, the attention fusion module automatically calculates the similarity and importance between different modal data by using a self-attention mechanism, and dynamically assigns attention weights, so as to realize deep information fusion across modalities at the feature level. In this process, the system can adaptively enhance the influence of high-quality information while suppressing the interference of noise or low-quality data. Finally, a feature map is constructed based on the fused fusion features, and the feature map is input into a graph neural network model.

[0085] Through the combination of spatio-temporal alignment and attention fusion, the spatio-temporal inconsistency problem of multi-sensor data in the collection process is effectively alleviated, and the complementary advantages of different modal data are fully utilized, thereby providing solid technical support for subsequent high-precision and low-delay detection.

[0086] The present application is based on a sensor extrinsic calibration matrix The point cloud data collected by the laser radar is accurately registered, so as to be mapped into the corresponding image coordinate system.

[0087] Firstly, the T calib The four-dimensional homogeneous coordinates of each point in the laser radar point cloud data are subjected to matrix multiplication operation, so as to realize the conversion from the laser radar coordinate system to the camera coordinate system. This process not only involves basic geometric transformation, but also requires normalization processing of the point cloud data, so as to ensure the consistency and accuracy of the data in the mapping process. Subsequently, the point cloud data subjected to the above transformation can find corresponding pixel positions in the image coordinate system, so as to realize effective mapping from three-dimensional space to two-dimensional image plane.

[0088] This T calibThe point cloud-image registration method can also make full use of the high-precision depth information provided by the laser radar and the rich texture and color information in the image to complement and fuse, so that the overall performance and robustness of the detection system are greatly improved.

[0089] p aligned =T calib·LiDAR ;

[0090] The acquisition time sequence of the RGB camera and the laser radar is aligned by using a hardware trigger signal, and the time deviation is controlled within ±5ms.

[0091] The application designs a double-branch gating network, the core idea of which is to decouple environment perception and modal fusion, and the environment perception branch first analyzes the environmental parameters such as light intensity I light and humidity H humidity , and then outputs the corresponding modal weight coefficient, which provides a dynamic adjustment basis for subsequent data fusion. Specifically, the environment perception branch first normalizes the input light intensity and humidity data, and then processes them through multiple fully connected layers and nonlinear activation functions (such as ReLU or Sigmoid) to extract features closely related to the environmental state, and uses the gating mechanism to automatically generate weight coefficients, which reflect the importance of each modal data under different environmental conditions.

[0092] The weight coefficient of the infrared modal data is calculated by formula (1).

[0093] w IR =Sigmoid(k1·I light +k2·H humidity ) Formula (1)

[0094] In the formula, w IR is the weight coefficient of the infrared modal data, Sigmoid is the activation function, k1 is the learnable parameter, I light is the light intensity, k2 is the learnable parameter, and H humidity is the humidity.

[0095] k1 and k2 are learnable parameters, and under rain and fog environment automatically rises to above 0.8.

[0096] The feature quality evaluation branch is mainly used for real-time calculation of the signal-to-noise ratio (SNR) of each modal feature, so as to dynamically identify and eliminate low-quality data. The branch first pre-processes each modal feature to extract key statistical information, and then quantitatively evaluates the effective signal and noise level of each modal data through a specially designed signal-to-noise ratio calculation module. In this way, the system can accurately determine which data has low quality in the current environment, thereby automatically reducing its weight or directly eliminating it, ensuring that only high-quality information is used in the subsequent feature fusion and detection process, thereby improving the overall detection accuracy and robustness:

[0097] The signal-to-noise ratio of the RGB modal data is calculated by formula (2);

[0098]

[0099] In the formula, SNR RGB is the signal-to-noise ratio of the RGB modal data, μ texture is the mean of the texture feature in the RGB modal data, σ noise is the standard deviation of the noise in the RGB modal data;

[0100] If SNR RGB <2.0, the RGB branch is closed, and only infrared and point cloud data are relied on.

[0101] The weighted concatenation strategy is used to fuse multi-modal features, aiming to fully utilize the complementary advantages of different sensor data, and achieve efficient and accurate feature fusion and joint optimization.

[0102] Specifically, after pre-processing and independent feature extraction of each modal data, the system performs weighted processing on the RGB image, infrared thermal imaging, and laser radar point cloud and other multi-modal features according to the pre-calculated dynamic weight coefficient, and then concatenates the weighted feature vectors in a certain order to form a unified feature representation. This weighted concatenation method not only ensures that each modal data can maintain its important characteristics in the fusion process, but also realizes the complementation and strengthening of information, providing more comprehensive and accurate feature input for subsequent detection and evaluation tasks:

[0103]

[0104] represents channel dimension concatenation, and the final feature dimension is d fused = 256.

[0105] S103, construct a graph neural network model.

[0106] The graph neural network model comprises an input layer, a dynamic graph convolution module, a dynamic edge weight learning module, a three-layer graph convolution module, a lightweight optimization module and an output layer, the input layer is used for receiving historical fusion features, the dynamic graph convolution module is used for constructing a graph structure based on the historical fusion features, the graph structure comprises nodes and edges, the dynamic edge weight learning module is used for dynamically adjusting the weight of the edge in the graph structure in a training process by using a learnable similarity measurement function, the three-layer graph convolution module is used for performing feature fusion on adjacent nodes in the graph structure, the lightweight optimization module is used for performing lightweight processing on the graph neural network model by using channel pruning and weight binarization, and the output layer is used for outputting a road crack detection result.

[0107] The road surface is divided into high-density grids, and the center point of each grid is accurately defined as a graph node and used as a basic unit in the graph convolution module.

[0108]

[0109] wherein f RGB is a texture feature of an RGB image (extracted by a pre-trained MobileNetV3), f IR is a temperature gradient feature of infrared thermal imaging, f Depth and f Reflect are respectively a depth and a reflection intensity of a laser radar point cloud. (N is the number of nodes), a local neighborhood is screened based on an Euclidean distance (a radius r=5 pixels), and redundant connections are reduced.

[0110] To improve the adaptability and robustness of the edge weight in the graph, a learnable similarity measurement function is introduced in the application, which automatically learns the similarity between nodes in the training process and dynamically adjusts the edge weight accordingly.

[0111] The weight of the edge between node i and node j in the graph structure is calculated by formula (3).

[0112]

[0113] In the formula, A ij is the weight of the edge between node i and node j in the graph structure, Softmax is an activation function, φ is an independent linear transformation layer, ψ is an independent linear transformation layer, v i is a feature vector of node i in the formula, v j is a feature vector of node j, and d is a dimension of the feature vector after linear transformation. to improve the feature expression capability,

[0114] The Gumbel-Softmax is used to realize the differentiable optimization of the discrete adjacency relationship, and the limitation of the predefined graph structure is avoided.

[0115] To further extract the multi-level features of the road cracks, the application designs a three-layer graph convolution network (GCN) to realize the step-by-step feature fusion and deep expression from local details to global structure.

[0116] The node feature matrix output by the l+1 layer of the graph convolution network is calculated through formula (4);

[0117] H (l+1) =LayerNorm(σ(AH (l) w (l) )) formula (4);

[0118] In the formula, H (l+1) is the node feature matrix output by the l+1 layer of the graph convolution network, LayerNorm is a layer normalization operation, sigma is a Swish activation function, A is an adjacency matrix, H (l) is the node feature matrix of the l layer of the graph convolution network, and W (l) is the trainable weight matrix of the l layer of the graph convolution network.

[0119] The application proposes a lightweight optimization strategy, in which the channel pruning technology is used as a key component, and the model parameters and calculation complexity are significantly reduced through a structured pruning method.

[0120] This structured pruning strategy not only helps to reduce redundant calculations and reduce memory occupation, but also significantly improves the inference speed of the model on edge devices without significantly losing detection accuracy.

[0121] The loss function of the structured pruning is calculated through formula (5);

[0122]

[0123] wherein, is the loss function of structured pruning, and λ is a hyperparameter, and W c is the weight corresponding to the cth channel, c is the number of channels, and ||W c is the L1 norm of the cth channel weight W c ;

[0124] After pruning, the model parameter quantity is reduced to 28% of the original size, and the FLOPs is reduced by 65%.

[0125] The weight binarization technology is adopted to map the weight parameters originally stored in the form of floating-point numbers in the network to binary parameters containing only {-1, +1}. This method significantly reduces the storage requirements and computational complexity of the model through quantization operation, and at the same time, in hardware implementation, it can utilize efficient binary operation to speed up the forward propagation and back propagation process. In order to reduce the risk of precision loss while improving the stability of training, the floating-point gradient information is retained in the present application for weight update and parameter adjustment.

[0126] The binarized weight parameters are calculated by formula (6);

[0127]

[0128] wherein, W binary is the binarized weight parameter, and Sign is the sign function, and W float is the weight parameter originally stored in the form of floating-point numbers in the network, is the loss function, is the partial derivative, is the small change amount of the loss function .

[0129] After binarization, the model volume is compressed to 12MB, and the memory occupation is reduced by 80%.

[0130] S104, input the historical fusion features into the graph neural network model for training to obtain an optimal graph neural network model.

[0131] The algorithm expansion of contrastive learning pre-training is carried out using the MoCo v3 framework for self-supervised pre-training, and the specific optimization strategies include:

[0132] Data augmentation combination: geometric transformation: random rotation (±15), scaling (0.8-1.2 times), cropping; photometric transformation: adjusting brightness (±20%), contrast (±30%), Gaussian noise (σ=0.1).

[0133] In the process of self-supervised pre-training, in order to stabilize the learning of feature representation, the present application adopts the momentum encoder strategy to update the teacher network parameters θ teacherMomentum update is performed.

[0134] Specifically, the teacher network parameters are updated by an exponential weighted average with the student network parameters, so as to maintain a smoother parameter change during the training process. This strategy not only helps to alleviate the oscillation problem that may occur during model updating, but also provides the model with stable and discriminative target features, thereby promoting the convergence of the entire self-supervised pre-training process:

[0135] The parameters of the teacher network are calculated by formula (7);

[0136] θ teacher =m·θ teacher +(1-m)·θ student Formula (7)

[0137] In the formula, θ teacher is the parameter of the teacher network, m is the momentum coefficient, θ student is the parameter of the student network, and m is the momentum coefficient.

[0138] The momentum coefficient m=0.999, and the feature consistency is maintained.

[0139] In the present application, the present application designs a loss function, which aims to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, so as to effectively distinguish the features of data of different categories. Through contrastive learning, the loss function encourages the model to map the features from the same sample or the same category of sample to the similar representation space, while pulling the representations of different categories of samples away, so that the model can capture more discriminative features.

[0140] The loss function of contrastive learning is calculated by formula (8);

[0141]

[0142] In the formula, is the loss function of contrastive learning, s p is the cosine similarity of positive sample pairs, s n is the cosine similarity of negative sample pairs, τ is the temperature parameter, N is the number of negative sample pairs, and τ=0.07.

[0143] In order to better maintain existing knowledge and prevent catastrophic forgetting in the process of incremental learning, the invention improves the elastic weight consolidation (EWC) and adopts a dynamic Fisher matrix updating strategy. Specifically, the strategy updates the Fisher matrix in real time according to the data distribution change in the model training process, so as to accurately evaluate the importance of each parameter to the current task. Only important parameters are constrained, so that only the parameter weights that make key contributions to previous tasks are retained in the subsequent incremental learning process, thereby reducing unnecessary constraints on new task learning.

[0144] The loss function of the elastic weight consolidation method is calculated by formula (9);

[0145]

[0146] In the formula, is the loss function of the elastic weight consolidation method, is the index of the model parameter, F i is the value of the corresponding parameter i in the Fisher information matrix, θ i is the value of parameter i in the current model training process, θ i，old is the value of parameter i at the end of the previous task training, i is the index of the model parameter, is the index of the top 10% important parameters, F i is updated by moving average.

[0147] In order to balance the acquisition of new and old knowledge in the process of incremental learning, the invention adopts an incremental data caching strategy. Specifically, by using a ring buffer to store recently collected data samples, the capacity is set to 1000 samples, ensuring that the model can refer to enough recent data when updating parameters, thereby maintaining sensitivity to new knowledge while preventing rapid forgetting of learned knowledge.

[0148] S105, obtaining real-time fusion features, inputting the real-time fusion features into the optimal graph neural network model to obtain road crack detection results of the real-time fusion features, and obtaining discrete point clouds in the road crack area based on the road crack detection results.

[0149] S106, regarding the discrete point cloud as a set of discrete sampling points, constructing a continuous surface model through a triangulation algorithm, calculating the geometric parameters of the road crack based on the continuous surface model, calculating the danger index of the road crack based on the geometric parameters, and obtaining the road crack danger level evaluation result based on the graph neural network model based on the danger index.

[0150] To achieve accurate estimation of the crack volume, the present application first performs Delaunay triangulation on the point cloud data in the crack area. Specifically, the discrete point cloud in the crack area is regarded as a set of discrete sampling points, and a continuous surface model is constructed by the Delaunay triangulation algorithm. This method can automatically generate a triangular mesh covering the crack area, and under the premise of ensuring geometric accuracy, it can reduce the number of holes and overlaps as much as possible, and ensure that the model can truly reflect the complex morphology of the crack area. The number of triangular facets N tri ∝ is proportional to the geometric complexity of the crack, the more complex the crack structure, the denser the point cloud sampling, and the number of generated triangular facets will also increase accordingly, thereby more detailedly describing the local features of the crack.

[0151] Through further processing of the continuous surface model, the depth, width and volume of the crack and other key parameters can be calculated, providing quantitative basis for subsequent risk assessment and maintenance decision-making. This mathematical model not only improves the accuracy of crack volume estimation, but also realizes efficient conversion from discrete data to continuous structure through automatic processing, providing solid technical support for crack detection and evaluation in practical engineering applications.

[0152] Based on the area and average depth of the triangular facet, the numerical integration method is used to estimate the crack volume. Specifically, the area of each triangular facet is calculated, and then combined with the average depth of the area where it is located, the total volume of the crack is obtained by integral summation. This method can fully utilize the geometric information of the continuous surface model, and realize the fine estimation of the volume of complex cracks, providing reliable data support for subsequent risk assessment and maintenance decision-making:

[0153] The total volume of the crack is calculated by formula (10);

[0154]

[0155] In the formula, V is the total volume of the crack, N tri is the total number of triangular facets in the crack area, A t is the area of the tth triangular facet, is the average depth of the three vertices of the tth triangular facet;

[0156] A t is the area of the tth triangular facet, is the average depth of the three vertices.

[0157] In combination with the geometric properties of the cracks (including length, width, depth, volume, etc.) and the safety standards in road engineering specifications, the application defines a risk index R as a quantitative indicator for risk level evaluation. The risk index R is calculated by normalizing each crack parameter and using a weighted summation method, wherein the weights of each parameter are reasonably set according to road engineering practices, maintenance requirements, and risk assessment standards.

[0158] The risk index is calculated by formula (11);

[0159]

[0160] In the formula, R is the risk index, L is the length of the crack, L max is the length threshold parameter, W max is the maximum width of the crack, W crit is the width threshold parameter, D is the depth of the crack, D crit is the depth threshold parameter.

[0161] L max = 10 m, W crit = 5 mm, D crit = 20 mm are the threshold parameters, and R > 0.7 is determined as high risk.

[0162] The application realizes seamless integration and visualization with the GIS platform through a well-designed data interface. Specifically, the system uploads the coordinates, dimensions, and risk levels of the cracks and other key information to the GIS platform in GeoJSON format, thereby supporting real-time map rendering and historical data comparison with a time resolution of 1 hour. At the same time, based on the risk level evaluation results, the system automatically generates a maintenance priority list and pushes it to the municipal management terminal, providing a scientific basis for road maintenance decision-making. To further optimize the inspection operation, the application also integrates the Dijkstra algorithm to calculate the optimal inspection path, significantly reducing the driving distance of the maintenance vehicle and improving the overall operation efficiency.

[0163] Figure 2 is a schematic diagram of the main components of the road crack risk level evaluation system based on the graph neural network large model in the embodiment of the application. As Figure 2 shown, the road crack risk level evaluation system based on the graph neural network large model 1 provided by the embodiment of the application includes a collection module 10, a data processing module 20, a model construction module 30, a training module 40, a road crack detection module 50, and a road crack evaluation module 60.

[0164] The collection module 10 is used to collect multi-modal historical data, and the multi-modal historical data includes RGB modal data, infrared modal data, and laser modal data.

[0165] The data processing module 20 is configured to perform data processing on the multi-modal historical data to obtain pre-processed data, perform spatio-temporal alignment on the pre-processed data, input the spatio-temporally aligned pre-processed data into a self-attention mechanism to assign weights to different modal data, and perform weighted splicing on the different modal data according to the weights to obtain historical fusion features.

[0166] The model construction module 30 is configured to construct a graph neural network model, which includes an input layer, a dynamic graph convolution module, a dynamic edge weight learning module, a three-layer graph convolution module, a lightweight optimization module, and an output layer. The input layer is configured to receive the historical fusion features. The dynamic graph convolution module is configured to construct a graph structure based on the historical fusion features. The graph structure includes nodes and edges. The dynamic edge weight learning module is configured to dynamically adjust the weights of the edges in the graph structure in a training process by using a learnable similarity measurement function. The three-layer graph convolution module is configured to perform feature fusion on adjacent nodes in the graph structure. The lightweight optimization module is configured to perform lightweight processing on the graph neural network model by using channel pruning and weight binarization. The output layer is configured to output a road crack detection result.

[0167] The training module 40 is configured to input the historical fusion features into the graph neural network model for training to obtain an optimal graph neural network model.

[0168] The road crack detection module 50 is configured to obtain real-time fusion features, input the real-time fusion features into the optimal graph neural network model to obtain a road crack detection result of the real-time fusion features, and obtain discrete point clouds in a road crack region based on the road crack detection result.

[0169] The road crack evaluation module 60 is configured to regard the discrete point clouds as a group of discrete sampling points, construct a continuous surface model by using a triangulation algorithm, calculate a geometric parameter of the road crack based on the continuous surface model, calculate a danger index of the road crack based on the geometric parameter, and obtain a road crack danger level evaluation result based on the graph neural network model based on the danger index.

[0170] Figure 3 An electronic device entity structure schematic diagram provided by an embodiment of the present application is shown in FIG. 7. Figure 3 As shown in FIG. 7, the electronic device 70 includes a processor 701, a memory 702, and a bus 703.

[0171] The processor 701 and the memory 702 can communicate with each other through the bus 703.

[0172] The processor 701 is configured to invoke program instructions in the memory 702 to execute the method provided by the above-mentioned method embodiments, and to execute the method provided by the embodiment of the application.

[0173] The embodiment provides a non-transitory computer readable storage medium storing computer instructions, and the computer instructions cause a computer to execute the method provided by the embodiment of the application.

[0174] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps including the above-mentioned method embodiments when executed; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various storage media that can store program codes.

[0175] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for assessing the risk level of road cracks based on a large graph neural network model, characterized in that, include: Collect multimodal historical data, including RGB modal data, infrared modal data, and laser modal data; The multimodal historical data is processed to obtain preprocessed data. The preprocessed data is spatiotemporally aligned, and the spatiotemporally aligned preprocessed data is input into a self-attention mechanism to assign weights to different modal data. The different modal data are then weighted and concatenated according to the weights to obtain historical fusion features. A graph neural network model is constructed, comprising an input layer, a dynamic graph convolution module, a dynamic edge weight learning module, a three-layer graph convolution module, a lightweight optimization module, and an output layer. The input layer receives the historical fusion features. The dynamic graph convolution module constructs a graph structure based on the historical fusion features. The graph structure includes nodes and edges. The dynamic edge weight learning module dynamically adjusts the weights of edges in the graph structure during training using a learnable similarity metric function. The three-layer graph convolution module performs feature fusion on adjacent nodes in the graph structure. The lightweight optimization module uses channel pruning and weight binarization to lightweight the graph neural network model. The output layer outputs the road crack detection results. The historical fusion features are input into the graph neural network model for training to obtain the optimal graph neural network model; Real-time fusion features are acquired and input into the optimal graph neural network model to obtain the road crack detection results of the real-time fusion features. Based on the road crack detection results, discrete point clouds within the road crack area are obtained. The discrete point cloud is regarded as a set of discrete sampling points. A continuous surface model is constructed by the triangulation algorithm. The geometric parameters of the road cracks are calculated based on the continuous surface model. The hazard index of the road cracks is calculated based on the geometric parameters. The hazard index is used to obtain the road crack hazard level assessment result based on the graph neural network model.

2. The method for assessing the risk level of road cracks based on a large graph neural network model according to claim 1, characterized in that, The preprocessed data is spatiotemporally aligned, and the spatiotemporally aligned preprocessed data is input into a self-attention mechanism to assign weights to different modalities, including: The weighting coefficients of the infrared modal data are calculated using formula (1); W R =Sigmoid(k1·I) light +k2·H humidity ) formula (1); In the formula, w IR Here are the weighting coefficients for the infrared modal data, sigmoid is the activation function, k1 is a learnable parameter, and I... light Let H be the light intensity, k2 be a learnable parameter, and H be the light intensity. humidity Humidity; The signal-to-noise ratio of the RGB modal data is calculated using formula (2); In the formula, SNR RGB Let μ be the signal-to-noise ratio of the RGB modal data. texture σ is the mean of the texture features in the RGB modal data. noise denoted as the standard deviation of noise in the RGB modal data.

3. The method for assessing the risk level of road cracks based on a large graph neural network model according to claim 1, characterized in that, The dynamic edge weight learning module uses a learnable similarity metric function to dynamically adjust the weights of edges in the graph structure during training, including: The weight of the edge between node i and node j in the graph structure is calculated using formula (3); In the formula, A ij Let be the weight of the edge between node i and node j in the graph structure, Softmax be the activation function, φ be an independent linear transformation layer, ψ be an independent linear transformation layer, and v be the weight of the edge between node i and node j. i Let v be the feature vector of node i in the vector. j Let d be the feature vector of node j, and d be the dimension of the feature vector after linear transformation.

4. The method for assessing the risk level of road cracks based on a large graph neural network model according to claim 1, characterized in that, The three-layer graph convolution module is used to perform feature fusion on adjacent nodes in the graph structure, including: The node feature matrix output by the (l+1)th layer of the graph convolutional network is calculated using formula (4); H (l+1) = LayerNorm(σ(AH (l) w (l) ) Formula (4); In the formula, H (l+1) Let H be the node feature matrix output by the (l+1)th layer of the graph convolutional network, where LayerNorm is the layer normalization operation, σ is the Swish activation function, A is the adjacency matrix, and H is the number of nodes. (l) Let W be the node feature matrix of the l-th layer graph convolutional network. (l) Let be the trainable weight matrix of the l-th layer graph convolutional network.

5. The method for assessing the risk level of road cracks based on a large graph neural network model according to claim 1, characterized in that, The lightweight optimization module employs channel pruning and weight binarization to lightweight the graph neural network model, including: The loss function of structured pruning is calculated using formula (5); In the formula, The loss function for structured pruning is λ, where λ is a hyperparameter and W is the loss function for structured pruning. c Let W be the weight corresponding to the c-th channel, where c is the number of channels. c ||1 represents the weight W of the c-th channel. c L1 norm; The weight parameters after binarization are calculated using formula (6); In the formula, W binary Here are the binary weight parameters, Sign is the sign function, and W... float These are the weight parameters that were originally stored as floating-point numbers in the network. For loss function, For partial derivatives, loss function The minute change.

6. The method for assessing the risk level of road cracks based on a large graph neural network model according to claim 1, characterized in that, The step of inputting historical fusion features into the graph neural network model for training to obtain the optimal graph neural network model includes: The parameters of the teacher network are calculated using formula (7); θ teacher = m·θ teacher +(1 - m)·θ student Equation (7); In the formula, θ teacher Here are the parameters of the teacher network, m is the momentum coefficient, and θ is the... student Here are the parameters of the student network, and m is the momentum coefficient; The loss function for contrastive learning is calculated using formula (8); In the formula, For the loss function of contrastive learning, s p For the cosine similarity of positive sample pairs, s n Let τ be the cosine similarity of negative sample pairs, τ be the temperature parameter, and N be the number of negative sample pairs. The loss function of the elastic weight solidification method is calculated using formula (9); In the formula, Let the loss function be the elastic weight solidification method. For the index of model parameters, F i Let θ be the value of parameter i in the Fisher information matrix. i Let θ be the value of parameter i during the current model training process. i,old Let i be the value of parameter i at the end of the previous task training, where i is the index of the model parameter.

7. The method for assessing the risk level of road cracks based on a large graph neural network model according to claim 1, characterized in that, The calculation of road crack geometric parameters based on the continuous surface model, and the obtaining of road crack assessment results based on the geometric parameters, includes: The total volume of the crack is calculated using formula (10); In the formula, V is the total volume of the crack, and N is the total volume of the crack. tri A represents the total number of triangular facets within the crack region. t Let be the area of ​​the t-th triangular facet. Let be the average depth of the three vertices of the t-th triangular facet; The risk index is calculated using formula (11); In the formula, R is the hazard index, L is the length of the crack, and L max W is the length threshold parameter. max W is the maximum width of the crack. crit Here, D is the width threshold parameter, and D is the crack depth. crit This is the depth threshold parameter.

8. A road crack hazard level assessment system based on a large graph neural network model, characterized in that, include: The acquisition module is used to acquire multimodal historical data, which includes RGB modal data, infrared modal data, and laser modal data. The data processing module is used to process the multimodal historical data to obtain preprocessed data, perform spatiotemporal alignment on the preprocessed data, and input the spatiotemporally aligned preprocessed data into a self-attention mechanism to assign weights to different modal data. Based on the weights, the different modal data are weighted and concatenated to obtain historical fusion features. A model building module is used to construct a graph neural network model. The graph neural network model includes an input layer, a dynamic graph convolution module, a dynamic edge weight learning module, a three-layer graph convolution module, a lightweight optimization module, and an output layer. The input layer is used to receive the historical fusion features. The dynamic graph convolution module constructs a graph structure based on the historical fusion features. The graph structure includes nodes and edges. The dynamic edge weight learning module uses a learnable similarity metric function to dynamically adjust the weights of edges in the graph structure during training. The three-layer graph convolution module is used to perform feature fusion on adjacent nodes in the graph structure. The lightweight optimization module uses channel pruning and weight binarization to perform lightweight processing on the graph neural network model. The output layer is used to output the road crack detection results. The training module is used to input historical fusion features into the graph neural network model for training, so as to obtain the optimal graph neural network model. The road crack detection module is used to acquire real-time fused features, input the real-time fused features into the optimal graph neural network model to obtain the road crack detection result of the real-time fused features, and obtain the discrete point cloud in the road crack area based on the road crack detection result; The road crack assessment module is used to treat the discrete point cloud as a set of discrete sampling points, construct a continuous surface model through a triangulation algorithm, calculate the geometric parameters of the road crack based on the continuous surface model, calculate the hazard index of the road crack based on the geometric parameters, and obtain the road crack hazard level assessment result based on the hazard index and the graph neural network model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images