Road facility disease evolution prediction and maintenance decision method and system based on cnn-transformer hybrid architecture

Abstract

The application discloses a road facility disease evolution prediction and maintenance decision method and system based on a CNN-Transformer hybrid architecture. The method comprises the following steps: acquiring multi-period aerial images, constructing a time series dataset and performing spatial alignment and simulation degradation processing; a hybrid network model with an encoder-decoder structure is constructed, the encoder adopts a double-channel structure, local features are extracted through a convolution block and global semantic features are extracted based on an attention mechanism and integrated edge enhancement components, the decoder adopts a bidirectional propagation unit and fuses features through a cross-scale attention mechanism; the model is collaboratively trained and compressed to obtain a simplified prediction model; the model is used to predict the facility state, a multi-criteria optimization objective is constructed, a genetic algorithm is used to solve to obtain a non-dominated solution set; finally, a maintenance decision report is generated according to the decision preference and visualized display. The application realizes accurate prediction of road facility disease evolution and scientific maintenance decision.

Description

Technical Field

[0001] This invention relates to the field of intelligent road infrastructure maintenance technology, and in particular to a method and system for predicting the evolution of road infrastructure defects and making maintenance decisions based on deep learning and multi-objective optimization. Background Technology

[0002] With the continuous expansion of my country's transportation infrastructure, the health of road facilities is directly related to the safety and efficiency of transportation. Traditional road maintenance mainly relies on manual inspections and regular maintenance, which is not only inefficient and costly, but also makes it difficult to accurately predict the development trend of road damage and make scientific decisions.

[0003] In recent years, with the development of drone technology and computer vision technology, deep learning-based road defect detection methods have been gradually applied. Convolutional Neural Networks (CNNs) excel in image feature extraction, effectively identifying local defect features such as cracks and potholes. However, CNNs have a limited receptive field, making it difficult to capture the global spatial correlation and long-term evolution of defects over time. While Transformer models are adept at modeling long-distance dependencies, they are insufficient in extracting local detailed features.

[0004] Most existing road maintenance decision-making methods are based on static assessments of the current state, lacking dynamic predictions of future trends in road damage. A few studies have attempted to combine time-series prediction models, but these often separate feature extraction and prediction modeling, failing to achieve end-to-end joint optimization. Furthermore, maintenance decision-making requires comprehensive consideration of multiple objectives, including economic costs, structural safety, and construction impacts; traditional methods struggle to find the optimal balance between these conflicting objectives.

[0005] Therefore, there is an urgent need to develop a new technical solution that can simultaneously take into account local details and global contextual information, achieve accurate prediction of disease evolution, and provide scientific maintenance decision support. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for predicting the evolution of road facility defects and making maintenance decisions based on a CNN-Transformer hybrid architecture, so as to achieve accurate prediction of the evolution of road facility defects and scientific maintenance decisions.

[0007] In a first aspect, embodiments of this application provide a method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture, the method comprising:

[0008] Acquire aerial images of the target road facilities from multiple periods, construct a time-series dataset, and perform spatial alignment and simulated degradation processing;

[0009] Construct a hybrid network model with an encoder-decoder structure. The encoder adopts a dual-path structure. The first path extracts local features through convolutional blocks, and the second path is based on an attention mechanism and integrates an edge enhancement component. The decoder adopts a bidirectional propagation unit and fuses features through a cross-scale attention mechanism.

[0010] The hybrid network model is co-trained and compressed to obtain a simplified prediction model;

[0011] The simplified prediction model is used to predict facility status. Based on the prediction results, a multi-criteria optimization objective is constructed, and a genetic algorithm is used to solve for the non-dominated solution set.

[0012] The final solution is determined from the non-dominated solution set based on decision preferences, and a maintenance decision report is generated and visualized.

[0013] Secondly, embodiments of this application provide a road infrastructure defect evolution prediction and maintenance decision-making system based on a CNN-Transformer hybrid architecture, applied to the road infrastructure defect evolution prediction and maintenance decision-making method based on a CNN-Transformer hybrid architecture as described in the first aspect, the system comprising:

[0014] The data acquisition and preprocessing module is used to acquire multi-period aerial images of the target road facilities, construct a time-series dataset, and perform spatial alignment and simulated degradation processing.

[0015] The hybrid network model building module is used to build a hybrid network model with an encoder-decoder structure. The encoder adopts a dual-path structure. The first path extracts local features through convolutional blocks, and the second path is based on an attention mechanism and integrates edge enhancement components. The decoder adopts a bidirectional propagation unit and fuses features through a cross-scale attention mechanism.

[0016] The model training and compression module is used to perform co-training and model compression on the hybrid network model to obtain a simplified prediction model.

[0017] The state prediction and optimization module is used to predict the state of facilities using the simplified prediction model, construct a multi-criteria optimization objective based on the prediction results, and solve the non-dominated solution set using a genetic algorithm.

[0018] The decision generation and display module is used to determine the final solution from the non-dominated solution set according to decision preferences, generate and maintain decision reports, and display them visually.

[0019] Thirdly, embodiments of this application provide an electronic device, including:

[0020] processor;

[0021] Memory used to store processor-executable instructions;

[0022] The processor is configured to implement the road infrastructure disease evolution prediction and maintenance decision-making method based on the CNN-Transformer hybrid architecture as described in the first aspect when executing the instructions.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the road infrastructure defect evolution prediction and maintenance decision-making method based on the CNN-Transformer hybrid architecture as described in the first aspect.

[0024] The beneficial effects of this invention are as follows: By employing a CNN-Transformer hybrid architecture, the local feature extraction capability of CNN and the global modeling advantage of Transformer are effectively combined, improving the accuracy of disease evolution prediction; a dual-path encoder design is adopted to optimize feature extraction of local details and global semantics respectively, enhancing the model's feature expression capability; a multi-objective optimization algorithm is introduced, comprehensively considering multiple factors such as maintenance cost, structural safety, and construction impact, providing a more scientific maintenance decision-making scheme; model compression technology is used to achieve efficient deployment in resource-constrained environments, improving the practicality of the method; and a complete technical solution from data acquisition to decision output is provided, realizing intelligent management of road facility maintenance. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a road infrastructure disease evolution prediction and maintenance decision-making method based on a CNN-Transformer hybrid architecture, provided as an embodiment of this application.

[0026] Figure 2 The architecture diagram of the road facility disease evolution prediction and maintenance decision system based on the CNN-Transformer hybrid architecture provided in this application.

[0027] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0029] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0030] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] Example 1

[0032] Figure 1 This is a schematic diagram illustrating the process of a road infrastructure defect evolution prediction and maintenance decision-making method based on a CNN-Transformer hybrid architecture, provided as an embodiment of this application. Figure 1 As shown, a method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture includes:

[0033] S1. Acquire multi-period aerial images of the target road facilities, construct a time-series dataset, and perform spatial alignment and simulated degradation processing. Data preparation and augmentation: This step is the data input and preprocessing stage of the entire method, used to construct a high-quality, robust training and prediction dataset. Acquiring multi-period aerial images is the data source, ensuring the temporal sequence of the data, which is a prerequisite for evolution prediction. Spatial alignment is to eliminate geometric errors caused by different shooting angles and positions, ensuring the spatial consistency of the image sequence and allowing the model to focus on content changes. Simulated degradation processing is a data augmentation technique that improves the model's generalization ability and robustness in complex real-world environments by simulating image blurring and noise problems that may occur in the real world.

[0034] Spatial alignment and simulated degradation processing are decomposed into four sub-steps with clearly defined inputs and outputs: image registration → lesion annotation → degradation enhancement → dataset construction. This forms a complete data preprocessing pipeline, ensuring the feasibility and repeatability of the operation. Specifically, in this embodiment, acquiring multi-period aerial images of the target road facilities, constructing a time-series dataset, and performing spatial alignment and simulated degradation processing includes:

[0035] Image registration and spatial cropping are performed on aerial images from multiple periods to construct a spatiotemporally aligned image sequence; a unified coordinate system is established to eliminate spatiotemporal differences. Image alignment can be achieved using feature point matching and transformation matrix estimation. Where T is the affine transformation matrix, and To match feature point pairs, the processing parameters are correlated with the spatial resolution of the image and the acquisition time span, adapting the processing method to the spatiotemporal characteristics of the data itself.

[0036] After registration, disease areas are labeled to form a disease image sample set. A comprehensive index combining IoU and Dice coefficients can be used for labeling quality control.

[0037] Simulated degradation operations, including degraded filtering and random interference, are applied to the image sequence to enhance data diversity and robustness. Controlled data augmentation can be achieved using parameterized degradation models (Gaussian blur, noise injection, JPEG compression). The parameters of the degradation model are correlated with actual acquisition conditions (such as flight speed and ambient temperature), making the simulated degradation more closely resemble real physical processes and enhancing the data's realism and generalization ability.

[0038] Based on labeled and augmented image sequences, a time-series dataset of state evolution is constructed for model training. The strength of data augmentation is correlated with the logarithm of the time span, automatically applying stronger augmentations to long-term sequence data to better cover possible morphological changes.

[0039] By establishing an intelligent preprocessing system with tightly coupled internal parameters and dynamic correlation with the physical world, high-quality, high-fidelity, and strongly correlated training data is provided for subsequent prediction models from the source. This is an important cornerstone in the entire method chain to ensure the accuracy of the final prediction and decision.

[0040] S2. Construct a hybrid network model with an encoder-decoder structure. The encoder employs a dual-path structure: the first path extracts local features through convolutional blocks, and the second path integrates an edge enhancement component based on an attention mechanism. The decoder uses bidirectional propagation units and fuses features through a cross-scale attention mechanism. The core prediction engine is constructed; this step is the technological core and innovative essence of the entire invention. Its function is to build a dedicated neural network architecture capable of simultaneously capturing local details and global context, and effectively fusing them for accurate prediction.

[0041] The encoder-decoder structure is a classic framework for mapping from images to prediction results. The dual-path encoder is the core innovation. Specifically, in this embodiment, the hybrid network model constructing the encoder-decoder structure includes: in the dual-path structure of the encoder, the first path uses inverted dense blocks containing depthwise separable convolutions to extract local subtle features of the facility's state layer by layer; the first path (CNN path) is used to accurately extract local subtle features of the disease, such as crack width, texture, and small-scale peeling. Using depthwise separable convolutions to construct inverted dense blocks enhances feature reuse while reducing the number of parameters. The output of each layer is not only passed to the next layer but also fed back to the previous layer, forming a local feature cyclic enhancement mechanism. The formula is expressed as:

[0042] ,

[0043] in, Indicates the first Local feature output of the layer; This represents a depthwise separable convolution (3×3 convolution kernel). This indicates the number of lookback layers in the reverse connection (hyperparameter). Indicates the preceding Local feature maps of the layer are used for reverse dense connections.

[0044] Specifically, the first path uses an inverse dense connect block containing depthwise separable convolutions to extract local subtle features in the following way: depthwise separable convolutions perform channel separation and spatial feature extraction on the input features, thereby achieving parameter efficiency optimization and feature decoupling.

[0045] In the inverse dense connection block, the input features are first separated into channels and spatial features are extracted using depthwise separable convolution. The input features are then processed by depthwise separable convolution, which decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. These steps respectively extract spatial features and fuse channel information, reducing the number of parameters while enhancing the model's ability to perceive local defects such as cracks and spalling. Depthwise separable convolution separates the input features into channels and extracts spatial features, achieving parameter efficiency optimization and feature decoupling. In the depthwise convolution stage, spatial convolution is performed independently on each input channel to extract channel-specific spatial features, reducing the number of parameters while maintaining feature independence.

[0046] Each channel of the input feature map is convolved individually, with the convolution kernel operating only in the spatial dimension. The convolution kernel slides upwards, and each channel corresponds to a unique kernel. The formula can be written as:

[0047] ,

[0048] in, For channel indexing, For kernel width, In the output feature map, the first individual channels, spatial locations The value at that location, For the first individual channels, spatial locations The weight of the position, In the input feature map, the first The input positions involved in the calculation on each channel (corresponding to the output channel). The offset used to locate the center of the convolution kernel, ensuring that the operation is relative to the current position. symmetry.

[0049] Depthwise convolution stage: ,

[0050] in, : Feature map output from the previous layer (which can come from ordinary convolution, residual block, or token mapping of Transformer). Each channel corresponds to Convolution kernel set, Perform convolution independently on each channel to obtain This feature retains the original number of channels but has undergone fine-grained filtering at the spatial scale. Depthwise Convolution, through channel-independent spatial convolution, significantly reduces computational cost and parameter size while maintaining the spatial receptive field, making it a core component of mobile and lightweight networks. It is often used in conjunction with 1×1 pointwise convolution to form efficient depthwise-separable convolution, which has been widely validated in models such as MobileNet, Xception, and EfficientNet.

[0051] Pointwise Convolution Stage: Information fusion between channels is achieved through 1×1 convolutions, combining the features output by depthwise convolutions along the channel dimension to generate new feature representations. This is equivalent to performing a fully connected layer at each spatial location, changing only the number of channels without altering the spatial resolution. First, DepthwiseConv (independent convolution for each channel) is performed. Convolution is performed to obtain spatial features; then PointwiseConv is used to linearly fuse these features along the channel dimension to obtain the final features. This step is specifically designed to extract the subtle spatial features of localized defects such as cracks and pits, while maintaining the independence of features for different types of defects through channel separation.

[0052] Workflow (in conjunction with DepthwiseConv): 1. DepthwiseConv: Perform separate operations on each input channel. 1. Spatial Convolution: Extracts local spatial features. 2. Pointwise Conv: Uses a 1×1 convolution to linearly combine features from all channels, achieving cross-channel information fusion and allowing adjustment of the number of channels. 3. Activation (Optional): Often followed by non-linear activations such as ReLU / Swish to enhance expressive power.

[0053] Subsequently, a reverse dense connection structure is employed to fuse shallow features with deep features that have undergone nonlinear transformation across layers. This reverse dense connection structure further fuses shallow features with deep features that have undergone nonlinear activation and normalization, enhancing gradient flow through feature reuse and backpropagation paths. This effectively alleviates the information decay problem in deep networks and improves the richness and diversity of feature representation. Specifically, the initial dense features are derived from the Pointwise features obtained in the previous layer. First, batch normalization is performed to eliminate internal covariate bias, making the feature distribution more stable. Then, the GELU (Gaussian Error Linear Unit) activation function is applied to introduce non-linearity, retaining probabilistic activation of inputs less than 0, thereby improving the model's expressive power. This can be represented as:

[0054] .

[0055] The previous layer's output is processed using BatchNorm→GELU to ensure consistent feature distribution and introduce non-linearity. A 1×1 convolution is then used to perform a linear transformation along the channel dimension (adjustable channel number for feature blending). The original features... By concatenating new features, dense connections (DenseNet-style) are formed, enabling feature reuse and direct gradient propagation. This is represented as:

[0056] ,

[0057] in, Indicates density, .

[0058] Layer Normalization is performed on the dense features of layer L along the channel dimension to further unify the mean and variance, improve numerical stability, and prepare for subsequent layers (such as fully connected layers and Transformers). This is represented as:

[0059] .

[0060] A channel attention mechanism is introduced during feature fusion to dynamically adjust the weight distribution of features in each channel. This mechanism generates channel weight vectors by applying global average pooling and mapping to fully connected layers for each channel feature. The original features are then recalibrated using the Sigmoid function after normalization, thereby dynamically strengthening key channel features related to the disease and suppressing redundant information. The implementation process is as follows:

[0061] By global average pooling The spatial dimension is compressed to 1, forming a channel description vector. .

[0062] ,

[0063] Squeeze: Performs global average pooling on the spatial features of each channel c to obtain a scalar. This scalar represents global statistics for that channel, capturing the context of the entire feature map.

[0064] Excitation: The vector obtained from compression The input is fed into a two-layer fully connected (FC) network. The importance coefficient of each channel is learned through this two-layer fully connected network (dimensionality reduction-ReLU-dimensionality increase-Sigmoid). .

[0065] ① First linear transformation (Usually accompanied by dimensionality reduction) + ReLU introduces nonlinearity; ② Second linear transformation (Upgrading to the original number of channels) + Sigmoid maps the output to the [0,1] interval, obtaining the attention weight for each channel. .

[0066] ,

[0067] This two-layer bottleneck structure can learn the nonlinear dependencies between channels, and at the same time generate coefficients that can be directly used for weighting through the Sigmoid function.

[0068] Scale: Multiply the attention weights channel-by-channel back into the original feature map to achieve channel-level feature recalibration. Multiply the resulting coefficients channel-by-channel back into the original feature map to obtain the channel-attention-weighted features. :

[0069] ,

[0070] in, This indicates channel-wise multiplication, meaning multiplying by each channel c. .

[0071] By cascading multiple levels of the aforementioned inverse dense connective blocks, the ability to express local details of facility defects is progressively enhanced. Through the cascading and stacking of these multiple levels of inverse dense connective blocks, a deep local feature extraction network is constructed. Each level of the block progressively enhances the perception and expression of local details of facility defects, ultimately forming a highly discriminative local feature representation. This provides a high-precision feature foundation for subsequent defect evolution prediction and maintenance decisions.

[0072] The first reversedense block accepts the original input features. Generate feature maps .

[0073] ,

[0074] Each layer inside the reverse dense block first performs BN→GELU→1×1 Conv (or 3×3 Conv) on the current feature, and then concatenates the new feature with the existing feature in the channel dimension, but the concatenation order is from deep to shallow (i.e., reverse), which is the opposite of the shallow to deep order of ordinary DenseBlock.

[0075] Recursive call: the first The reverse dense block with the first The output of each block is used as input to continue feature extraction and reverse concatenation.

[0076] .

[0077] Each call further accumulates channels, and the number of channels grows linearly with the number of blocks (the growth rate depends on the growth rate of each layer). In a normal DenseBlock, layers... The input is the concatenation of features from all previous layers (from shallow to deep). In ReverseDenseBlock, the layers... The input is still a concatenation of existing features, but the new features are placed before the concatenation, which is equivalent to feeding back from deep layers to shallow layers. This approach can better fuse high-level semantics and low-level details in the decoder / upsampling stage (such as the upsampling path of U-Net).

[0078] Aggregate the outputs of all inverse dense blocks to obtain local features. .

[0079] .

[0080] Common aggregation methods include channel-by-channel addition, channel-by-channel averaging, or re-concatenation (with a second 1×1 Conv for dimensionality reduction). The resulting features possess multi-scale and multi-level information.

[0081] The second path integrates a direction-sensitive edge enhancement filter component at the front end to strengthen contour information and models the global semantics and long-term dependencies of facility status based on a hierarchical sliding window attention mechanism. This second path (Transformer path) models the global semantic information and long-term dependencies of defects, understanding the relationships between different defect areas and the context of the entire road structure. The edge enhancement component a priori strengthens the boundary information of defects before global feature extraction, providing the Transformer with clearer structural cues.

[0082] These four steps together constitute a highly efficient feature extraction pipeline: depthwise separable convolution provides parametrically efficient basic feature extraction; inverse dense connections ensure maximum feature utilization and effective gradient propagation; channel attention mechanism enables adaptive optimization of feature selection; and multi-level cascaded structures construct a hierarchical feature representation system. In the task of predicting the evolution of road infrastructure defects, this design can: accurately capture the microscopic change trends of defects; effectively handle the fusion of defect features at different scales; adaptively focus on the feature information most important for prediction; and provide high-quality feature input for subsequent time-series predictions.

[0083] The second pathway models global semantics and long-term dependencies in the following way:

[0084] In the orientation-aware edge enhancement filter component, a multi-directional Gabor filter bank is used to extract texture features from different angles, and a Laplacian Gaussian filter is combined to enhance the edge response. This comprehensive capture of the directional features of the disease is achieved. The second path, the global semantic modeling path, integrates an orientation-aware edge enhancement filter (OEEF) at the front end to enhance the contour and structural information of the facility image. The back end employs hierarchical sliding window attention (H-SWA) to model global semantics and long-term dependencies at different scales. The edge enhancement filter formula is expressed as follows:

[0085] ,

[0086] in, For in position Edge enhancement output at the location, For direction The Gabor filter bank, where Θ is the set of directions (e.g., 0°, 45°, 90°, 135°). For the input image in Pixel value at that location, This is a convolution operation.

[0087] In the hierarchical sliding window attention mechanism, the feature map is divided into multiple non-overlapping local windows. Self-attention is calculated within each window to reduce computational complexity, and cross-window information interaction is achieved through window shifting operations.

[0088] ,

[0089] in, The query, key, and value matrix is ​​obtained by linear transformation of the input feature map. ， The relative position is offset to adapt to different window sizes. The dimension of the key vector (scaling factor to prevent softmax saturation). This mechanism can effectively model long-distance spatial dependencies in large-scale road images, such as identifying associated disease patterns distributed in different regions of the image.

[0090] A pyramid structure is constructed by stacking multiple hierarchical sliding window attention layers to capture multi-scale semantic information from local to global perspectives; the hierarchical feature extraction process includes:

[0091] Level 1: For original edge features We directly use ShiftedWindow Multi-Head Self-Attention (SW-MSA) to model local self-attention. SW-MSA first performs ordinary window attention (W-MSA) within non-overlapping windows, and then achieves cross-window information interaction through window translation (shift).

[0092] Level 2: The Level 1 output is then subjected to another SW-MSA to capture higher-level context. PatchMerging is used to merge adjacent patches, achieving 2× downsampling (halving the feature map size and doubling the number of channels), reducing computation and expanding the receptive field. PatchMerging merges adjacent 2×2 patches into a larger patch, and then uses 1×1 Conv to adjust the number of channels.

[0093] Level 3: Similar to Level 2, SW-MSA is performed on a coarser feature map, followed by downsampling to further expand the receptive field. Each PatchMerging operation divides the width and height of the feature map by 2, doubling the number of channels to form a hierarchical structure.

[0094] Level 4: The final SW-MSA+PatchMerging process yields the deepest feature representation, suitable as a global semantic feature or for input into the decoding / classification head. In the Swin-Transformer, W-MSA and SW-MSA are usually used alternately in each stage (i.e., before each PatchMerging layer), but here SW-MSA is used uniformly to strengthen cross-window association.

[0095] The global semantic features output from each level are passed to the decoder through skip connections, and together with the corresponding local features from the first path, they constitute the input of the cross-scale attention aggregation mechanism.

[0096] First, consider the hierarchical features of the i-th layer. LayerNorm normalization is performed, followed by mapping the channel dimensions to the dimensions required for subsequent cross-scale attention using a fully connected (Linear) layer. This step is equivalent to a projection of the skip-connection, ensuring that features from different layers remain compatible across channels.

[0097] LayerNorm performs mean-variance normalization on the channel vector at each position to improve numerical stability. Linear typically uses a 1×1 Conv (equivalent to fully connected) to achieve channel compression / dimensionality increase while maintaining the same spatial size.

[0098] skip-feature With local features The data is fed into the Cross-Scale Attention module to complete the interaction and information fusion between features of different scales. The output is the [number]th [item / section]. Decoding features of the layer.

[0099] .

[0100] The decoder employs a bidirectional state propagation unit to enhance global contextual understanding through forward-backward information transmission and introduces a cross-scale attention aggregation mechanism to dynamically fuse feature information from both encoder pathways. The decoder integrates and reconstructs the rich features extracted by the encoder: the bidirectional propagation unit enhances the understanding of the global context through forward and backward information flows, ensuring predictive consistency. The cross-scale attention mechanism adaptively fuses features from different levels (i.e., different scales) of the encoder's dual pathways, organically combining shallow details with deep semantics to accurately reconstruct the morphology and location of lesions.

[0101] The decoder achieves feature fusion and information transmission in the following ways:

[0102] In the bidirectional state propagation unit, a bidirectional processing mechanism of forward scanning and backward scanning is adopted to propagate state information in the forward and backward directions along the spatial dimension, respectively, to receive and fuse features from the first and second paths and capture global contextual dependencies. In the cross-scale attention aggregation mechanism, feature maps from different levels of the encoder are aligned at multiple scales, and shallow detail features and deep semantic features are fused by channel-space dual attention weighting. The attention-weighted multi-scale features are residually connected with the output of the bidirectional state propagation unit to gradually reconstruct high-resolution feature maps for final prediction.

[0103] The decoder architecture employs a forward-backward information transmission mechanism, receiving features from both ends of the encoder to enhance global context understanding. State propagation is implemented using a gated recurrent unit (GRU) structure. The formula is as follows:

[0104] , ,

[0105] ,

[0106] in, , For the first The forward and backward hidden states at each time step , This represents the forward state of the previous time step and the reverse state of the next time step. , The feature inputs come from the forward and backward inputs of the encoder. It is a gated loop unit used for sequence state propagation. This represents the final state of bidirectional fusion.

[0107] A cross-scale attention aggregation mechanism is proposed, and a multi-scale feature alignment module is designed to align the features of the encoder's dual pathways in both spatial and channel dimensions. Features are then dynamically fused using a channel-space dual attention mechanism.

[0108] ,

[0109] in, The fused feature map These are feature maps from local and global pathways. These are channel attention weights and spatial attention weights (learnable parameters). This is element-wise multiplication (Hadamard product). The formula implements a gated fusion mechanism: through... Dual attention weighting (channel and spatial) is applied to both local and global features; a dynamic fusion weight map is generated using the Sigmoid algorithm, with values ​​between 0 and 1; this weight map controls the fusion ratio of local and global features. The weighting component is used to preserve local features; The remaining part is used to preserve global features, and the final output is... It is an adaptive fusion result of local details and global context.

[0110] S3. Perform co-training and model compression on the hybrid network model to obtain a simplified prediction model. Model optimization and deployment assurance: This step ensures that the complex model built in step two can be trained efficiently and stably, and ultimately transformed into a lightweight model that can be deployed in real-world environments. Co-training utilizes advanced training strategies such as adversarial training to improve the model's performance and training stability, making its prediction results closer to the true distribution. Model compression uses techniques such as knowledge distillation to significantly reduce the model size and computational overhead while maintaining accuracy as much as possible, enabling it to be applied to edge devices with limited computing resources or to achieve real-time prediction.

[0111] The co-training and model compression of the hybrid network model includes:

[0112] A collaborative training framework based on generative adversarial networks (GANs) is adopted, where the generator is the hybrid network model and the discriminator distinguishes between the predicted results and the true labels. The generator learns the complex mapping relationship from the input data to the predicted disease results. Adversarial training forces the generator to produce predictions that are closer to the true distribution, improving the model's ability to model complex disease patterns. The discriminator evaluates the distributional differences between the predicted results and the true labels, providing adversarial feedback signals to guide the generator optimization and ensure that the predicted results are statistically close to the actual observed data. This framework can effectively handle noise and uncertainty in road disease data, generating more realistic and reliable disease evolution prediction results.

[0113] A distributed gradient constraint algorithm is employed during training, distributing gradient penalty calculations across multiple training batches for parallel processing. This improves training stability and efficiency. Distributed gradient penalty computation: The computationally intensive gradient penalty term is decomposed into multiple smaller batches for parallel processing, reducing memory requirements and computational complexity per iteration, supporting larger batch training and more complex model structures. Training stability assurance: Gradient constraints prevent training instability caused by an overly strong discriminator, ensuring co-evolution of the generator and discriminator during training and avoiding pattern collapse and training divergence. This algorithm enables stable training of complex hybrid network models on large-scale road monitoring data, guaranteeing convergence and generalization performance.

[0114] By combining a dynamic adaptive learning rate control mechanism, the learning rate is dynamically adjusted based on the gradient norm and performance metrics during training, achieving intelligent optimization of the training process. Specifically, gradient adaptive adjustment automatically adjusts the learning rate based on the gradient magnitude, increasing the learning rate in flat regions to accelerate convergence and decreasing it in steep regions to maintain stability, thus improving convergence speed and training stability. Performance-oriented adjustment dynamically adjusts the learning rate strategy based on validation set performance, maintaining aggressive learning when performance improves and adjusting promptly when performance declines to avoid overfitting and premature convergence.

[0115] After training, a multi-level feature mapping transfer strategy is employed to use the trained hybrid network model as the teacher model, transferring its feature extraction capabilities to the structurally simplified student model, thus obtaining the simplified prediction model. After training, knowledge distillation is used to use the trained hybrid network model as the teacher model, transferring its feature extraction capabilities to the structurally simplified student model.

[0116] Specifically, this strategy employs several methods: First, through feature layer knowledge transfer, the student model learns the feature representations of the teacher model's intermediate layers, retaining the teacher model's feature extraction capabilities and acquiring feature discrimination abilities similar to the teacher model. Second, through attention transfer, the teacher model's attention distribution pattern is transferred, allowing the student model to focus on the same important feature regions as the teacher model, thus improving the student model's sensitivity to key disease features. Third, through output layer distillation, the soft labels from the teacher model guide the student model's training, transferring the inter-category relationship information learned by the teacher model and improving the student model's classification accuracy and generalization ability. This strategy maintains prediction accuracy while compressing a complex model into a lightweight model suitable for embedded device deployment, meeting the needs of practical engineering applications.

[0117] S4. Utilize the simplified prediction model to predict facility condition. Based on the prediction results, construct a multi-criteria optimization objective and use a genetic algorithm to solve for the non-dominated solution set. This step bridges the gap between prediction and optimization decision-making, transforming the model's prediction results into a quantifiable, multi-objective engineering optimization problem, which is then solved using intelligent algorithms. Facility condition prediction involves calling the model obtained in step three and executing its core prediction function. Constructing a multi-criteria optimization objective links the predicted future condition with multiple real-world factors such as maintenance costs, safety risks, and construction impacts, establishing a comprehensive evaluation system. Using a genetic algorithm efficiently searches for a series of optimal compromise solutions (Pareto solutions) in this complex space of multiple objectives that may conflict, providing decision-makers with multiple scientific options.

[0118] Specifically, in this embodiment, the simplified prediction model is used to predict the condition of road facilities. A multi-criteria optimization objective is constructed, and multi-period images of the road facilities to be predicted are input into the trained simplified prediction model to obtain quantitative prediction results of the type, severity, and development trend of the defects in future time periods. This includes:

[0119] Multi-period images of road facilities to be predicted are input into the simplified prediction model to obtain quantitative predictions of the type, severity, and development trend of road defects in future periods. Historical multi-period image data is analyzed to capture the temporal patterns of defect development, establishing a mapping relationship from historical conditions to future development, thus achieving accurate prediction of the defect evolution trajectory. Specific defect types such as cracks, potholes, and ruts are identified, and defect level scores (0-1 continuous values ​​or discrete levels) are output to predict the rate of expansion and direction of defect development. This step provides a data foundation for subsequent optimization decisions, ensuring that maintenance plans are based on scientific predictions.

[0120] Based on the quantitative prediction results, a multi-criteria optimization objective function is constructed, encompassing maintenance economic costs, facility structural safety levels, and construction traffic impact coefficients. The economic cost objective (f1) includes: direct maintenance costs (materials, labor, and equipment); traffic impact costs (traffic delays and detours caused by construction); and risk costs (structural risk costs due to untimely maintenance). The structural safety objective (f2) includes: safety level assessment (structural safety score based on defect prediction); reliability index (reliability of the facility within its expected service life); and negative optimization (transforming safety maximization into cost minimization). The comprehensive impact objective (f3) includes: traffic impact coefficient (the degree of impact of construction on traffic flow); and environmental impact assessment (noise, pollution, and other environmental impacts during construction). This function transforms the complex maintenance decision-making problem into a quantifiable multi-objective optimization problem, laying the foundation for algorithmic solutions.

[0121] A non-dominated sorting genetic algorithm with an elitist strategy is employed to solve the multi-criteria optimization objective function in parallel, iteratively evolving the population through crossover, mutation, and selection operations. Non-dominated sorting stratifies the solutions according to Pareto dominance, retaining individuals that perform well across different objectives to ensure population diversity and convergence. Crowding density is calculated to measure the distribution density of solutions in the objective space, prioritizing sparsely distributed individuals within the same non-dominated level to maintain the diversity and uniformity of the solution set. An elitist retention strategy ensures that outstanding individuals from each generation directly enter the next, preventing the loss of superior genes, accelerating convergence, and improving the algorithm's convergence speed and stability. This algorithm effectively handles the trade-offs among the three optimization objectives, finding a high-quality solution set that balances economy, safety, and impact.

[0122] After a predetermined number of iterations, the output is a non-dominated solution set containing multiple Pareto optimal solutions, where each solution represents a maintenance scheme that achieves balance among multiple optimization objectives. By identifying non-dominated solutions, individuals that are not dominated by other solutions on any objective are selected, constructing a complete Pareto front and providing a comprehensive decision-making space. By setting solution set diversity, it is ensured that the solution set covers all regions of the Pareto front, providing decision-makers with diverse options to adapt to different decision preferences and constraints.

[0123] This step outputs a variety of maintenance options with different focuses, such as: Economy-first: lowest cost but longer construction period; Safety-first: immediate construction to ensure structural safety but higher cost; Balanced: achieving a balance between cost, safety and impact.

[0124] S5. Determine the final solution from the non-dominated solution set based on decision preferences, generate a maintenance decision report, and visualize it. The final decision output and human-computer interaction: This step transforms the mathematical solution set output by the optimization algorithm into a concrete and feasible maintenance plan that human decision-makers can understand and execute, and achieves efficient interaction through visualization. Determining the final solution based on decision preferences involves combining the priorities of specific scenarios (such as prioritizing cost control in a limited budget, or prioritizing safety in sensitive road sections) to select the most suitable solution from multiple optimal solutions. Generating a maintenance decision report and visualizing it presents the final decision solution in a structured report and intuitive charts, lowering the understanding threshold and supporting rapid and scientific decision-making.

[0125] Specifically, in this embodiment, determining the final solution based on decision preferences and generating a maintenance decision report includes:

[0126] Establish a decision preference matrix that includes cost weights, safety priorities, and impact tolerance; by establishing this matrix, the value orientations and priorities of decision-makers can be quantified. For example, by defining weights for dimensions such as cost control, safety requirements, and impact tolerance, subjective preferences can be transformed into calculable evaluation criteria, providing a unified measurement scale for scheme selection.

[0127] The approximation-ideal-solution ranking method is used to comprehensively evaluate and rank the schemes in the non-dominated solution set. Through the evaluation of the approximation-ideal-solution ranking method, multiple alternative schemes are scientifically ranked, the degree of closeness of each scheme to the ideal optimal solution is calculated, and the relative advantages and disadvantages of multiple evaluation indicators are comprehensively considered to generate an objective and quantitative priority order of the schemes.

[0128] Based on the ranking results, the optimal solution is selected, and a maintenance decision report is generated that includes specific maintenance measures, implementation time windows, resource allocation plans, and expected results. The optimal solution is determined and the report is generated, forming an executable maintenance plan and detailed schedule. Based on the ranking results, the solution that best meets the requirements is selected, such as formulating specific maintenance measures and time arrangements, planning resource allocation, and predicting implementation results.

[0129] The maintenance decision report is displayed through a visual interface, including a disease development trend chart, a maintenance plan comparison chart, and a resource allocation Gantt chart. The visual interface intuitively presents the decision-making basis and plan content, such as using charts to show disease development trends, comparing the advantages and disadvantages of different plans, and using Gantt charts to clearly define resource allocation plans and timelines.

[0130] These five steps are interconnected, forming a complete technological closed loop from data perception → intelligent prediction → model optimization → multi-objective decision-making → solution output. Step two is the core technology for achieving accurate prediction, step one is its foundation, step three is its guarantee, and steps four and five are the key applications for transforming its predictive capabilities into practical engineering value.

[0131] Example 2

[0132] like Figure 2 As shown in the figure, this application provides an architecture diagram of a road facility disease evolution prediction and maintenance decision system based on a CNN-Transformer hybrid architecture, which is applied to the road facility disease evolution prediction and maintenance decision system based on a CNN-Transformer hybrid architecture as described in Embodiment 1. It includes a data acquisition and preprocessing module 11, a hybrid network model construction module 12, a model training and compression module 13, a state prediction and optimization module 14, and a decision generation and display module 15.

[0133] The data acquisition and preprocessing module 11 is used to acquire aerial images of the target road facilities from multiple periods, construct a time-series dataset, and perform spatial alignment and simulated degradation processing.

[0134] Hybrid network model building module 12 is used to build a hybrid network model with an encoder-decoder structure. The encoder adopts a dual-path structure. The first path extracts local features through convolutional blocks, and the second path is based on an attention mechanism and integrates edge enhancement components. The decoder adopts a bidirectional propagation unit and fuses features through a cross-scale attention mechanism.

[0135] The model training and compression module 13 is used to perform collaborative training and model compression on the hybrid network model to obtain a simplified prediction model.

[0136] The state prediction and optimization module 14 is used to predict the state of facilities using the simplified prediction model, construct a multi-criteria optimization objective based on the prediction results, and use a genetic algorithm to solve for the non-dominated solution set.

[0137] The decision generation and display module 15 is used to determine the final solution from the non-dominated solution set according to decision preferences, generate a maintenance decision report and display it visually.

[0138] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 101 and memory 100, communication interface 103, and bus 102.

[0139] In this embodiment of the application, memory 100 is used to store executable instructions of processor 101, which, when configured to execute instructions, implements the method as described in the first aspect.

[0140] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0141] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0142] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0143] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0144] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0145] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0146] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture, characterized in that, Includes the following steps: Acquire aerial images of the target road facilities from multiple periods, construct a time-series dataset, and perform spatial alignment and simulated degradation processing; Construct a hybrid network model with an encoder-decoder structure. The encoder adopts a dual-path structure. The first path extracts local features through convolutional blocks, and the second path is based on an attention mechanism and integrates an edge enhancement component. The decoder adopts a bidirectional propagation unit and fuses features through a cross-scale attention mechanism. The hybrid network model is co-trained and compressed to obtain a simplified prediction model; The simplified prediction model is used to predict facility status. Based on the prediction results, a multi-criteria optimization objective is constructed, and a genetic algorithm is used to solve for the non-dominated solution set. The final solution is determined from the non-dominated solution set based on decision preferences, and a maintenance decision report is generated and visualized. The hybrid network model for constructing the encoder-decoder structure includes: In the dual-path structure of the encoder, the first path uses inverse dense connecting blocks containing depthwise separable convolution to extract local subtle features of the facility state layer by layer. The second path integrates a direction-sensitive edge-enhancing filter component at the front end to strengthen contour information and models the global semantics and long-term dependencies of facility status based on a hierarchical sliding window attention mechanism. The decoder employs a bidirectional state propagation unit, which enhances global context understanding through forward-backward information transmission, and introduces a cross-scale attention aggregation mechanism to dynamically fuse feature information from both paths of the encoder. The co-training and model compression of the hybrid network model includes: A collaborative training framework based on generative adversarial networks is adopted, wherein the generator is the hybrid network model described above, and the discriminator is used to distinguish the predicted results from the real labels; During training, a distributed gradient constraint algorithm is used to distribute gradient penalty calculations across multiple training batches for parallel processing. Combined with a dynamic adaptive learning rate control mechanism, the learning rate is dynamically adjusted based on the gradient norm and performance indicators during the training process; After training, a multi-level feature mapping transfer strategy is adopted to use the trained hybrid network model as the teacher model and transfer its feature extraction capabilities to the simplified student model to obtain the simplified prediction model.

2. The method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The process of acquiring multi-period aerial images of the target road facilities, constructing a time-series dataset, and performing spatial alignment and simulated degradation processing includes: Image registration and spatial cropping are performed on aerial images from multiple periods to construct a spatiotemporally aligned image sequence; After registration, disease areas are labeled to form a disease image sample set. The image sequence is subjected to simulated degradation operations, including degrading filtering and random interference, to enhance the diversity and robustness of the data; Based on labeled and enhanced image sequences, a state evolution time series dataset is constructed for model training.

3. The method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, in, The first path uses inverse dense connect blocks containing depthwise separable convolutions to extract local subtle features in the following way: In the reverse dense connection block, the input features are first separated into channels and spatial features are extracted by depthwise separable convolution. Then, a reverse dense connection structure is adopted to fuse shallow features with deep features that have undergone nonlinear transformation across layers. A channel attention mechanism is introduced during the feature fusion process to dynamically adjust the weight distribution of features in each channel; By cascading multiple levels of the aforementioned reverse dense connecting blocks, the ability to express the characteristics of local details of facility defects is gradually enhanced.

4. The method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The second pathway models global semantics and long-term dependencies in the following way: In the direction-sensitive edge enhancement filtering component, a multi-directional Gabor filter bank is used to extract texture features at different angles, and a Laplacian Gaussian filter is combined to enhance the edge response; In the hierarchical sliding window attention mechanism, the feature map is divided into multiple non-overlapping local windows. Self-attention is calculated within each window to reduce computational complexity, and cross-window information interaction is achieved through window shifting operations. A pyramid structure is constructed by stacking multiple hierarchical sliding window attention layers to capture multi-scale semantic information from local to global. The global semantic features output from each level are passed to the decoder through skip connections, and together with the corresponding local features from the first path, they constitute the input of the cross-scale attention aggregation mechanism.

5. The method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The decoder achieves feature fusion and information transmission in the following ways: In the bidirectional state propagation unit, a bidirectional processing mechanism of forward scanning and backward scanning is adopted to propagate state information in the forward and backward directions along the spatial dimension, respectively, to receive and fuse features from the first path and the second path, and to capture global context dependencies. In the cross-scale attention aggregation mechanism, feature maps from different levels of the encoder are aligned at multiple scales, and shallow detail features and deep semantic features are fused by channel-space dual attention weighting. The attention-weighted multi-scale features are residually concatenated with the output of the bidirectional state propagation unit to progressively reconstruct high-resolution feature maps for final prediction.

6. The method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The simplified prediction model is used to predict facility status, and a multi-criteria optimization objective is constructed, including: By inputting multi-period images of road facilities to be predicted into the simplified prediction model, quantitative prediction results of the type, severity and development trend of the defects in the future period are obtained. Based on the quantitative prediction results, a multi-criteria optimization objective function is constructed, which includes maintenance economic costs, facility structural safety level, and construction traffic impact coefficient. The multi-criteria optimization objective function is solved in parallel using a non-dominated sorting genetic algorithm with an elitist strategy, and the population is iteratively evolved through crossover, mutation, and selection operations. After a predetermined number of iterations, the output is a non-dominated solution set containing multiple Pareto optimal solutions, where each solution represents a maintenance scheme that achieves balance among multiple optimization objectives.

7. The method for predicting the evolution of road infrastructure defects and making maintenance decisions based on a CNN-Transformer hybrid architecture according to claim 6, characterized in that, The step of determining the final solution based on decision preferences and generating a maintenance decision report includes: Establish a decision preference matrix that includes cost weights, security priorities, and impact tolerance; The schemes in the non-dominated solution set are comprehensively evaluated and ranked using the approximation of ideal solution ranking method; Select the optimal solution based on the ranking results and generate a maintenance decision report that includes specific maintenance measures, implementation time windows, resource allocation plans, and expected results. The maintenance decision report is displayed through a visual interface, including a disease development trend chart, a maintenance plan comparison chart, and a resource allocation Gantt chart.

8. A road infrastructure defect evolution prediction and maintenance decision-making system based on a CNN-Transformer hybrid architecture, applied to the road infrastructure defect evolution prediction and maintenance decision-making method based on a CNN-Transformer hybrid architecture as described in any one of claims 1 to 7, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire multi-period aerial images of the target road facilities, construct a time-series dataset, and perform spatial alignment and simulated degradation processing. The hybrid network model building module is used to build a hybrid network model with an encoder-decoder structure. The encoder adopts a dual-path structure. The first path extracts local features through convolutional blocks, and the second path is based on an attention mechanism and integrates edge enhancement components. The decoder adopts a bidirectional propagation unit and fuses features through a cross-scale attention mechanism. The model training and compression module is used to perform co-training and model compression on the hybrid network model to obtain a simplified prediction model. The state prediction and optimization module is used to predict the state of facilities using the simplified prediction model, construct a multi-criteria optimization objective based on the prediction results, and solve the non-dominated solution set using a genetic algorithm. The decision generation and display module is used to determine the final solution from the non-dominated solution set according to decision preferences, generate and maintain decision reports, and display them visually.

Images