Pavement crack semantic segmentation method based on Transform and CNN architecture

Abstract

The invention discloses a pavement crack semantic segmentation method based on Transform and CNN architecture, and relates to the technical field of pavement crack detection, and the method comprises the steps: obtaining a pavement image, marking a crack disease image in the pavement image through an image marking tool, and constructing a data set based on the marked crack disease image; preprocessing the data set, and constructing and training a self-adaptive semantic segmentation model combining a self-attention neural network and a convolutional neural network based on the preprocessed data set; and deploying the trained adaptive semantic segmentation model to a local platform, and inputting the crack disease image into the adaptive semantic segmentation model for segmentation and marking. According to the method, a dynamic scale selector composed of a texture perceptron and a scale regulation and control unit is introduced, the size of a convolution kernel is dynamically adjusted according to the local texture complexity of an image, and fine modeling of different structure areas is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road crack detection, in particular to a road crack semantic segmentation method based on a Transformer and a CNN architecture. BACKGROUND

[0002] As the core of the urban transportation system, the health of the road directly affects the traffic safety and transportation efficiency. With the advancement of urbanization, road cracks, as the most common disease, not only accelerate the deterioration of road structure, but also can cause secondary diseases, leading to safety hazards and increased maintenance costs. Traditional manual detection methods are inefficient and subjective. Although early image processing techniques such as Canny edge detection and threshold segmentation have achieved preliminary automation, they are easily affected by noise and light interference in complex environments, making it difficult to meet the high-precision detection requirements.

[0003] The introduction of deep learning technology, especially convolutional neural networks (CNN), has significantly improved the automation level of crack identification. However, when processing high-resolution images, CNNs have the drawbacks of low computational efficiency and insufficient global information capture ability. In the face of challenges such as diverse crack patterns, sample imbalance, and low contrast, CNNs have limited performance in fine-grained segmentation. Therefore, researchers have attempted to integrate Transformer architecture with CNN to complement the global modeling ability of Transformer with the local feature extraction advantage of CNN, improving the ability to represent crack details while considering global context information, providing a new approach to semantic segmentation tasks.

[0004] Although the Transformer-CNN hybrid model has achieved some progress, existing technologies still have room for optimization in terms of model complexity, real-time performance, and detection accuracy balance. Therefore, there is an urgent need to design a road crack semantic segmentation method that balances high accuracy, low computational cost, and strong robustness to meet the needs of rapid response and accurate decision-making in road maintenance. SUMMARY

[0005] To address the problems in the related art, the present application proposes a road crack semantic segmentation method based on a Transformer and a CNN architecture to overcome the above technical problems existing in the prior art.

[0006] To this end, the specific technical solutions adopted by the present application are as follows:

[0007] The road crack semantic segmentation method based on the Transformer and the CNN architecture comprises:

[0008] Obtaining a road image, labeling crack disease images in the road image using an image labeling tool, and constructing a data set based on the labeled crack disease images;

[0009] Preprocess the data set, and build and train an adaptive semantic segmentation model combining self-attention neural network and convolutional neural network based on the preprocessed data set;

[0010] Deploy the adaptive semantic segmentation model after training to a local platform, and input the crack disease image into the adaptive semantic segmentation model for segmentation annotation.

[0011] Preferably, the adaptive semantic segmentation model combining self-attention neural network and convolutional neural network comprises:

[0012] A global feature capturing module for taking a FastViT visual network as a backbone network and capturing image global features through the FastViT visual network;

[0013] A multi-scale adaptive module for dynamically allocating an optimal processing scale based on a scale mask;

[0014] A local feature enhancement module for extracting image local feature representation;

[0015] A feature perception module for fusing the SAM-CAFusion module with deformable convolution to obtain the perception ability of the adaptive semantic segmentation model for irregular geometric structures;

[0016] A loss function module for constructing a loss function by combining a dice coefficient loss and a focal loss, and introducing a modulation coefficient in the loss function to balance the segmentation annotation task samples;

[0017] A feature critical point detection module for constructing a sequence parameter monitoring model between the global feature capturing module and the local feature enhancement module, and balancing the global feature and the local feature through the sequence parameter monitoring model.

[0018] Preferably, the FastViT visual network is a four-stage network architecture, the first three stages have the same network architecture, and the FastViT visual network processes global features through a deep convolution layer and a batch normalization layer in the first three stages.

[0019] The FastViT visual network utilizes a self-attention mechanism for feature fusion in the fourth stage.

[0020] Preferably, the multi-scale adaptive module comprises the following when dynamically allocating an optimal processing scale based on a scale mask:

[0021] A pre-defined shared feature extraction network is used to generate a unified feature map, which is input into a plurality of parallel scale adaptive branches, and the adaptive branches perform feature extraction according to a pre-set scale range.

[0022] The scale probability distribution is generated pixel by pixel by using a predefined lightweight scale estimator, and the optimal processing scale mask of each pixel is dynamically assigned according to the scale probability distribution;

[0023] The feature extraction results output by each adaptive branch and the optimal processing scale mask of each pixel are weightedly fused to generate a final image segmentation result.

[0024] Preferably, the local feature enhancement module comprises the following when extracting the local feature representation of the image:

[0025] The input feature is convoluted by using the Stem module, and batch normalization and ReLU activation function processing are sequentially performed after the convolution processing is completed to obtain smoothed features;

[0026] The smoothed features are respectively input into a lightweight MLP branch and a learnable visual center branch, wherein:

[0027] The lightweight MLP branch is used to capture the global long-distance dependence of the smoothed features, and the learnable visual center branch is used to encode the features and map them to a predefined visual center to capture the local region information of the smoothed features;

[0028] The output features of the lightweight MLP branch and the learnable visual center branch are spliced to obtain the local feature representation of the image;

[0029] The local feature representation of the image is compared with a standard reference image, and a random matrix theory spectral matrix is constructed based on the comparison result, and the free product relationship between the learnable visual center branch and the lightweight MLP branch is regulated through the random matrix spectral theory matrix;

[0030] Whether the adaptive semantic segmentation model has structural overfitting is predicted based on the free product relationship, if yes, the local feature representation of the image is structurally optimized, otherwise, the current local feature representation of the image is taken as the final output of the local feature enhancement module.

[0031] Preferably, the calculation formula of the lightweight MLP branch is:

[0032] ;

[0033] ;

[0034] In the formula, DConv represents a depth convolution operation, GN represents group normalization, CMLP represents a channel lightweight MLP branch, represents input features enhanced by depth convolution and normalization, represents the processing result of the channel lightweight MLP branch, X in represents that the input feature is a sub-branch feature stream independently processed in the module;

[0035] The calculation formula of the learnable visual center branch is:

[0036]

[0037] In the formula, b represents the encoded features, b k s represents the kth visual center, s k K represents the number of visual centers, e k N represents the number of input features, i and j represent feature indexes and auxiliary index variables in the summation process, respectively.

[0038] Preferably, a random matrix theory spectrum matrix is constructed based on the comparison results, and the free product relationship between the learnable visual center branch and the lightweight MLP branch is regulated by the random matrix spectrum theory matrix, including:

[0039] Based on the comparison results, the learnable visual center branch and the lightweight MLP branch are constructed into feature matrices respectively, and the weight of each feature matrix is calculated using a predefined empirical spectrum distribution function;

[0040] According to the weight of each feature matrix, the distribution evolution of the spectrum distribution structure of the learnable visual center branch and the lightweight MLP branch is simulated by wavelet Brownian motion;

[0041] Based on the single ring theory, a distribution geometric property model is established for the spectrum distribution structure of each branch on the complex plane, and the spectral radius range, condition number and structural sparsity of each feature matrix are adjusted through the distribution geometric property model;

[0042] An exponential test function is set to amplify the spectral region of each feature matrix, and the eigenvalue set of the amplified feature matrix is obtained to form a global structure measurement index;

[0043] The proportion of the learnable visual center branch and the lightweight MLP branch is controlled by the global structure measurement index combined with the average spectral radius to regulate the spectrum distribution structure tension until it meets the free product relationship between the learnable visual center branch and the lightweight MLP branch.

[0044] Preferably, the distribution evolution of the spectrum distribution structure of the learnable visual center branch and the lightweight MLP branch is simulated by wavelet Brownian motion, including:

[0045] The wavelet basis function is defined, and the wavelet basis function covers the low-frequency global features and high-frequency local features of the learnable visual center branch and the lightweight MLP branch;

[0046] The outputs of the visual branch and the MLP branch are projected into different frequency bands through wavelet transform, the frequency band energy is calculated, and the frequency band energy is used as a disturbance intensity modulation factor;​

[0047] The wavelet basis function is introduced into the disturbance intensity modulation factor, the wavelet inverse transform is performed on the wavelet basis function after the disturbance, and the time interval of the Brownian motion is set. The simulated Brownian motion path is reconstructed after the wavelet inverse transform;

[0048] The spectral distribution structure is analyzed in the Brownian motion path, and the distribution evolution of the spectral distribution structure is obtained based on the spectral analysis result.

[0049] Preferably, the feature perception module includes the following when the SAM-CAFusion module is fused with the deformable convolution to obtain the perception ability of the adaptive semantic segmentation model for irregular geometric structures:

[0050] The coordinate attention module is used to generate an attention map, and the position encoding vector of the attention map is used to guide the backbone feature extraction network to focus on the crack area, so as to realize global coarse positioning of the crack area.

[0051] The interest region is extracted from the attention map, the interest region is obtained through the guided deformable convolution module, the attention offset of the interest region is calculated, and the position offset of the convolution kernel is dynamically adjusted based on the attention offset.

[0052] The scale selector is introduced into the guided deformable convolution module, the scale selector is used to adaptively select the convolution kernel according to the local image texture complexity, and the representation requirement of the crack area at different scales is obtained.

[0053] Preferably, the feature critical point detection module is used to construct a sequence parameter monitoring model between the global feature capture module and the local feature enhancement module, and the sequence parameter monitoring model is used to balance the global feature and the local feature, including:

[0054] A sequence parameter is constructed in the overlapping area of the global feature and the local feature, a critical threshold is defined, and the sequence parameter and the critical threshold are compared;

[0055] If the sequence parameter is greater than the critical threshold, it indicates that the adaptive semantic segmentation model is in a global dominant mode, the shallow parameters of the global feature capture module are frozen, and a heterogeneous convolution kernel is introduced in the local feature enhancement module.

[0056] If the sequence parameter is less than or equal to the critical threshold, it indicates that the adaptive semantic segmentation model is in a local dominant mode, and a non-local attention compensation mechanism is inserted in the global feature capture module.

[0057] The global dominant mode feedback and the local dominant mode feedback are extracted, and a bidirectional closed-loop control mechanism is constructed, and the bidirectional closed-loop control mechanism includes:

[0058] The sequence parameter gating is integrated into the FastViT visual network; and the critical threshold is corrected through gradient reverse propagation of the local feature enhancement module.

[0059] The present application has the following advantages:

[0060] 1、The present application combines the Transformer and CNN architectures to establish a semantic segmentation model, processes different scale targets through parallel branches, introduces a lightweight scale estimator to realize pixel-level dynamic scale selection, fuses EVC Block to enhance the detail perception ability, combines a SAM-CAFusion module to optimize edge modeling, and dynamically adjusts the convolution kernel size, thereby improving the robustness and segmentation performance of the model in a multi-scale scene.

[0061] 2、The present application generates a unified feature map through a shared feature extraction network, inputs it into multiple parallel scale adaptive branches, each branch extracts features for different scale targets, then introduces a lightweight scale estimator to generate a pixel-by-pixel scale probability distribution, realizes pixel-level dynamic scale selection, and fuses the multi-branch features according to the scale mask in the fusion stage to improve the segmentation accuracy.

[0062] 3、The present application fuses the SAM-CAFusion module of the coordinate attention and the deformable convolution, guides the attention shift and the dynamic adjustment of the convolution kernel through the image gradient and the edge direction information, accurately models the complex geometric edge structure, and introduces a dynamic scale selector composed of a texture perception device and a scale regulation unit to dynamically adjust the convolution kernel size according to the local texture complexity of the image (such as the gradient change rate and the edge density), realizing fine modeling of different structure regions.

[0063] 4、The present application sets up a scale perception interaction mechanism, including a multi-scale feature fusion and cross-attention module, captures complementary information between different scales through scale alignment, gated fusion and attention weighting, realizes cross-scale collaborative perception and information sharing, and dynamically adjusts the perception range in the multi-scale path, thereby significantly enhancing the adaptability and segmentation accuracy of the model to complex crack structures.

[0064] 5、The loss function in the present application combines Focal Loss and Dice Loss, and by introducing an adjustment factor, the semantic segmentation model can pay more attention to difficult-to-classify areas, thereby improving the performance of difficult-to-segment areas, and effectively balancing the problem of sample imbalance in the pavement crack segmentation task. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0066] Figure 1 is a flow chart of a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0067] Figure 2 is a principle block diagram of an adaptive semantic segmentation model in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0068] Figure 3 is one of network structure diagrams of a backbone network FastViT in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0069] Figure 4 is another one of network structure diagrams of a backbone network FastViT in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0070] Figure 5 is a SAM multi-scale adaptive module diagram in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0071] Figure 6 is an EVC Block network structure diagram in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0072] Figure 7 is a light coordinate attention mechanism structure diagram in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application;

[0073] Figure 8 is a CAFusion network structure diagram of a feature fusion module in a road crack semantic segmentation method based on a Transformer and CNN architecture according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention. They are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. By referring to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and the advantages of the present invention.

[0075] According to an embodiment of the present invention, a pavement crack semantic segmentation method based on Transformer and CNN architecture is provided.

[0076] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a pavement crack semantic segmentation method based on a Transformer and CNN architecture includes:

[0077] S1. Obtain a pavement image, use an image annotation tool to annotate crack disease images in the pavement image, and construct a dataset based on the annotated crack disease images.

[0078] It should be noted that pavement images are obtained from the Pavementscapes dataset, and images containing only crack damage are relabeled using image annotation tools (such as the Labelme tool) to obtain a JSON file containing pavement crack information, thereby constructing the dataset.

[0079] S2. Preprocess the dataset, and build and train an adaptive semantic segmentation model that combines self-attention neural network and convolutional neural network based on the preprocessed dataset.

[0080] It should be noted that the labeled dataset is converted into the standard VOC format, and all images are uniformly adapted to a size of 512×512 before being input into the model. Data enhancement is performed by flipping, symmetry, scaling, length and width twisting, and random adjustment of brightness and contrast. The dataset is divided into a training validation set and a test set in a 4:1 ratio, where the training validation set is divided into a training set and a validation set in a 4:1 ratio.

[0081] like Figure 2 As shown in the figure, based on FastViT (Fast Hybrid Visual Transformer) as the backbone network, the SAM (Multi-scale Adaptive Module) adaptive module is used to introduce EVC Block to build a separate branch (Explicit Visual Center Module), and the SAM-CAFusion (Multi-scale Adaptive Visual Center Module) module is used to fuse global and local information. Focal Dice Loss is used as the loss function for semantic segmentation training, and the performance of the trained semantic segmentation model is verified using the divided test set.

[0082] It should be noted that the self-attention neural network is a Transformer neural network model, the Transformer neural network model discards the traditional recurrent neural network (RNN) and convolutional neural network (CNN), and instead adopts a self-attention mechanism (Self-Attention) to process sequence data, and the self-attention mechanism can capture the dependency between different positions in the sequence, thereby better modeling sequence data.

[0083] The adaptive semantic segmentation model combining the self-attention neural network and the convolutional neural network comprises:

[0084] The global feature capturing module is configured to use the FastViT visual network as a backbone network and capture global image features through the FastViT visual network.

[0085] The FastViT visual network is a four-stage network architecture, the first three stages have the same network architecture, and the FastViT visual network processes global features through a deep convolutional layer and a batch normalization layer in the first three stages; the FastViT visual network uses a self-attention mechanism for feature fusion in the fourth stage.

[0086] It should be noted that, as shown in Figures 3-4 FastViT is introduced as a backbone network: the advantages of the FastViT hybrid visual Transformer architecture are used to effectively capture long-range dependencies in images, extract high-dimensional global features, and significantly improve the performance of the model in complex image segmentation tasks. Compared with traditional CNN architectures, FastViT significantly reduces the parameter quantity and computational complexity of the model while maintaining high accuracy, improving detection efficiency. FastViT adopts a four-stage architecture, the resolution of each stage is gradually halved, and the number of channels is doubled, the first three stages use the same architecture, and the features are processed through deep convolution (DWConv) and batch normalization (BN), the calculation formula is as follows:

[0087] ;

[0088] In the formula, Y1 represents the output of the first stage, X1 represents the feature map input to the first stage, DWConv represents the deep convolutional layer, and BN represents the batch normalization. During inference, through structure reparameterization, the nonlinear activation function and batch normalization are removed, and the calculation formula is as follows:

[0089] ;

[0090] In the formula, Y2 represents the output of the second stage, X2 represents the input feature map to the second stage, the RepMixer convolutional mixing module is the core module of FastViT, which is responsible for mixing the input tokens (features), but it is different from the traditional Transformer model, which does not use a skip connection, thereby reducing the memory access cost and reducing the delay, the RepMixer convolutional mixing module uses the formula as follows during training:

[0091] ;

[0092] In the formula, Y3 represents the output of the RepMixer convolutional mixing module, σ represents a nonlinear activation function, and X3 represents the feature map input to the RepMixer convolutional mixing module. During inference, the formula is simplified through structural reparameterization as follows:

[0093] ;

[0094] In the formula, Y4 represents the output of the fourth stage, X4 represents the output of the fourth stage, and FastViT uses a self-attention mechanism in the fourth stage to fuse tokens (features), while the feedforward network (FFN) uses a convolution structure (ConvFFN) with a large kernel (7x7), and the calculation formula is as follows:

[0095] ;

[0096] In the formula, Y5 represents the output of the last convolution structure (ConvFFN), and X5 represents the feature map input to the last convolution structure (ConvFFN).

[0097] Patch Embedding (image block embedding) uses a 7x7 large kernel deep convolution and a 1x1 convolution during inference, and a 3x3 convolution branch is added during training for reparameterization. Position encoding is generated dynamically with conditional position encoding and combined with Patch Embedding to enhance the relative position perception ability of the model. Through the convolution and reparameterization strategy, the overall architecture realizes efficient calculation and inference.

[0098] The multi-scale adaptive module is used to dynamically allocate the optimal processing scale based on a scale mask.

[0099] The multi-scale adaptive module includes the following when dynamically allocating the optimal processing scale based on a scale mask:

[0100] A unified feature map is generated using a predefined shared feature extraction network, and the unified feature map is input into a plurality of parallel scale adaptive branches. The adaptive branches perform feature extraction according to a predetermined scale range;

[0101] The scale probability distribution is generated pixel by pixel using a predefined lightweight scale estimator, and the optimal processing scale mask of each pixel is dynamically assigned according to the scale probability distribution;

[0102] The feature extraction results output by each adaptive branch are fused with the optimal processing scale mask of each pixel, and the final image segmentation result is generated.

[0103] It should be noted that, as Figure 5 shown, the multi-scale adaptive module SAM is used to solve the performance degradation problem caused by target scale difference in semantic segmentation. The module is implemented based on the following process: a unified feature map is generated through a shared feature extraction network, which is then input into multiple parallel scale adaptive branches, each branch performing feature extraction on targets of a specific scale range; a lightweight scale estimator (including convolutional layers and Softmax layers) is used to generate a scale probability distribution (i.e. scale mask) pixel by pixel, dynamically assigning the optimal processing scale of each pixel; finally, high-precision segmentation results are generated by weighting and fusing the branch outputs and the scale mask, which ensures that each pixel is processed by the most suitable scale branch, significantly improving the segmentation quality of small cracks and large scale difference scenes. The calculation formula of the multi-scale adaptive module is as follows:

[0104] ;

[0105] In the formula, denotes the final segmentation result, i denotes a number of discrete scales, denotes the score of scale i generated by the scale adaptive branch, denotes the scale mask, and I denotes the total number of discrete scales.

[0106] The local feature enhancement module is used to extract image local feature representation.

[0107] The local feature enhancement module includes the following steps when extracting image local feature representation:

[0108] The input features are convolved using the Stem module, and batch normalization and ReLU activation functions are performed in sequence after convolution, resulting in smooth features;

[0109] The smooth features are input into the lightweight MLP branch and the learnable visual center branch, respectively, wherein:

[0110] The lightweight MLP branch is used to capture the global long-distance dependencies of the smooth features; the learnable visual center branch is used to encode the features and map them to a predefined visual center, capturing the local region information of the smooth features;

[0111] The output features of the lightweight MLP branch and the learnable visual center branch are spliced to obtain the image local feature representation.

[0112] It should be noted that, as shown in Figure 6 EVC Block enhances local feature representation: by explicitly modeling visual centers, it enhances local feature representation in images, optimizes the fusion of global and local information, which enables the model to better focus on and process small targets such as cracks, improves the perception of local details, and further improves the segmentation accuracy of the model. The calculation process includes:

[0113] First, the input features are processed by the Stem module through 7x7 convolution, batch normalization (BN) and ReLU activation function to smooth the features. The formula of this module is:

[0114] ;

[0115] In the formula, X6 represents the input features from the last layer of the feature pyramid, σ represents the ReLU activation function, BN represents the batch normalization operation, X in represents the input features, and Conv represents the convolution operation.

[0116] Then, the features are sent to two parallel branches: one is a lightweight MLP branch that captures global long-range dependencies. The formula of this module is:

[0117] ;

[0118] ;

[0119] In the formula, DConv represents the depth convolution operation, GN represents the group normalization, CMLP represents the channel lightweight MLP branch, represents the input features enhanced by depth convolution and normalization, represents the processing result of the channel lightweight MLP branch.

[0120] The other is the learnable visual center (LVC) branch, which captures local region information by encoding features and mapping them to predefined visual centers. The formula of this module is:

[0121] ;

[0122] In the formula, represents the encoded features, b k represents the kth visual center, s k represents the scaling factor, K represents the number of visual centers, and e kwhere denotes the response corresponding to visual center k, i.e., the attention weighted vector focusing on local features, N denotes the number of input features, i.e., the number of local features, i and j denote the feature index and auxiliary index variable in the summation process, respectively.

[0123] Finally, the output features of the lightweight MLP branch and the learnable visual center branch are concatenated to form the final EVC output, which is calculated as follows:

[0124]

[0125] where denotes the concatenated feature output, cat(·) denotes the concatenation of features in the channel dimension, denotes the output feature of the lightweight MLP branch, denotes the output feature of the learnable visual center branch.

[0126] By comparing the image local feature representation with the standard reference graph and constructing a random matrix theory spectral matrix based on the comparison results, the free product relationship between the learnable visual center branch and the lightweight MLP branch is regulated through the random matrix spectral theory matrix.

[0127] It should be noted that by comparing the local features with the standard reference graph, constructing a random matrix theory spectral matrix, and regulating the free product relationship between the learnable visual center branch and the lightweight MLP branch using the random matrix spectral theory, the representation ability of the model for image local features can be effectively enhanced while maintaining computational efficiency. Specifically, random matrix spectral analysis can help optimize the correlation distribution between features, enabling the visual center branch to more focusedly learn key region features, while the lightweight MLP branch can efficiently process global information. The free product relationship between the two is regulated through spectral theory to achieve better synergy, ultimately improving the representation ability and generalization performance of the overall model, especially in handling complex visual tasks, which can obtain more robust feature representation.

[0128] where constructing a random matrix theory spectral matrix based on the comparison results and regulating the free product relationship between the learnable visual center branch and the lightweight MLP branch through the random matrix spectral theory matrix includes:

[0129] Based on the comparison results, the learnable visual center branch and the lightweight MLP branch are constructed into feature matrices, respectively, and a pre-defined empirical spectral distribution function is used to calculate the weight of each feature matrix.

[0130] It should be noted that the corresponding local and global features are extracted from the visual center branch and the MLP branch, respectively, to obtain two feature matrices.

[0131] ​​The two feature matrices are fused or spliced to form a joint feature matrix; then, a covariance matrix or Gram matrix of the feature matrix is calculated to reflect the statistical relationship between the features;

[0132] Eigenvalue decomposition is performed on the matrix to obtain its spectrum (i.e., eigenvalue distribution); a predefined empirical spectral distribution function (ESD) is used to fit or estimate the eigenvalues, and the ESD refers to the empirical distribution function of all eigenvalues in a finite-dimensional random matrix, i.e., after sorting the eigenvalues from small to large, the proportion of each eigenvalue below is counted;

[0133] The distribution function is used to assign weights to the channels or units in each feature matrix, so that the feature weights are more reasonable in the spectral structure, thereby adjusting the contribution weights of the visual center branch and the MLP branch to improve the feature representation ability after fusion.

[0134] According to the weight of each feature matrix, the distribution evolution of the spectral distribution structure of the learnable visual center branch and the lightweight MLP branch is simulated by using wavelet Brown motion.

[0135] The distribution evolution of the spectral distribution structure of the learnable visual center branch and the lightweight MLP branch is simulated by using wavelet Brown motion, including:

[0136] The wavelet basis function is defined, and the wavelet basis function covers the low-frequency global features and high-frequency local features of the learnable visual center branch and the lightweight MLP branch;

[0137] The outputs of the visual branch and the MLP branch are projected into different frequency bands through wavelet transform, the frequency band energy is calculated, and the frequency band energy is used as a disturbance intensity modulation factor;

[0138] The wavelet basis function is introduced into the disturbance intensity modulation factor, the wavelet inverse transform is performed on the disturbed wavelet basis function, the time interval of Brown motion is set, and the simulated Brown motion path is reconstructed after the wavelet inverse transform.

[0139] It should be noted that the wavelet basis function is introduced into the disturbance intensity modulation factor, the wavelet inverse transform is performed on the disturbed wavelet basis function, the time interval of Brown motion is set, and the simulated Brown motion path is reconstructed after the wavelet inverse transform.

[0140] The wavelet basis function (such as Daubechies or Symlet) is selected, the feature outputs of the learnable visual center branch and the lightweight MLP branch are wavelet decomposed, and frequency coefficients of different scales are extracted;

[0141] The energy of each frequency band is calculated as a perturbation intensity modulation factor, which is introduced into the wavelet coefficients to construct a perturbed wavelet representation; the wavelet inverse transform is performed on the perturbed wavelet coefficients to reconstruct the signal;

[0142] The time interval of Brownian motion is set, and path simulation is performed on the reconstructed signal along the time interval to simulate the random evolution process of the feature in the time domain; and spectrum analysis is performed on the simulated Brownian motion path to obtain the evolution trend of the spectral distribution structure over time;

[0143] Combining the multi-scale decomposition capability of the wavelet and the random evolution characteristics of the Brownian motion, the evolution process of the complex feature structure in the frequency and time dimensions can be dynamically simulated, thereby improving the expression capability, robustness and frequency sensitivity of the model to non-stationary image features.

[0144] Spectrum analysis is performed on the spectral distribution structure in the Brownian motion path, and the distribution evolution of the spectral distribution structure is obtained based on the spectrum analysis result.

[0145] It should be noted that the wavelet Brownian motion path simulation combined with wavelet decomposition can simultaneously model the non-stationary spectral distribution evolution of the learnable visual center branch and the lightweight MLP branch features in the frequency and time domains; through wavelet decomposition, the local and global feature frequency changes at different scales can be finely captured, and through the introduction of random perturbation using Brownian motion path, the dynamic changes in the feature evolution process can be simulated, so that the spectral distribution structure has time correlation and multi-scale robustness in the feature space, thereby enhancing the adaptability of the model to complex visual patterns and improving the fine-grained modeling effect and generalization performance of the spectral expression.

[0146] Based on the single ring theory, a distribution geometric characteristic model of the spectral distribution structure of each branch is established on the complex plane, and the spectral radius range, condition number and structural sparsity of each feature matrix are adjusted through the distribution geometric characteristic model.

[0147] It should be noted that the single ring theory (Single Ring Theorem) is an important result in random matrix theory, which is used to describe the distribution law of the eigenvalues of a non-Hermitian random matrix on the complex plane in the large-dimensional limit. The theory points out that under certain conditions, the eigenvalues of a non-Hermitian random matrix will be concentrated in a single ring region on the complex plane, and the inner and outer radii are determined by the singular value distribution of the matrix. The specific implementation steps include:

[0148] The feature matrix extracted by each branch (the visual center branch and the lightweight MLP branch) is regarded as a non-Hermite random matrix, and singular value decomposition is performed to obtain spectral features; the eigenvalue distribution is fitted on the complex plane, and a distribution geometric property model is constructed, which describes the geometric properties of the eigenvalues in the single ring structure, such as radius distribution, eccentricity and density function; the spectral distribution of the feature matrix on the single ring is adjusted (the spectral radius is controlled), the dominant direction and the aggregation (the condition number is adjusted), and the sparsity of the eigenvalue distribution (achieved by sparse regularization or low-rank approximation) by controlling the norm, sparse structure or adding a regularization term of the feature matrix, so that the model enhances the stability while maintaining the expression ability, avoids overfitting caused by too dense feature matrix, and makes the spectral distribution more consistent with the stable structure under the single ring theory.

[0149] An exponential test function is set to amplify the spectral region of each feature matrix, and the eigenvalue set of the amplified feature matrix is obtained to form a global structure measurement index;

[0150] The global structure measurement index is combined with the average spectral radius to control the proportion of the learnable visual center branch and the lightweight MLP branch to regulate the spectral distribution structure tension until the learnable visual center branch and the lightweight MLP branch satisfy the free product relationship.

[0151] Based on the free product relationship, it is predicted whether the adaptive semantic segmentation model has structural overfitting, if yes, the local feature representation of the image is optimized in structure, otherwise, the current local feature representation of the image is taken as the final output of the local feature enhancement module.

[0152] The feature perception module is used to fuse the SAM-CAFusion module with the deformable convolution to obtain the perception ability of the adaptive semantic segmentation model to irregular geometric structures.

[0153] When the feature perception module fuses the SAM-CAFusion module with the deformable convolution to obtain the perception ability of the adaptive semantic segmentation model to irregular geometric structures, it includes:

[0154] The coordinate attention module is used to generate an attention map, and the position encoding vector of the attention map is used to guide the backbone feature extraction network to focus on the crack area, so as to realize global coarse positioning of the crack area;

[0155] The interest region is extracted from the attention map, the interest region is obtained through the guided deformable convolution module, the attention offset of the interest region is calculated, and the position offset of the convolution kernel is dynamically adjusted based on the attention offset;

[0156] The scale selector is introduced in the guided deformable convolution module, and the scale selector is used to adaptively select the convolution kernel according to the local image texture complexity, so as to obtain the representation requirement of the crack region at different scales.

[0157] It should be noted that the SAM-CAFusion module fusing the coordinate attention mechanism and the deformable convolution includes:

[0158] As shown in Figures 7-8 , the module is based on the coordinate attention mechanism, guides the attention weight distribution by constructing the position encoding vector, and then realizes the global rough positioning of the crack region; subsequently, the gradient amplitude map and the edge direction map of the image are combined to construct the attention guide map, which is used as the input to guide the generation of the attention offset, so as to dynamically adjust the position offset of the convolution kernel, and effectively enhance the perception ability and feature extraction ability of the model to irregular geometric structures (such as crack edges and burrs).

[0159] The overall structure includes two stages: the first stage uses the coordinate attention module to generate an attention map, guides the backbone feature extraction network to focus on the region where the crack may exist, and realizes the rough positioning of the crack region. The second stage extracts the region of interest from the attention map, and further models these regions with high precision through the guided deformable convolution module, wherein the offset of the convolution kernel is dynamically generated by the fused image edge direction and gray gradient information, so as to more accurately depict the crack detail structure and improve the precision of the model.

[0160] To further improve the multi-scale adaptability of the model, the module introduces a dynamic scale selector, which automatically selects convolution kernels of different sizes according to the local image texture complexity (such as image gradient change rate), to adapt to the representation requirement of the crack region at different scales. At the same time, in the feature fusion stage, a cross-scale attention interaction mechanism is introduced, which realizes the information flow and weighted enhancement between feature maps by constructing a cross-scale attention matrix, so as to improve the cooperativity and recognition robustness between different scales. The CAFusion calculation process is as follows:

[0161] ;

[0162] In the formula, represents the output feature value of the coordinate in the channel C, represents the input feature value of the coordinate in the channel C, represents the weight on the x coordinate on the channel C, represents the weight on the y coordinate on the channel C.

[0163] ;

[0164] In the formula, Represents the Sigmoid operation, represents the convolution operation on the x coordinate, Represents the eigenvalue at the x-coordinate.

[0165] ;

[0166] Where, Represents the Sigmoid operation, represents the convolution operation on the y coordinate, Represents the eigenvalue on the y-coordinate.

[0167] ;

[0168] Where, represents batch normalization and nonlinear activation layers, represents channel stacking and convolution operations, , Represents the 1D average pooling results of x and y coordinates respectively.

[0169] ;

[0170] Where W represents the width of the input feature map, Represents the eigenvalue at the x-coordinate in channel C.

[0171] ;

[0172] Where H represents the height of the input feature map, Represents the eigenvalue at the y-coordinate in channel C.

[0173] Bilinear interpolation takes a weighted average of the values ​​of the four adjacent pixels corresponding to the target point in the original image, first interpolating horizontally (x) and then vertically (y) to obtain the grayscale or color value of the target pixel. This method is visually superior to nearest neighbor interpolation, effectively reducing aliasing and blurring caused by image deformation, improving the smoothness and accuracy of the transformed image, and thus enhancing the accuracy and robustness of subsequent lane line extraction. It is suitable for real-time processing scenarios. The bilinear interpolation formula is as follows:

[0174] ;

[0175] Where, represents the integer pixel coordinates of the upper left corner of the target point, (i, j) represents the integer pixel coordinates of the upper left corner of the target, a=xI, b=yj represents the relative distance between the target point and the upper left corner pixel.

[0176] The loss function module is configured to combine the dice coefficient loss and the focal loss to construct a loss function, and introduce a modulation coefficient in the loss function to balance the segmentation annotation task samples.

[0177] It should be noted that the Focal Dice Loss is introduced as the loss function: effectively balancing the sample imbalance problem in the pavement crack segmentation task, the function combines the design ideas of Dice Loss and Focal Loss, by introducing a modulation coefficient, reducing the contribution of easy classification samples to the loss, enhancing the attention to difficult classification samples, and improving the overall performance of the model, the specific calculation formula is as follows:

[0178] ;

[0179] ;

[0180] In the formula, represents the similarity of two sets A and B, A represents the region predicted by the model, and B represents the region of the real label, represents the intersection size of set A and set B, and respectively represent the size of the predicted region and the real label region.

[0181] In order to solve the problem of class imbalance, a modulation factor is introduced to reduce the contribution of easy classification samples to the loss and enhance the attention to difficult classification samples, and the calculation formula is as follows:

[0182] ;

[0183] In the formula, , both represent the modulation coefficient, P t represents the probability prediction size.

[0184] Focal Dice Loss is a combination of Dice Loss and Focal Loss design ideas, aiming to solve the problems of class imbalance and difficult classification samples. Focal Dice Loss introduces a modulation factor similar to Focal Loss on the basis of traditional Dice Loss, thereby adjusting the loss function so that the model can pay more attention to small objects or rare targets such as pavement cracks that are difficult to segment. The calculation formula is as follows:

[0185] ;

[0186] In the formula, A represents the region predicted by the model, and B represents the region of the real label, , both represent the modulation coefficient.

[0187] It should be noted that the present application adopts multiple indexes including Mean Intersection over Union (mIoU), Average Precision (mPA), Precision (mPrecision), Recall (mRecall) and F1-Score for verifying the performance of the model.

[0188] Mean Intersection over Union (mIoU): mIoU is the average value of the IoU values of all classes, where IoU refers to the ratio of the intersection of the predicted positive class area and the actual positive class area to their union, and the formula is as follows:

[0189] ;

[0190] In the formula, represents the number of pixels of True Positives of the i-th class, represents the number of pixels of False Positives of the i-th class, represents the number of pixels of False Negatives of the i-th class, and N represents the total number of classes.

[0191] Average Precision (mPA): used to measure the average proportion of correctly classified pixels in the entire dataset, and the formula is as follows:

[0192] ;

[0193] In the formula, represents the number of pixels of True Positives of the i-th class, represents the number of pixels of True Negatives of the i-th class, represents the number of pixels of False Positives of the i-th class, represents the number of pixels of False Negatives of the i-th class, and N represents the total number of classes.

[0194] Precision (mPrecision): the average value of the proportion of correct classification in all samples classified as positive classes, and the formula is as follows:

[0195] ;

[0196] In the formula, the number of pixels of True Positives of the i-th class, the number of pixels of False Positives of the i-th class, and N represents the total number of classes.

[0197] Recall (mRecall): the average value of the proportion of correctly classified samples in all actual positive classes, the formula is as follows:

[0198] ;

[0199] In the formula, the number of pixels of True Positives of the i-th class, the number of pixels of False Negatives of the i-th class, and N represents the total number of classes.

[0200] F1 score comprehensively considers mPrecision and mRecall, and is the harmonic mean of mPrecision and mRecall, the formula is as follows:

[0201] ;

[0202] In the formula, mPrecision represents accuracy, and mRecall represents recall.

[0203] The feature critical point detection module is configured to construct a sequence parameter monitoring model between the global feature capture module and the local feature enhancement module, and balance the global feature and the local feature through the sequence parameter monitoring model.

[0204] The feature critical point detection module is configured to construct a sequence parameter monitoring model between the global feature capture module and the local feature enhancement module, and balance the global feature and the local feature through the sequence parameter monitoring model, including:

[0205] Constructing a sequence parameter in the overlapping area of the global feature and the local feature, defining a critical threshold, and comparing the sequence parameter with the critical threshold;

[0206] If the sequence parameter is greater than the critical threshold, it indicates that the adaptive semantic segmentation model is in a global dominant mode, then the shallow layer parameters of the global feature capture module are frozen, and a heterogeneous convolution kernel is introduced in the local feature enhancement module.

[0207] If the sequence parameter is less than or equal to the critical threshold, it indicates that the adaptive semantic segmentation model is in a local dominant mode, then a non-local attention compensation mechanism is inserted in the global feature capture module.

[0208] It should be noted that the feature critical point detection module realizes dynamic monitoring and adjustment of the current feature attention center of the model by constructing a sequence parameter in the global and local feature overlapping area and setting a critical threshold. When the model is in a global dominant mode, the shallow global parameters are frozen and anisotropic convolution is introduced to enhance local structure perception, avoiding redundant global information interfering with local detail expression. In the local dominant mode, a non-local attention mechanism is introduced to compensate for the loss of global information, thereby realizing dynamic weighting and complementarity between global and local features, and improving the adaptability, feature expression accuracy and semantic boundary recognition performance of the semantic segmentation model under multi-scale structures.

[0209] The global dominant mode feedback and the local dominant mode feedback are extracted to construct a bidirectional closed-loop control mechanism, and the bidirectional closed-loop control mechanism comprises:

[0210] The sequence parameter gating is integrated into the FastViT visual network; and the critical threshold is corrected through gradient back propagation of the local feature enhancement module.

[0211] It should be noted that the sequence parameter gating mechanism is introduced to dynamically perceive and adjust the global dominant mode feedback and the local dominant mode feedback. The sequence parameter gating is used to monitor the semantic distribution change of the global and local feature interaction area and control the information flow path. Specifically, a gating unit based on sequence parameters is inserted at the key layer, and the gating weight is used to determine whether to activate or suppress a specific feature channel. At the same time, the gradient information in the local feature enhancement module is used to dynamically adjust the critical threshold during back propagation. The gradient size reflects the contribution of local features to the final prediction. When the influence of local features on loss increases, the critical threshold is automatically reduced to enhance the role of global compensation mechanism. Conversely, the threshold is increased to enhance the capture of local fine structures, thereby constructing a bidirectional closed-loop control mechanism to realize dynamic feedback adjustment between global and local features, and enhancing the feature balancing ability and semantic adaptability of FastViT in complex scenes.

[0212] S3, deploying the adaptive semantic segmentation model after training to a local platform, and inputting a crack disease image into the adaptive semantic segmentation model for segmentation labeling.

[0213] It should be noted that the road crack pictures in the test set are input into the deployed network for automatic segmentation and labeling of road cracks.

[0214] The method for semantic segmentation of road cracks based on the Transformer and CNN architecture provided by the present application will be further described in combination with specific implementation manners:

[0215] The pavement image is obtained from the Pavementscapes dataset, and the image containing only crack disease is re-labeled using the Labelme tool; then, the image is pre-processed and data is enhanced; then, the dataset is divided into a training set, a validation set and a test set according to a certain proportion; in the model construction and training process, FastViT is used as the backbone network, a multi-scale adaptive module (SAM) is used to dynamically allocate the optimal processing scale of the pixel level, an EVC Block is used to enhance the local detail perception ability, and a SAM-CAFusion module is used to fuse the coordinate attention mechanism and the deformable convolution to accurately model the crack edge geometry; at the same time, the Focal Dice Loss loss function is introduced to alleviate the sample imbalance problem; finally, the trained model is deployed to the local platform, the model performance is verified through the test set, and the automatic segmentation and labeling of the pavement crack are carried out.

[0216] The application can effectively balance the detection accuracy and the calculation efficiency, is suitable for pavement crack detection in a complex environment, has strong robustness and engineering application value, and specifically:

[0217] Step one, construction of the dataset:

[0218] The dataset comes from Pavementscapes, and is an expressway pavement disease image obtained by a pavement disease detection vehicle. In the dataset, the original image size is 2048x1024, in order to adapt to the pavement crack semantic segmentation task, the dataset is cut to 512x512. Considering the labeling accuracy, a total of 2699 images containing only crack diseases are re-labeled using the Labelme tool, so as to obtain a JSON file containing pavement crack information, so as to construct the dataset.

[0219] Step two, data preprocessing:

[0220] The application adopts the VOC dataset format for training, in the label image, the background pixel is 0, and the crack target pixel is 1, all images are uniformly adapted to 512x512 in size before being input into the model, and data enhancement is performed by means of flipping, symmetry, scaling, long and wide torsion and random adjustment of brightness, contrast and the like.

[0221] Step three, division of the dataset:

[0222] The 2699 labeled images are divided into a training and validation set and a test set according to a ratio of 4:1 (2159 training and validation set images, 540 test set images), and the training set is divided into a training set and a validation set according to a ratio of 4:1 (1727 training set images, 432 validation set images).

[0223] Step four, model training:

[0224] The present application relies on the deep network learning framework PyTorch2.0.0 to develop the model construction and deployment, and runs on a workstation equipped with Intel i9-12900K (16-core) CPU and NVIDIA GeForce RTX 3070 Ti (8GB video memory) GPU, the CUDA version is 12.1, and the system memory is 64GB RAM. The training parameters are shown in Table 1, the maximum learning rate is set to 0.0001, the Batch-size is 4, and a total of 100 rounds of transfer learning training is performed.

[0225] Step five, model verification:

[0226] The trained model is verified, and multiple indicators including mean intersection over union (mIoU), average precision (mPA), accuracy (mPrecision), recall (mRecall), and F1 score are used to verify the performance of the model.

[0227] Step six, road surface crack segmentation:

[0228] The road surface crack pictures in the test set are input into the deployed network for automatic segmentation and labeling of road surface cracks.

[0229] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A road crack semantic segmentation method based on Transformer and CNN architecture, characterized by: The method includes: Obtain a pavement image, use an image annotation tool to annotate crack disease images in the pavement image, and construct a dataset based on the annotated crack disease images; Preprocess the dataset, and build and train an adaptive semantic segmentation model based on the preprocessed dataset, combining a self-attention neural network with a convolutional neural network. The trained adaptive semantic segmentation model is deployed to the local platform, and the crack disease image is input into the adaptive semantic segmentation model for segmentation and annotation.

2. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 1 is characterized in that: The adaptive semantic segmentation model combining the self-attention neural network and the convolutional neural network include: The global feature capture module is used to use the FastViT visual network as the backbone network and capture the global features of the image through the FastViT visual network; Multi-scale adaptive module for dynamically assigning the optimal processing scale based on the scale mask; Local feature enhancement module, used to extract local feature representation of the image; The feature perception module is used to fuse the SAM-CAFusion module with the deformable convolution to obtain the adaptive semantic segmentation model's ability to perceive irregular geometric structures; The loss function module is used to construct a loss function by combining the dice coefficient loss and the focus loss, and introduces a modulation coefficient into the loss function to balance the segmentation and labeling task samples; The feature critical point detection module is used to construct an order parameter monitoring model between the global feature capture module and the local feature enhancement module, and balance the global features and local features through the order parameter monitoring model.

3. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 2 is characterized in that: The FastViT visual network is a four-stage network architecture. The network architecture of the first three stages is the same, and the FastViT visual network processes global features through deep convolutional layers and batch normalization layers in the first three stages. The FastViT visual network uses the self-attention mechanism to perform feature fusion in the fourth stage.

4. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 2 is characterized in that: The multi-scale adaptive module includes: A predefined shared feature extraction network is used to generate a unified feature map, which is then fed into several parallel-scale adaptive branches. The adaptive branches then extract features based on a preset scale range. A predefined lightweight scale estimator is used to generate a scale probability distribution pixel by pixel, and the optimal processing scale mask of each pixel is dynamically assigned according to the scale probability distribution; The feature extraction results output by each adaptive branch are weightedly fused with the optimal processing scale mask of each pixel to generate the final image segmentation result.

5. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 4 is characterized in that: The local feature enhancement module includes: The Stem module is used to perform convolution processing on the input features, and after the convolution processing is completed, batch normalization and ReLU activation function processing are performed in sequence to obtain smooth features; The smooth features are input into the lightweight MLP branch and the learnable visual center branch respectively, where: The lightweight MLP branch is used to capture the global long-distance dependency of smooth features; the learnable visual center branch is used to encode features and map them to predefined visual centers to capture local area information of smooth features; The output features of the lightweight MLP branch and the learnable visual center branch are concatenated to obtain the local feature representation of the image; The local feature representation of the image is compared with the standard reference image, and a random matrix theory spectrum matrix is ​​constructed based on the comparison results. The random matrix spectrum matrix can be used to regulate the free product relationship between the learnable visual center branch and the lightweight MLP branch; Based on the free product relationship, it is predicted whether the adaptive semantic segmentation model has structural overfitting. If so, the local feature representation of the image is structurally optimized. Otherwise, the current local feature representation of the image is used as the final output of the local feature enhancement module.

6. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 5 is characterized in that: The calculation formula of the lightweight MLP branch is: ； ； Where DConv represents the depth convolution operation, GN represents group normalization, CMLP represents the channel lightweight MLP branch, represents the input features enhanced by depthwise convolution and normalization, represents the channel lightweight MLP branch processing result, X in Indicates that the input features are sub-branch feature streams that are processed independently in the module; The calculation formula of the learnable visual center branch is: ； Where, represents the encoded features, b k represents the kth visual center, s k represents the scaling factor, K represents the number of visual centers, and e k represents the response corresponding to the visual center k, N represents the number of input features, i and j represent the feature index and auxiliary index variables in the summation process, respectively.

7. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 5 is characterized in that: The random matrix theory spectrum matrix is ​​constructed based on the comparison results, and the free product relationship between the learnable visual center branch and the lightweight MLP branch is regulated by the random matrix theory spectrum matrix. Based on the comparison results, the learnable visual center branch and the lightweight MLP branch are respectively used to construct feature matrices, and the predefined empirical spectral distribution function is used to calculate the weight of each feature matrix; According to the weight of each feature matrix, wavelet Brownian motion is used to simulate the distribution evolution of the spectral distribution structure of the learnable visual center branch and the lightweight MLP branch; Based on the single ring theory, a distribution geometry model is established on the complex plane for the spectral distribution structure of each branch, and the spectral radius range, condition number and structural sparsity of each feature matrix are adjusted through the distribution geometry model. An exponential test function is set to amplify the spectral region of each characteristic matrix, and the eigenvalue set of the amplified characteristic matrix is ​​obtained to form a global structure measurement index; The ratio of the learnable visual center branch to the lightweight MLP branch is controlled by combining the global structure metric with the average spectral radius to regulate the structural tension of the spectral distribution until it is adjusted to satisfy the free product relationship between the learnable visual center branch and the lightweight MLP branch.

8. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 7 is characterized in that: The distribution evolution of the spectral distribution structure of the learnable visual center branch and the lightweight MLP branch can be simulated by using wavelet Brownian motion, including: Define wavelet basis functions, and the wavelet basis functions cover the low-frequency global features and high-frequency local features of the learnable visual center branch and the lightweight MLP branch; The outputs of the visual branch and the MLP branch are projected to different frequency bands through wavelet transform, the frequency band energy is calculated, and the frequency band energy is used as the perturbation intensity modulation factor; The wavelet basis function is introduced into the disturbance intensity modulation factor, and the wavelet inverse transform is performed on the disturbed wavelet basis function. At the same time, the time interval of Brownian motion is set, and the simulated Brownian motion path is reconstructed after the wavelet inverse transform. The spectral distribution structure is spectrally analyzed in the Brownian motion path, and the distribution evolution of the spectral distribution structure is obtained based on the spectral analysis results.

9. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 2 is characterized in that: The feature perception module, when fusing the SAM-CAFusion module with the deformable convolution to obtain the adaptive semantic segmentation model's perception capability of irregular geometric structures, includes: The coordinate attention module is used to generate an attention map. The position encoding vector of the attention map guides the backbone feature extraction network to focus on the crack area, thus achieving a global coarse positioning of the crack area. Extract the region of interest from the attention map, obtain the region of interest through the guided deformable convolution module, calculate the attention offset of the region of interest, and dynamically adjust the position offset of the convolution kernel based on the attention offset; A scale selector is introduced into the guided deformable convolution module. The scale selector is used to adaptively select the convolution kernel according to the local image texture complexity to obtain the representation requirements of the crack area at different scales.

10. The pavement crack semantic segmentation method based on Transformer and CNN architecture according to claim 9, characterized in that: The feature critical point detection module is used to construct an order parameter monitoring model between the global feature capture module and the local feature enhancement module. Balancing the global features and local features through the order parameter monitoring model includes: Constructing an order parameter in the overlapping region of global features and local features, defining a critical threshold, and comparing the order parameter with the critical threshold; If the order parameter is greater than the critical threshold, it means that the adaptive semantic segmentation model is in the global dominant mode, then the shallow parameters of the global feature capture module are frozen, and heterogeneous convolution kernels are introduced into the local feature enhancement module; If the order parameter is less than or equal to the critical threshold, it means that the adaptive semantic segmentation model is in the local dominant mode, and a non-local attention compensation mechanism is inserted into the global feature capture module; Extract global dominant mode feedback and local dominant mode feedback to construct a bidirectional closed-loop control mechanism, and the bidirectional closed-loop control mechanism includes: Incorporate order parameter gating into the FastViT visual network; correct the critical threshold through gradient backpropagation of the local feature enhancement module.

Images