Multi-scale feature pavement crack detection method based on double-branch attention mechanism

Abstract

The invention relates to the technical field of road detection, in particular to a multi-scale feature pavement crack detection method based on a double-branch attention mechanism. The method comprises the following steps: S1, obtaining a to-be-detected pavement crack image; s2, inputting the image obtained in the step S1 into a crack detection network DBA-UNet; s3, an SVD-IASPP module is adopted in a bottleneck layer of the crack detection network DBA-UNet; s4, channel-space double-branch attention weighting is carried out on the same-scale low-level features by using DBAM at an encoder-decoder jump connection part; and S5, the decoder performs up-sampling step by step and performs fusion with the weighted low-level features obtained in the step S4, and a crack segmentation image with the same resolution as the input image is output. The invention provides a multi-scale feature pavement crack detection method based on a double-branch attention mechanism. The method comprises the following steps: providing an inverse residual multi-scale feature extraction module-SVD-IASPP; by using a plurality of 3 * 3 convolution kernels with different void values, features of different scales of cracks are effectively captured; and in combination with the idea of an inverse residual structure, outputting a final feature map.

Description

Technical Field

[0001] This invention relates to the field of road inspection technology, and in particular to a multi-scale feature pavement crack detection method based on a bi-branch attention mechanism. Background Technology

[0002] With the progress of the times, transportation has become increasingly important to people's livelihoods. Highway transportation plays a crucial role, not only promoting economic development and improving people's lives, but also connecting different regions and eliminating barriers to communication. Although my country's highway transportation has made significant progress in recent years, this has also brought enormous challenges to road maintenance. The increasing age of highways, the surge in the number of vehicles, improper driving practices, and the impact of natural disasters all seriously threaten the condition of highway surfaces. Therefore, for highway maintenance units, timely monitoring of the degree of road wear and aging is particularly important.

[0003] Cracks are a common "disease" of road surfaces, not only compromising the safety and comfort of transportation but also significantly reducing the lifespan of highways and causing substantial losses to the collective interests of society if cracks are not detected and detected in a timely and accurate manner. Therefore, the top priority for traffic management departments is to detect road surface cracks promptly and accurately, ensuring that dispatching personnel is not ineffective and reducing unnecessary losses. In the early stages, road surface crack detection typically relied on visual inspection and manual assessment, which was costly, inefficient, risky, and susceptible to subjective errors, leading to missed or false detections. With technological advancements, automated inspection vehicles have emerged, saving significant labor costs. However, the equipment itself is expensive, and even with significant improvements in crack detection efficiency, they struggle to operate in complex environments, hindering large-scale deployment.

[0004] Meanwhile, researchers have begun to combine machine learning techniques with crack detection, resulting in numerous solutions, including traditional segmentation algorithms, Hough transform, ant colony optimization, wavelet transform, morphological algorithms, and multi-scale methods. While these methods offer benefits such as reduced costs, increased efficiency, improved accuracy, and greater convenience for pavement crack detection, they unfortunately fail to deliver ideal results in complex environments.

[0005] To address this issue, a multi-scale feature pavement crack detection method based on a bi-branch attention mechanism is designed to provide a technical solution for the aforementioned technical problems. Summary of the Invention

[0006] Therefore, it is necessary to provide a multi-scale feature pavement crack detection method based on a bi-branch attention mechanism to address the aforementioned technical problems.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A multi-scale feature pavement crack detection method based on a bi-branch attention mechanism, characterized by the following steps:

[0009] S1: Obtain the image of the road surface cracks to be detected;

[0010] S2: Input the image obtained in step S1 into the crack detection network DBA-UNet;

[0011] S3: The SVD-IASPP module is used in the bottleneck layer of the crack detection network DBA-UNet;

[0012] S4: At the encoder-decoder jump connection, DBAM is used to perform channel-space dual-branch attention weighting on low-level features of the same scale;

[0013] S5: The decoder upsamples step by step and fuses it with the weighted low-level features obtained in step S4, outputting a crack segmentation map with the same resolution as the input image.

[0014] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, in step S1, the crack detection network DBA-UNet is based on UNet, and the dual-branch attention module DBAM is embedded in the skip connection and bottleneck layer.

[0015] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, the DBAM module includes:

[0016] The channel branch CB obtains global statistics through global average pooling and global max pooling, generates channel weight vectors through a fully connected layer, and multiplies them channel by channel with the original feature map;

[0017] The spatial branch SB obtains spatial statistics through average pooling and max pooling in the channel dimension, generates a spatial weight map through 1×1 convolution, and multiplies it pixel by pixel with the original feature map.

[0018] The outputs of CB and SB are added or concatenated element-wise to obtain the final attention-enhanced feature map.

[0019] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, in step S3, the SVD-IASPP module extracts multi-scale crack features in parallel with multiple convolutions of different void ratios through inverted residual structure and depth separable convolution, and performs equalization stitching on the feature maps of each scale.

[0020] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, the SVD-IASPP module comprises the following steps:

[0021] Feature extraction was performed sequentially using InvertedDenseConv with porosities of 3, 6, 12, 18, and 24.

[0022] The output of each InvertedDenseConv stage is concatenated with the input feature map before being fed into the next stage.

[0023] All scale outputs are dimensionality reduced by 1×1 convolution and the original number of channels is restored to form a balanced multi-scale feature representation.

[0024] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, the construction steps of InvertedDenseConv are as follows:

[0025] The number of channels in a 1×1 convolution is 1.5 times that of the input;

[0026] 3×3 depth-separable dilated convolution, where the number of groups equals the number of input channels;

[0027] 1×1 convolution reduces the dimensionality back to the original number of channels.

[0028] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, the training steps of the crack detection network DBA-UNet in step S3 are as follows:

[0029] Data augmentation was performed using CrackForest, Crack500, or DeepCrack datasets, with rotation, scaling, random cropping, and pixel filtering.

[0030] Use a joint loss function;

[0031] The Stochastic Gradient Descent (SGD) optimizer was used with an initial learning rate of 0.001, a decaying step size, a batch size of 8, and a training duration of 100 epochs.

[0032] The optimal weights are saved on the validation set using MioU as the metric.

[0033] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, it further includes the following steps:

[0034] The ResNet50 and Swin-Transformer are used in parallel to build the TCencoder to extract local-global complementary features;

[0035] At the end of each stage of the TCencoder, a TCCF module is introduced. The TCCF module injects the global features of the Swin-Transformer branch into the local features of the ResNet branch through Cross-Attention.

[0036] TCA-UNet is formed by embedding TCA-UNet into UNet, and SVD-IASPP and DBAM are retained at the bottleneck layer and skip connections.

[0037] During the training phase, OHEMCross-EntropyLoss and Canny edge enhancement loss are introduced.

[0038] As a preferred embodiment of the multi-scale feature pavement crack detection method based on the dual-branch attention mechanism provided by the present invention, the TCCF module includes:

[0039] Global average pooling is performed on the ResNet branch features to generate key vectors;

[0040] The Swin-Transformer branch features are convolved with 1×1 to generate the Query matrix;

[0041] Using the original ResNet features as Values, calculate the Cross-Attention weights of Query and Key and weight the Values ​​to obtain the fused feature map.

[0042] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0043] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0044] 1. The present invention provides a multi-scale feature road surface crack detection method based on a dual-branch attention mechanism, proposing an inverse residual multi-scale feature extraction module—SVD-IASPP; by using 3×3 convolution kernels with various dilation values, it effectively captures features of cracks at different scales; combining the idea of ​​inverse residual structure, it first improves the dimensionality of the multi-scale feature map through 1×1 convolution, then promotes the expression of deep-level features of cracks through channel-wise dilation convolution, and finally uses 1×1 convolution to reduce the dimensionality to output the final feature map.

[0045] 2. This invention proposes a hybrid attention mechanism module—DBAM. This module consists of two branches: a Channel Branch and a Spatial Branch. In the Channel Branch, pooling operations are used to capture each weight value representing the current feature map, and then multiplied with the current feature to strengthen features highly relevant to the current task and weaken features with low relevance. In the Spatial Branch, pooling operations are performed on the channel dimension to obtain the average and maximum values ​​in the channel dimension, and these two values ​​are concatenated and fused to extract the spatial dimension weight map of the feature map. This weight map is then multiplied with the current feature to strengthen the feature contours and details of the foreground target and weaken noise.

[0046] 3. This invention uses Swin-Transformer and ResNet50 as backbones, and fuses the feature maps output by each stage before proceeding to the next stage. This fully extracts the contextual semantic information and local detail features of the crack, thus optimizing the texture information of the crack while ensuring the integrity of the crack segmentation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the DBA-UNet network structure of the present invention;

[0049] Figure 2 This is a schematic diagram of the SVD-IASPP module of the present invention;

[0050] Figure 3 This is a schematic diagram of the Inverted DenseConv module of the present invention;

[0051] Figure 4 This is a schematic diagram of the DBAM module of the present invention;

[0052] Figure 5 A schematic diagram illustrating the expanded CrackForest dataset and its corresponding labels of this invention.

[0053] Figure 6 A schematic diagram illustrating the expanded Crack500 dataset and its corresponding labels of this invention.

[0054] Figure 7 This is a schematic diagram illustrating the comparative experimental results of this invention on the CrackForest dataset;

[0055] Figure 8 This is a schematic diagram of the comparative experimental results of this invention on the Crack500 dataset;

[0056] Figure 9 This is a schematic diagram of the Swin-Transformer network architecture of the present invention;

[0057] Figure 10 This is a schematic diagram of the sliding window mechanism of the present invention;

[0058] Figure 11 This is a schematic diagram of the TSAUNet network structure of the present invention;

[0059] Figure 12 This is a schematic diagram of the TCCF module of the present invention;

[0060] Figure 13 This is a schematic diagram illustrating the expanded Deepcrack dataset of this invention and examples of its labels.

[0061] Figure 14 This is a schematic diagram of the crack label and the corresponding edge feature map calculated by the Canny operator of the present invention.

[0062] Figure 15 This is a front view of the Jetson NX of this invention;

[0063] Figure 16 This is a reverse view of the Jetson NX of this invention;

[0064] Figure 17 This is a hardware architecture diagram of the system of the present invention;

[0065] Figure 18 This is a diagram of the Jetson NX embedded device of the present invention;

[0066] Figure 19 This invention relates to the Sony IMX219 camera and its parameters.

[0067] Figure 20 This is a physical connection diagram of the system of the present invention;

[0068] Figure 21 This is a diagram of the overall system software architecture of the present invention;

[0069] Figure 22 This is a flowchart of the pavement crack segmentation system of the present invention;

[0070] Figure 23 This is an example diagram of the Tensor RT convolutional neural network optimization of the present invention;

[0071] Figure 24 This is a physical connection diagram of the present invention;

[0072] Figure 25 This is a physical test image of the model of the present invention;

[0073] Figure 26 This is a flowchart illustrating the web-based access and interaction process of this invention.

[0074] Figure 27 This is the overall architecture design diagram of the Web-based system of this invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0076] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0077] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0078] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0079] Example 1

[0080] Reference Figures 1-14 A multi-scale feature road surface crack detection method based on a bi-branch attention mechanism.

[0081] 1. Construct a multi-scale feature pavement crack detection network based on a dual-branch attention mechanism

[0082] 1.1 Overall Model Architecture

[0083] Based on UNet, improvements were made to its skip connection parts. A dual-branch attention mechanism was added to each skip connection and the bottle part of the UNet network to simultaneously calculate the channel correlation and spatial correlation of the feature maps, enhancing crack feature information in the feature maps while downplaying background and redundant information. This module with the dual-branch attention mechanism was named DBAM (Dual Branch Attention Module). Simultaneously, by adjusting the proportion of feature maps with large receptive fields in the traditional Dense-ASPP module, and using the ideas of depthwise separable convolution and inverse residual structures, the computational cost of the traditional ASPP (Atrous Spatial Pyramid Pooling) module was reduced while further enhancing the extraction of multi-scale feature information of cracks. This ASPP module was named SVD-IASPP (Same Value Dense Inverted ASPP). The network was named DBA-UNet (Dual Branch Attention UNet), and its overall architecture is as follows. Figure 1 As shown.

[0084] Network, with crack image X∈R H×W×C The input is used as input. In the encoder part, the scale of the feature map is gradually reduced through multiple cascaded max pooling and double convolution operations, continuously expanding the overall receptive field of the feature map. At the same time, the number of channels is also continuously increased to obtain feature maps with more information. After the above operations, the encoder part obtains high-level features.

[0085] Subsequently, in the bottle part of the network, the high-level feature X... r The input is fed into the mean SVD-IASPP module, where spatial pyramid pooling is performed. Convolutional operations with different dilation values ​​are used to capture foreground feature information at different scales, and a dual-branch attention mechanism (DBAM) is used to enhance the features of the foreground target.

[0086] In the decoder section, four successive upsampling operations are performed, each using bilinear interpolation and double convolution, to gradually increase spatial resolution and decrease the number of channels, thus gradually restoring the crack information contained in the high-level features. Simultaneously, after each upsampling operation in the decoder, DBAM is used to enhance the low-level features of the encoder layer at the same level, and then the enhanced low-level features are fused with the upsampled high-level features through skip connections. This process preserves low-level features that are closer to the original image while extracting more information highly correlated with cracks. Combined with the crack features in the decoder, which have more scale information, the upsampling operation is completed. Finally, a semantic segmentation map with the same pixel-level resolution as the original image is output.

[0087] 1.2SVD-IASPP

[0088] The neural network module used for semantic segmentation is the ASPP module. It controls the receptive field size of convolutional operations by varying the dilation rate, thereby capturing semantic information at different levels of the image. DenseASPP is an extension of ASPP, further enhancing the expressive power of the ASPP module by introducing dense connections. In DenseASPP, the outputs of each ASPP convolutional layer are connected to the outputs of all other layers. This dense connection design allows DenseASPP to better transfer and utilize feature information, thus improving semantic segmentation performance. However, both modules require a large number of convolutional operations and feature connections, resulting in high computational costs. Furthermore, because the feature information with a large receptive field accounts for a relatively small proportion in DenseASPP, while the original feature map accounts for a larger proportion, there is less multi-scale feature information in the features, making it impossible to extract complete crack information.

[0089] The SVD-IASPP method balances the proportion of feature maps at different scales in ASPP and uses the idea of ​​inverse residual structure to reconstruct the Dense Conv Block in DenseASPP. This not only captures features at different scales but also enhances the multi-scale information features of cracks, greatly reducing the amount of computation.

[0090] First, feature map The Inverted DenseConv module with a hole rate of 3 captures contextual information with a hole rate of three times the normal rate. This contextual information is then compared with the input features. Figure X r Maintain the same scale and dimensions. Then compare the output with the input features. Figure X r The concatenation is performed, and an Inverted Dense Conv convolution with a dilation rate of 6 is used to capture contextual information at a dilation rate of six times. Subsequent operations are the same as before, using Inverted Dense Conv convolution operations with dilation rates of 12, 18, and 24 times respectively to output X. r Features of the same dimension are used, and all the outputs are concatenated together, then a 1×1 convolution is used to reduce the dimensionality before outputting the final result. The SVD-IASPP module architecture is as follows: Figure 2 As shown.

[0091] Finally, the output X of the above process d After a 1×1 convolution, the number of channels is restored to the same level as the input X. r Same, that is, the output is X dIt contains feature information of different scales, which makes the crack information decoded by the decoder more complete.

[0092] 1.3 Inverted Dense Conv

[0093] In traditional residual module design, a 1×1 convolutional kernel is often used first to reduce the dimensionality of the feature map, aiming to reduce the computational complexity of subsequent operations. Then, a 3×3 convolutional layer is used to extract more feature information. Finally, a 1×1 convolutional kernel is used again to increase the dimensionality, enriching the information content of the feature map. This structure, characterized by wide ends and a narrow middle, resembles an hourglass and is a typical form in neural network design.

[0094] However, in the inverted residual structure (also known as the reversed residual block), the first step is to use 1×1 convolution operations to broaden the channel dimension of the input feature map, thereby enriching its feature representation. Then, depthwise separable convolution techniques are used to effectively extract features. In this process, thanks to the initial dimensionality expansion, the depthwise convolution can operate independently for each channel, reducing computational complexity while fully capturing information. Subsequently, pointwise convolutions are performed on each channel using 1×1 kernels, and the output channels of the depthwise convolutions are summed with the pointwise kernels. This step is essentially equivalent to performing a fully connected operation at each location, achieving cross-channel information integration. Finally, the inverted residual structure completes the compact representation of features using 1×1 convolutions.

[0095] Compared to traditional residual structures, inverse residual structures revolutionize the order of dimensionality, innovatively replacing the standard 3×3 convolution with depthwise separable convolutions. The key feature of this architecture is its streamlined dimensions at both ends, while the dimensions in the middle are significantly expanded, resembling a shuttle-shaped structure.

[0096] Based on the idea of ​​inverted residual structure, this paper combines depthwise separable convolution and 1×1 convolution in the inverted residual structure to reduce the overall computational cost of DenseASPP while maintaining the same level of feature map information extraction. Based on the above, an Inverted DenseConv structure is proposed, the specific structure of which is as follows: Figure 3 As shown.

[0097] Assuming the feature map input to the Inverted DenseASPP module is of size h×w×c, the feature map first undergoes a 1×1 convolution to expand its dimensions. In the network structure, the expansion rate is set to 1.5, thus expanding the number of channels of the feature map to 1.5 times the original number of channels. Subsequently, a 3×3 dilated convolution is used, with the number of groups set to 1.5c, achieving depthwise dilation convolution. Finally, a 1×1 convolution is used directly, achieving both the effect of pointwise convolution extracting channel features (as in depthwise separable convolution) and the dimensionality reduction operation (as in inverse residual structures).

[0098] 1.4DBAM

[0099] DBAM is applied after the SVD-IASPP module and at the skip connections, enhancing the input feature map in both the channel and spatial dimensions. Its specific architecture is as follows: Figure 3 As shown.

[0100] First, in the Channel Branch (CB), the input feature map is transformed into a global average and a global maximum through global average pooling and global max pooling. This captures global information of the image from a global perspective, which helps the model better understand the context of the overall image. Then, the global information is compressed through a fully connected layer (FC), and the global average and maximum are added together to obtain a comprehensive global information feature. Finally, the sigmoid activation function is used to generate a weight vector, and the original feature map is multiplied by the weight vector to weight the input features, strengthen the foreground target-related features, and realize the global channel attention mechanism. The specific calculation details of this process are shown in Equation (1):

[0101] X CB =X⊙Sigmoid(fc1(avgpooling(X))+fc2(maxpooling(X))) (1)

[0102] Here, avg pooling and max pooling represent average pooling and max pooling over the spatial dimension, fc1 and fc2 represent fully connected layers that compress two different types of global information, the addition operation is the numerical addition of the two features, X is the original input in the CB branch, and ⊙ represents the dot product operation. CB The output of the CB branch incorporates features that integrate global information.

[0103] In the Spatial Branch, global average pooling (avgpool) and global max pooling (maxpool) are used to statistically analyze the features along the channel dimension, obtaining the average and maximum values ​​along the channel dimension. These two values ​​are then concatenated to preserve the different features of different channels in the feature map. After processing by the sigmoid activation function, the original input feature map is multiplied by the weight vector obtained through the sigmoid activation function. This branch makes the network pay more attention to the contribution of different information along the channel dimension to a specific location, thereby enhancing the relevant features of the foreground target. The specific calculation details of this process are shown in Equation (2):

[0104] X SB =X⊙Sigmoid(Conv(Cat(avgpooling(X) ， maxpooling(X)))) (2)

[0105] Where avg pooling and max pooling represent average pooling and max pooling across the channel dimension, Cat represents the concatenation of feature maps, Conv represents 1×1 convolutional dimensionality reduction of the concatenated feature maps, X represents the original input in the SB branch, and ⊙ represents the dot product operation. SB To focus on the characteristics of different channel dimensions at a specific location, i.e., the output of the SB branch.

[0106] 1.5 Experiment and Results Analysis

[0107] 1.5.1 Dataset

[0108] Experiments were conducted using two pavement crack datasets: the CrackForest dataset and the Crack500 dataset, both containing pavement cracks of varying complexity. The CrackForest dataset was collected from concrete roads, while the Crack500 dataset was collected from asphalt roads.

[0109] The CrackForest dataset, created by Cui et al., is an annotated database of road crack images that provides a general overview of urban road surface conditions. It contains 119 images of road cracks at a resolution of 224×224, each an RGB image with two categories of labels: crack and background. Ten images out of the 119 contained blocky defects labeled as cracks; these ten images and their labels were therefore discarded from the dataset.

[0110] The Crack500 dataset, created by Zhang et al., is specifically designed for crack detection in asphalt roads. It contains 494 images of asphalt roads at a resolution of 2560×1440 and corresponding crack annotations. After the original authors enhanced the images through cropping and other methods, they obtained 3368 images at a resolution of 640×360. Of these, 1896 images were assigned to the training set, 1124 to the test set, and 348 to the validation set.

[0111] The DBA-UNet network was used to train and test the CrackForest and Crack500 datasets separately. Since the initial image sizes of the two datasets differed, data augmentation was used to expand both datasets, unifying the image resolution to 512×512. The specific data augmentation method was as follows: First, the original image was enlarged by a factor of k (CrackForest dataset: k = 4; Crack500 dataset: k = 1.2). Then, the image was rotated clockwise, and every 15° rotation, n images (CrackForest dataset: n = 15; Crack500 dataset: n = 10) with a resolution of 512×512 were randomly cropped from the image. After obtaining the dataset images and labels using the above method, the data was filtered: labels with a crack pixel percentage (pixel value = 255) below 0.5% and their corresponding images were deleted. Finally, the CrackForest dataset was augmented to 20709 images and their corresponding labels, and the Crack500 dataset was augmented to 22478 images and their corresponding labels. Images and their corresponding labels were distributed in a 6:3:1 ratio for the training, testing, and validation sets, resulting in the following datasets: CrackForest (12,425 images for training, 6,212 for testing, and 2,072 for validation) and Crack500 (13,486 images for training, 6,743 for testing, and 2,249 for validation). Table 1 shows the statistics for both datasets before and after augmentation.

[0112] Table 1: Statistical tables of the two datasets before and after augmentation

[0113]

[0114] The enhanced images and labels of the two datasets are as follows: Figure 5 and Figure 6 As shown, the first row contains the actual images, and the second row contains the corresponding labels.

[0115] 1.5.2 Experimental Environment Setup

[0116] The entire network architecture was implemented using the PyTorch framework. During network initialization, the default "uniform distribution" was used, and the bias of each convolutional layer was set to False. During training, BatchNormalization was used after each convolutional layer to improve the network's convergence speed. During testing or validation, all BatchNormalization and Dropout were disabled.

[0117] The training parameters are set as follows: the batch size is set to 8, the number of training epochs is 100, the initial learning rate is set to 0.001, and the step decay learning rate decay strategy is used. The learning rate decay is performed once every one iteration, and the calculation formula is shown in Equation (3).

[0118]

[0119] Wherein, Learning rate represents the current learning rate, Base Lr represents the initial learning rate, iterations represents the current number of iterations, and maxiterations represents the maximum number of iterations. The calculation formula is shown in Equation (4).

[0120]

[0121] Where train data nums represents the number of training data and max epoches represents the total number of training epochs.

[0122] The network optimizer uses stochastic gradient descent (SGD), with a momentum term set to 0.9 and a weight decay policy of 1e-4. After each epoch of training, the current network is validated on a validation set, and the optimal network weight parameters are saved using MioU as the metric.

[0123] All experiments were conducted on a device equipped with two NVIDIA GeForce RTX 409024-GB GPUs. The experimental environment was deployed on a server running Ubuntu 18.04LTS, with PyTorch version 2.1.2, CUDA version 12.0, and PyCharm 2022 as the compilation environment. The required software and hardware environment and their parameters are shown in Table 2.

[0124] Table 2: Hardware and software environment and parameters required for the experiment

[0125]

[0126] 1.5.3 Loss Function Setting

[0127] In the images of the CrackForest and Crack500 datasets, only two categories need to be segmented: background (RGB=0) and foreground cracks (RGB=255). For example... Figure 4 and Figure 5 As shown, in real crack images, the number of pixels in the foreground crack is often much lower than that in the background. This leads to an imbalanced segmentation problem in crack semantic segmentation. While class imbalance can often be addressed by assigning class weights, applying higher weights to the crack class in crack segmentation experiments increases false positives. Therefore, to mitigate the negative impact of foreground-background pixel imbalance in the samples, the experiment combined Dice Loss and CE Loss to enhance the foreground region mining during the training process.

[0128] Dice Loss is a loss function specifically designed for training image segmentation networks, aiming to measure the similarity between predicted and actual segmentation results. It originates from the Dice similarity coefficient, which is widely used in medical image segmentation. In image segmentation tasks, this coefficient is designed as a loss function, and its calculation process is similar to F1 score, considering both the accuracy and recall of the prediction results. The formula for calculating Dice Loss is shown in Equation (5).

[0129]

[0130] Where X represents the pixel label of the real segmented image in the dataset, and Y represents the pixel label of the predicted segmented image output by the model, then X∩Y| is the dot product approximation between the real pixel label and the predicted pixel label. |X| and |Y| can be approximated as the sum of the pixels in the real image and the predicted image, respectively.

[0131] In binary classification segmentation tasks, the segmentation labels of the real images only contain two values, 0 and 255. During the computation of the neural network, the predicted output will change these two values ​​to 0 and 1. Therefore, the calculation of |X∩Y| can clear the pixel values ​​in the predicted image that do not appear in the real image labels, thus eliminating false positive outputs. This penalizes low-confidence predictions, while higher-confidence predictions will yield higher Dice coefficients, and the Dice Loss will decrease accordingly.

[0132] According to equation (5), the value of Dice Loss ranges from 0 to 1. When the predicted image pixel content is exactly the same as the real image pixel content, the value of Dice Loss is 0, which means perfect prediction is achieved. When Dice Loss is close to 1, it means that the predicted pixel content is significantly different from the real image pixel content. During the training of the neural network, by continuously minimizing Dice Loss, the model can output a result that is more similar to the segmentation label of the real image. Dice Loss is also usually used in combination with other loss functions. Therefore, in the experiment, the cross-entropy loss function (CE Loss) and Dice Loss were combined as the final loss function. The calculation formula of CE Loss for binary classification is shown in equation (6):

[0133]

[0134] Where M represents the number of positive samples, N represents the number of negative samples, and y i p represents the true value. i This represents the predicted value.

[0135] Cross-entropy loss measures the accuracy of a model's predictions by multiplying the log probability of the true label by the log probability predicted by the model and summing the results. When the two probabilities are perfectly identical, the cross-entropy loss is zero; as the difference increases, the cross-entropy loss also increases, indicating uncertainty in the model's predictions.

[0136] In the experiment, the model was trained using a combination of Dice Loss and CE Loss. The joint loss function is shown in equation (7):

[0137] Loss=λ ce Loss ce +λ dice Loss dice (7) The model training effect is best when the weights of Dice Loss and CE Loss are the same. Meanwhile, testing has shown that the weight λ of CE Loss... ce The weights λ of Dice Loss dice When both are set to 0.5, the model training achieves a balanced result.

[0138] 1.5.4 Evaluation Indicator Setting

[0139] Pixels with a predicted image value greater than 0.5 are set to 1, representing the foreground crack, while all other pixels are set to 0, representing the background image. Therefore, the output image generates a binary matrix. Based on the binary prediction matrix, the foreground IoU and foreground F1 score (F1), commonly used in binary classification segmentation tasks, can be used to evaluate the model's performance. These three evaluation metrics are all calculated based on the confusion matrix, which is calculated using four parameters: true positive (TP), false positive (FP), false negative (TN), and false negative (FN). For each category, IoU is defined as the intersection-union ratio of the predicted value and the true value, calculated as shown in equation (8):

[0140]

[0141] The F1 score for each category is calculated as shown in equation (9):

[0142]

[0143] Where precision is TP / (TP+FP) and recall is TP / (TP+FN). Crack IoU is represented as IoU. crack Crack F1 is represented as F1 crack .

[0144] 2. Road surface crack detection network encoder based on self-attention mechanism

[0145] In the UNet network, ResNet50 is typically used directly as the encoder, and DBA-UNet also uses ResNet50 as the encoder. Although ResNet50 can expand its receptive field through multiple convolutional layers, it cannot capture long-range dependencies in the crack feature maps. Therefore, using ResNet50 as the encoder leads to discontinuities in the output crack image during crack segmentation, failing to guarantee the integrity of the segmented cracks. While DBA-UNet uses the SVD-IASPP module, this module only captures global information by capturing feature maps at different scales and cannot reflect the correlation between cracks. To enhance the correlation between each part of the feature map and other parts, the encoder part introduces a Swin-Transformer structure in the experiments. Through a hierarchical Swin-Transformer and the TCCF module, the feature maps output by the Swin-Transformer are used to supplement the global crack information extracted by the ResNet50 part. The encoder proposed above is named TCencoder. This encoder structure enhances the integrity and globality of cracks in the network output, and reduces the occurrence of crack breaks in the output image.

[0146] 2.1 Swin-Transformer

[0147] In the field of computer vision using deep learning, CNNs (Convolutional Neural Networks) and Transformers are the two main model architectures. CNNs can effectively process two-dimensional data structures and extract local features, but they mainly rely on local receptive fields and gradually expand them by stacking multiple convolutional layers. This approach is very limited when capturing long-distance dependencies. The self-attention mechanism relied upon by Transformers can capture global dependencies when processing images, making them more suitable for handling object relationships and details in complex scenes.

[0148] Swin-Transformer is a specially designed variant of Transformer that optimizes the computational cost and memory efficiency challenges encountered by traditional Transformer when processing large-scale image data. It introduces the concept of "shifted windows," using staggered windows at different levels to process information, effectively reducing computational complexity and improving the model's processing speed and efficiency. Its hierarchical design also allows the Transformer architecture to process images of different scales progressively through layer stacking, similar to CNNs. The overall architecture of Swin-Transformer is as follows: Figure 9 As shown.

[0149] The Swin-Transformer first segments the input image into multiple patches. Each patch is processed through a fully connected layer, transforming the raw pixel values ​​into a set of higher-dimensional feature representations. The Swin-Transformer then processes these vectors through multiple layers of Transformer blocks. After each layer, the Swin-Transformer uses pooling and convolution steps to reduce the resolution of the feature maps, gradually increasing the feature dimension and extracting more abstract features layer by layer. Unlike the Transformer, the Swin-Transformer uses local windows to compute self-attention; each block within a window only computes attention with other blocks within the same window. Between consecutive Transformer layers, the Swin-Transformer uses a sliding window mechanism, as shown in the diagram. Figure 10 As shown.

[0150] In the first layer of the Transformer, the self-attention mechanism uses a standard local window for computation. Before entering the next layer, the window is moved to the lower right by a fixed step. This shift causes some blocks at the window boundary to move to adjacent windows, and blocks that were not originally in the same window are grouped into the new window. The shifted layer uses the updated window to compute attention, so that information can be passed between the originally isolated windows, and the inter-layer information transfer is enhanced. In subsequent layers, shift or reverse shift operations will further enhance information integration.

[0151] The Swin-Transformer architecture saves resources and maintains a balance between global and local information when processing images. Compared with the standard Transformer, it performs better in multi-scale and complex visual tasks. Due to its hierarchical nature, it is easier to integrate with existing deep learning frameworks and technologies in the visual Transformer architecture and can be better used with traditional CNN feature extractors. By combining the Swin-Transformer with ResNet50, a TCCF module that fuses the two features was proposed, and then the TCencoder encoder network was proposed.

[0152] 2.2 Overall Model Architecture

[0153] In crack segmentation tasks, the inherent locality limitation of CNNs prevents the model's output from reflecting the overall picture. Using UNet as the baseline network, a novel encoder module, TCencoder, is proposed, combining a hierarchical Swin-Transformer with the original ResNet50 encoder network in the encoder section. These modules employ parallel structures for feature extraction. At the end of each layer, the feature map with global crack features extracted by the Swin-Transformer and the feature map with local crack features extracted by the CNN are combined as the input feature map for the next layer. Influenced by nonlocal networks and Cross-Attention, the features extracted by the Swin-Transformer branch are used to assist the feature map of the ResNet50 branch during the fusion process, thus proposing a fusion module, TCCF, which makes the crack information in the feature map more global. As the Swin-Transformer and ResNet50 branches downsample at each level, the fused output of each layer is used as a low-level feature and transmitted to the decoder in the skip connections. Finally, the SVD-IASPP and DBAM modules are inserted into the corresponding positions in the UNet network. Based on the above, a neural network model capable of effectively segmenting pavement cracks—TSA-UNet—is proposed. Furthermore, during training, the Canny boundary operator and its designed CannyBoundaryLoss are used at the output position to enhance the network's focus on crack boundaries while simultaneously reducing background information in the output feature map. The specific network structure of TSA-UNet is as follows: Figure 11 As shown.

[0154] The overall network architecture process begins with the crack image X∈R. H×W×C As input, since the crack image is an RGB image, C=3. The crack image then enters a multi-level TCencoder branch. After passing through four TCencoder blocks, the encoder finally outputs high-level features. The execution process of the bottle and decoder parts of the network is the same as that of the DBA-UNet network, and the final output is a semantic segmentation map with the same scale as the original image.

[0155] 2.3TCencoder

[0156] The proposed TCencoder is used as the encoder, which has an SWT (Swin-Transformer) branch with a global self-attention mechanism (Transformer) and a ResNet branch for local feature extraction (CNN). The ResNet branch has the same structure and process as ResNet50, where the crack image X∈RH×W×C After entering the ResNet branch, the feature maps of the ResNet branch are retained after each level of downsampling through the ResNet root and the three ResNet bodies. The feature maps of the SWT branch are then transferred to the feature maps of the ResNet branch through the TCCF module.

[0157] In another SWT branch, to reduce computational cost, the input features are first processed... Figure X ∈R H×W×C The patching operation is performed as follows: a 2×2 convolutional kernel is used to perform a convolution operation with a stride of 2. In the original Swin-Transformer, the patching process uses a 4×4 convolutional kernel to perform a convolution operation with a stride of 4. However, in order to maintain the same dimensionality as the output of the parallel ResNet Block, both the convolutional kernel size and the stride size are reduced to half that of the original Swin-Transformer.

[0158] After patching preprocessing, the feature map scale input to the SWT branch will be restored to its original scale, meaning the output feature map will become... Where width resnet The width of the ResNet layer is the number of channels in the feature map after processing by the ResNet root block. In order to reduce the computational cost of the self-attention mechanism and ensure that the features at the end of the SWT block are fully integrated with the features at the end of the ResNet block, the number of channels in the feature map after SWT patching preprocessing is set to half the number of channels in the ResNet root preprocessed feature map (set to 32 in this experiment).

[0159] After each level of the TCencoder block, the output feature map is compared with the feature map output by the ResNet block. Figure 1 The input is fed into the TCCF module, and then the features from the SWT branch are incorporated into the feature map of the ResNet branch. The new features, incorporating the SWT features, propagate in two directions simultaneously: into the next level's TCencoder block and into skip connection branches. After four cascaded TCencoder blocks, the encoder outputs high-level features.

[0160] 2.4TCCF

[0161] In the Transformer model, the Self-Attention mechanism endows it with exceptional insight when processing sequential data, enabling each element in the sequence to perceive the interactive effects of all other elements, thereby accurately capturing long-distance dependencies in the data. Specifically, each element of the input sequence is meticulously mapped into three representations: a query vector, a key vector, and a value vector. The output of Self-Attention is precisely calculated by precisely measuring the similarity between each query vector and all key vectors, and using these similarities as weights to perform a weighted summation of the corresponding value vectors. This process ensures that the output of each element is based on a comprehensive consideration of the entire input sequence, thus enabling the model to deeply understand and efficiently parse complex patterns in the input data.

[0162] Cross-Attention works similarly to Self-Attention but is often used in scenarios requiring the processing and integration of two different types of data. This mechanism typically involves a query sequence and a key-value sequence, allowing the model to focus on the information most relevant to the query for better integration of the two different types of data. In computer vision, Cross-Attention can also interact with two different image patches.

[0163] Based on the principle of Cross-Attention, a feature fusion module called TCCF (Transformer and CNN Cross Fusion) was designed to fuse feature maps from the SWT and ResNet branches. The network structure of TCCF is as follows: Figure 12 As shown.

[0164] The TCCF module receives two feature maps from the SWT branch and the ResNet branch, both of which are of the same size. First, the ResNet branch features undergo Mean Pooling, which captures global information and transforms the feature dimension to 1×C. Meanwhile, the SWT branch features retain their dimension after a 1×1 convolution. The SWT branch features, after the 1×1 convolution, are used as the Query, while the ResNet branch features, after Mean Pooling, are used as the Key, and the original ResNet feature map is used as the Value. The Key from the SWT branch and the Query from the ResNet branch are used for Cross-Attention calculation. The calculation process is shown in Equation (10).

[0165]

[0166] Among them. QuerySWT This refers to the query provided by the SWT branch. This represents the transpose of the Key provided by the ResNet branch, and the Value... Resnet This represents the value provided by the Resnet branch.

[0167] By calculating the correlation between these two features, they can be fused together more effectively. After the above calculation, a weight value of size HW×1 is obtained. This attention-weighted result is multiplied by the original ResNet branch features to obtain a feature map that fuses the SWT branch features, with a scale the same as the original two inputs.

[0168] The TCCF module strengthens the relationship between two feature maps through an attention mechanism and outputs a fused feature that integrates the information from both, thereby enhancing the information extraction effect of the feature map for complex scenes.

[0169] 2.4 Experiment and Results Analysis

[0170] 2.4.1 Dataset

[0171] Experiments were conducted using three pavement crack datasets: CrackForest, Crack500, and DeepCrack. The contents of the CrackForest and Crack500 datasets are as follows.

[0172] The DeepCrack dataset, created by Yang et al., contains various types of crack data. This dataset is frequently used to research and develop crack detection systems applicable to road or building maintenance. It contains 537 images at a resolution of 544×380, showcasing cracks of different types and widths, providing clear crack details, and featuring pixel-level crack annotations.

[0173] TSA-UNet was used to train, validate, and test the CrackForest, Crack500, and DeepCrack datasets, respectively. Since the DeepCrack dataset itself is relatively small, it still needs to be augmented using the data augmentation method described in Section 3.2.1. The specific parameters are set as follows: k = 3 (the magnification factor of the original image), n = 10 (the number of 512×512 images randomly cropped from the entire image after each 15° rotation). The data selection method remains unchanged, augmenting DeepCrack to 20683 images with a resolution of 512×512. These images were then divided into training, testing, and validation sets in a 6:3:1 ratio, resulting in a training set of 12409 images, a testing set of 6204 images, and a validation set of 2070 images. Table 11 shows the statistics before and after augmentation for the Crack500 dataset.

[0174] Table 11: Statistical table before and after Crack500 dataset augmentation

[0175]

[0176] The enhanced Deepcrack dataset includes images and labels such as Figure 13 as well as Figure 14 As shown, the first row is the enhanced real image, and the second row is the label corresponding to the real image.

[0177] 2.4.2 Experimental Environment Setup

[0178] The experimental setup was largely the same as described above, except that the total number of training epochs was set to 50. This was primarily because the training image data was plentiful, and the task was a binary classification semantic segmentation task. The model used was partly based on the Transformer architecture. For simple tasks, with sufficient data, the Transformer can learn effective features and patterns with fewer training epochs. Experiments showed that the same model exhibited similar performance at training epochs of 50 and 100, respectively; therefore, the total number of epochs in this experiment was set to 50.

[0179] 2.4.3 Loss Function Setting

[0180] Two loss methods were used: OhemCELoss and CannyBoundaryLoss.

[0181] In standard CE Loss, all samples are treated equally, with each sample contributing equally to the total loss. However, for many practical applications, certain misclassified samples provide more crucial information about model performance than other samples. OhemCELoss (Online Hard Example Mining Cross Entropy Loss) addresses this issue by focusing on these difficult-to-classify samples. Its calculation method is shown in Equations (11) and (12).

[0182]

[0183] In this process, the cross-entropy loss is calculated individually for each sample i according to equation (10), and then the loss value L is used to calculate the loss. i The K samples with the highest loss are selected and considered Hard Examples. K can be a fixed number or determined according to a certain proportion. Only these cases are included in the final loss calculation. Because in the road crack task, the foreground cracks are the more important part to focus on, as the number of pixels in the background is much greater than that of the foreground cracks. In the early stages of training, the loss for the background class is often much lower than that for the foreground cracks. Therefore, in the OhemCELoss calculation, crack samples are often treated as Hard Examples. This allows for focused training on the foreground crack part, helping to mitigate the impact of class imbalance.

[0184] Compared to Dice Loss, OhemCELoss can focus more on the most critical parts of model improvement when there are significant performance differences between categories (specifically, differences in loss values), making it more suitable for scenarios where the model needs to be optimized on samples that are difficult to identify or have a high error rate.

[0185] In addition, Canny Boundary Loss was incorporated into this experiment. The basic idea is to extract crack edge information from the mask labeled with the Canny operator. The calculation process of the Canny operator is as follows: First, a Gaussian filter is used to smooth the image to effectively reduce noise interference. Then, the Sobel operator is used to accurately calculate the gradient magnitude and direction of the image, thereby accurately identifying potential edge regions. Next, non-maximum suppression is used to remove non-edge pixels, retaining only the pixels with the largest gradient magnitude. Finally, a dual-threshold detection technique and an edge tracking strategy are used to identify and connect real edges, ensuring the clarity and continuity of edge lines. The crack labels and the edge feature maps generated after using the Canny edge operator are compared below. Figure 14 As shown.

[0186] The CannyBoundaryLoss used, with the help of the Canny operator, emphasizes the loss in the edge region. Its calculation method is shown in Equation (13).

[0187]

[0188] Where N is the number of elements, M i This is the edge mask of the i-th sample obtained using the Canny operator. In CannyBoundaryLoss, the edge mask M is first generated from the ground truth labels using the Canny operator, and then the predicted output is... The true label Y is used to calculate the cross-entropy loss, but this loss is multiplied by the edge mask, and the average loss is only calculated in the edge region.

[0189] Combining the OhemCELoss and CannyBoundaryLoss mentioned above, different weights are assigned to these two losses to train the TSA-UNet model. The final loss expression is shown in Equation (14).

[0190] Loss=λ Ohem Loss Ohem +λ CaB Loss CB (14)

[0191] Where, λ Ohem Let λ be 1. CB Set it to 20.

[0192] Example 2

[0193] Reference Figures 15-25 Based on the above embodiment one, a road surface crack segmentation system based on Jetson NX is disclosed.

[0194] 1. Overview of the Jetson NX Embedded Platform

[0195] With the rapid development of artificial intelligence technology, edge computing devices are widely used in fields such as intelligent detection, autonomous driving, and security monitoring. NVIDIA's Jetson Xavier NX is a high-performance embedded AI computing platform. With its super computing power and flexible deployment features, it has become one of the preferred platforms for edge intelligent applications in recent years. It has 384 NVIDIA CUDA cores, 48 ​​Tensor cores, and 6 Carmel ARMv8 264-bit CPU cores, with a theoretical computing power of 21 TOPS (Tera Operations Per Second), which can meet the real-time computing needs of high-load tasks such as deep learning inference and large-scale image processing.

[0196] In terms of storage and interface configuration, Jetson NX features 8GB of LPDDR4x memory and 16GB of eMMC storage, and supports an M.2 KeyM expansion interface, allowing connection to SSD storage devices to meet large-scale data access needs. The platform also integrates numerous peripheral interfaces, such as GPIO, I2C, SPI, UART, USB 3.1, and Gigabit Ethernet, which facilitates the access of various sensors, camera modules, and peripheral control devices and greatly enhances the flexibility and scalability of system design. Jetson NX supports the Ubuntu operating system and is compatible with NVIDIA JetPackSDK, providing developers with a complete software development environment covering various AI acceleration libraries such as CUDA, TensorRT, cuDNN, and DeepStream, thereby simplifying the deployment and optimization process of deep learning models on embedded systems.

[0197] Road surface crack detection requires embedded devices to meet the conditions of high computing performance, low power consumption, and miniaturization. Since the power consumption of deep neural network inference models running on Jetson NX can be controlled below 25W while balancing performance and energy efficiency, it is well-suited for use in road inspection robots, autonomous vehicles, and portable inspection equipment. Furthermore, after quantization, pruning, and Tensor RT acceleration optimization, Jetson NX can further reduce latency and resource consumption while ensuring high inference accuracy, achieving real-time detection and intelligent analysis of crack images. NVIDIA Jetson Xavier NX was selected as the deployment platform, and in-depth research was conducted on the system design and model deployment of the Jetson NX platform, providing technical support for the miniaturization and intelligentization of road surface crack segmentation systems. Figure 15 and Figure 16 The front and back views of the Jetson NX are shown in the images.

[0198] 2 System Overall Architecture Design

[0199] The Jetson NX embedded platform is the core of this system, supporting the inference and operation of the crack segmentation model. Since crack detection requires real-time acquisition of road images, the system is equipped with an image acquisition module connected to the Jetson NX via a CSI interface to achieve high-speed image data transmission. An external keyboard and mouse, communicating via a USB interface, meet operation and control requirements, making human-computer interaction more convenient. The display module, connected to the Jetson NX via an HDMI interface, can display detection results and system status information in real time. The system as a whole consists of the Jetson NX, a camera, input devices such as a keyboard and mouse communicating via USB, a display module, and a power supply module that provides stable power to each sub-module. These modules together constitute the hardware architecture of this system, as detailed below. Figure 17 The system hardware architecture diagram is shown.

[0200] Embedded devices: In the road crack segmentation system, model inference and system operation rely on embedded devices as the core component. To meet the system's high requirements for deep learning inference speed and processing accuracy, Jetson Xavier NX was chosen as the embedded platform. Figure 18 The Jetson NX embedded device shown features a 384-core NVIDIA Volta architecture GPU with 48 integrated Tensor cores to accelerate AI inference computation. The platform's built-in Deep Learning Accelerator (NVDLA) module also enhances the efficiency of deep neural network inference. It has 8GB of 128-bit LPDDR4x high-speed memory, sufficient for data transfer and storage during large-scale image processing. The Jetson NX offers a rich array of peripheral interfaces, including a CSI camera interface, a USB 3.1 interface, and GPIO expansion ports (including SPI, I2C, I2S, CAN bus, and UART communication interfaces), providing a hardware foundation for expanding the system's functionality with external cameras, LED modules, communication modules, and more.

[0201] Image Acquisition Equipment: For a road crack detection system to be accurate, it relies on the image acquisition module to obtain high-quality road images. Image quality significantly impacts the system's subsequent processing performance, making the selection of a suitable camera module crucial. Camera selection should prioritize resolution, as it determines the clarity of the acquired image. Low-resolution cameras may obscure details such as cracks, reducing detection accuracy. The system prioritizes high-resolution camera modules. Image distortion is common during recording, and wide-angle cameras are more prone to distortion at edges, affecting crack detection accuracy. Therefore, a camera module with minimal distortion should be selected. To ensure high-speed data transmission and reduce CPU resource consumption, a CSI interface camera is chosen to fully utilize the hardware advantages of the Jetson NX high-speed data transmission channel. Considering these factors, the Sony IMX219 camera was selected, with its 8-megapixel resolution and minimal image distortion during acquisition. Figure 19 The image shows the Sony IMX219 camera and its specifications.

[0202] Other devices: The Jetson NX embedded device and image acquisition module have already been introduced. The system also includes input devices such as a mouse and keyboard, a WiFi networking module, and a display output device. The connection relationships between these devices are as follows: Figure 20As shown, the mouse and keyboard can be directly connected to the Jetson NX platform via USB for system operation and parameter settings. The Intel dual-band WiFi-ac8265 module is used as a WiFi module and connects to the Jetson NX via the M.2 interface to achieve wireless data transmission and remote access. The display module is a 7-inch LCD capacitive touchscreen connected to the Jetson NX via HDMI to display test results and system status information in real time. With these peripheral devices configured, the human-computer interaction and ease of operation of the system are effectively improved, and hardware support is provided for the stable operation of the system.

[0203] 3 Design of a pavement crack segmentation system based on Jeston NX

[0204] Deep learning technology is used in this system for intelligent detection and analysis of acquired road surface crack images. Image acquisition is accomplished by a camera module, and the raw image data is obtained from the road surface. The acquired images undergo preprocessing, primarily edge detection and cropping, to remove background information outside the detection area and ensure the accuracy and validity of the input data. The processed image data is then input into a crack segmentation network to perform feature extraction and crack region recognition tasks, ultimately accurately locating and segmenting the crack positions. The overall workflow of the system is as follows: Figure 22 As shown.

[0205] After the system completes the camera input, it uses the CSI camera driver library of the Jetson platform to decode the input video stream, separating continuous video frames into single still images for subsequent processing. The designed road crack segmentation system uses a CSI interface camera and calls relevant functions in the driver library to achieve image acquisition.

[0206] The image data, after preprocessing, is input into the crack segmentation network for forward inference computation. To improve the inference speed of the model on the Jetson NX embedded platform, the TensorRT deep learning inference acceleration library is used to optimize the model. TensorRT is a high-efficiency inference framework based on C++ launched by NVIDIA, which can accelerate the execution of deep learning models in various GPU hardware environments. The model is first trained using the PyTorch framework, then the trained model format is converted to a standard format supported by TensorRT, and finally loaded and run by the TensorRT inference engine. The model can then perform efficient inference on the Jetson NX platform. Figure 23The example diagram of TensorRT's optimized convolutional neural network shows that TensorRT makes many optimizations to the characteristics of the convolutional neural network structure when parsing the model. It merges convolutional and activation layers, removes layers that don't use outputs to reduce redundant computation, and, where possible, combines operations such as convolution, bias, and ReLU activation into a single computational unit. TensorRT also integrates computational layers with similar parameters through horizontal fusion to further improve overall inference performance.

[0207] To verify the system's stability and functional integrity in a real hardware environment, physical testing was conducted on the Jetson NX embedded platform. The test platform included a Jetson NX main control unit, a CSI interface camera, a power adapter, a WiFi module, and an external display, just like... Figure 24 As shown in the physical connection diagram, the CSI camera was first correctly connected to the Jetson NX onboard camera interface via a ribbon cable to ensure a stable and reliable data transmission link. Next, a power adapter was used to provide a stable operating voltage to the Jetson NX, enabling the system to run continuously for extended periods. The WiFi module was connected via an M.2 interface to verify the system's remote communication and data transmission capabilities, and a display screen was also connected to monitor the system's operating status and inference results in real time. After the devices were connected, the system started and loaded the image crack segmentation detection program onto the Jetson NX. The camera acquired ground images in real time and transmitted them to the platform for inference processing. During testing, the focus was on checking the smoothness of image acquisition, the accuracy of model inference, and the stability of the overall system response, thus providing reliable experimental evidence for subsequent system optimization and functional expansion.

[0208] To verify the real-time performance and visualization effect of the image crack segmentation system, field tests were conducted on the Jetson NX embedded platform, and the inference results were displayed in real time on an external display screen to observe the physical model test. Figure 25 As can be seen, the left side of the display shows the original road surface image captured by the camera in real time, while the right side simultaneously displays the crack extraction results after segmentation processing by the TSA-UNet network. The inference output of Jetson NX is connected to a 7-inch portable LCD screen via HDMI interface, and with the help of keyboard and mouse peripherals, the system can be interactively controlled. During the test, the system processed the output image data in the form of video stream, which was very stable, and the crack edges were clearly extracted. The real-time response was smooth without obvious delay or stuttering. After visual observation and data verification, it can be confirmed that the crack segmentation model deployed on the Jetson NX platform has good real-time performance and accuracy, and can meet the requirements of real-time inference and result visualization for road crack detection. This physical test verified the effectiveness of the system's hardware and software co-design.

[0209] Example 3

[0210] Reference Figures 26-27Based on the above embodiments one and two, the web-based deployment and public network access of the road surface crack detection system are disclosed.

[0211] 1. Web-based system architecture design

[0212] The road surface crack detection system needs to be accessible in a multi-terminal environment and meet the system requirements of real-time performance, interactivity, and ease of deployment. Therefore, it is designed and implemented using a front-end and back-end separated web system architecture. This architecture separates the interface display and business logic processing, thereby effectively improving the modularity and scalability of the system. The front-end is mainly responsible for the construction and operation control of the interactive interface, while the back-end is responsible for data processing, model invocation, and response output. The two communicate with each other through the standardized HTTP protocol.

[0213] The Flask framework is used to build RESTful API interface services in the system backend. In the system, it receives image data sent by the frontend and calls the locally deployed TSA-UNet crack segmentation model to complete inference. Then, it returns the results to the frontend in JSON format. The frontend uses HTML5, CSS3 and JavaScript to implement the basic page structure, style beautification and interaction logic. AJAX technology is used to implement asynchronous requests and dynamic data updates to improve the system's response speed and user experience.

[0214] The system's overall functional modules cover image uploading, model inference, segmentation result display, and historical record management. Local images to be detected can be selected through the image upload module. The backend takes over the model inference module, performing image preprocessing, model invocation, and result saving. The results display module on the front-end interface presents the original image and the segmented image in real time, allowing for intuitive analysis of crack areas. The input and output data for each inference iteration are stored in the historical record module for easy comparison and retrieval later. Figure 26 The flowchart for web-based access and interaction is shown below.

[0215] During data transfer, the system typically employs a "request-response" mechanism. After clicking the upload button on the front-end page, the selected image is sent to the back-end API via an HTTP POST request. Upon receiving the request, the back-end automatically processes the image, performs inference, and generates a segmented image, then returns its path information. The front-end receives the response and dynamically updates the page content to display the original image and the corresponding crack detection results. This process is clearly structured, highly efficient, and adaptable, effectively meeting the system's remote access and deployment needs across different terminals. Figure 27 The overall architecture design diagram for the web-based system is shown below.

[0216] 2. Flask backend service implementation

[0217] To provide image processing and inference services for the road crack detection system, the backend uses Flask, a lightweight web framework in Python, as the main development tool. Flask is simple in structure, flexible in deployment, and has good scalability, making it suitable for building small and medium-sized web systems, and especially suitable for API encapsulation and embedded deployment scenarios of deep learning models. The system backend is responsible for image reception, model inference, result return, and historical record management. The frontend and backend communicate with each other via the HTTP protocol.

[0218] The image upload and model inference interface is the core function of the system. After the user uploads the image to be detected through the front-end page, the front-end will encapsulate the image file into multipart / form-data format via HTTP POST and then send it to the back-end interface / predict. The Flask back-end first obtains the uploaded file through request.files, and after judging its legality, it uses the secure_filename method to standardize the file name to avoid potential security risks, saves the file in the upload directory specified by the server, and then calls the deep learning model to perform crack detection inference.

[0219] The function `model_inference(image_path)` implements the model inference part. It takes the image path as input, calls the loaded TSA-UNet model to perform image segmentation, saves the generated crack mask image locally, and returns the result image path in JSON format to the front end for visualization. After the model is trained on the PC, it is integrated into the backend service after Tensor RT optimization to ensure stable inference and good performance, thus achieving efficient response.

[0220] The system also includes interfaces for historical record management and result retrieval. Each uploaded original image and corresponding segmentation result can be stored on the server. Through the query interface, historical data records can be returned to facilitate user comparison, analysis, and data tracing. Flask's default development server is used by the backend runtime environment to listen on the 0.0.0.0 address and port 5000, supporting access via LAN and public network requests.

[0221] The system backend has good maintainability and scalability due to its modular API design, which enables it to stably and reliably support remote interaction, embedded deployment and multi-terminal access.

[0222] 3. Front-end Interface Design and Interaction

[0223] The embedded pavement crack detection system needs to be usable in different terminal environments. The system front end needs to be based on the responsive design concept and use HTML5, CSS3 and JavaScript technologies to build a functional and easy-to-use image interaction interface. The front end is mainly responsible for image uploading, result display, status feedback and user interaction tasks, and communicates with the back end RESTful API to complete the complete closed loop of "request-reasoning-display" interaction.

[0224] After the user uploads the image, the front-end page will automatically display the original image uploaded by the user and the segmentation result image generated by the model inference side by side for the user to compare intuitively. In addition, the system also supports the function of popping up a full-screen preview when the image is clicked to improve usability. This can enhance the practical effect of crack localization analysis when the user zooms in to view the crack edges, morphology and other detection details. During image upload and model processing, the front-end will display a loading animation in real time to indicate the task status and prevent the user from making mistakes or submitting repeatedly. In addition, when the system encounters abnormal situations such as network errors or model response failures, the front-end will pop up a clear prompt to ensure the continuity and safety of the user's operation.

[0225] JavaScript is primarily used to implement front-end interaction logic, with image upload and asynchronous inference requests being its core functions. During upload, JavaScript uses `document.getElementById` to retrieve the image file from the upload control and prepares a `FormData` object. Then, it uses `fetch` to send a POST request to the ` / predict` interface to transmit the image to the backend for processing. After the backend completes the inference, it returns a JSON-formatted result path, which the frontend then dynamically updates using `data.result`. The `src` attribute of the tag enables the segmentation results to be displayed automatically, and the image display control is initially hidden. It is only made visible after the model inference is completed using `style.display='block'`, thus making the user interface interaction logic clear.

[0226] The system front-end realizes a complete closed loop from image input to result visualization. The interaction logic is simple, the page structure is clear, and it has good cross-platform compatibility and user experience, providing a reliable operation entry and visualization support for the entire pavement crack detection system.

[0227] 4 Huawei Cloud Public Network Penetration Solution

[0228] To enable remote access and independent platform deployment, the web service was deployed to a Huawei Cloud server. NAT gateway and port mapping were configured to achieve public network penetration for the internal network service. Simultaneously, domain name resolution was used to bind the system to a public IP address, allowing external users to directly access the system's front-end page through a browser to perform operations such as image uploading and result viewing, thus improving system availability and ease of access.

[0229] 5. System Security and Stability Design

[0230] In terms of security, the server firewall first controls ports, opening only necessary service ports such as 80 (HTTP) and 443 (HTTPS) while closing high-risk ports, thereby reducing the system's exposure surface.

[0231] To improve stability, the system introduces the process management tool Supervisor to monitor and restart backend services. When a backend process crashes or is unexpectedly interrupted, Supervisor can automatically restart the service to improve the system's fault tolerance in embedded environments.

[0232] 6 Deployment and Testing Process

[0233] The road surface crack detection system is deployed to Huawei Cloud servers to enable remote access and public network interaction. The entire deployment process includes backend service configuration, network forwarding strategies, HTTPS certificate installation, and domain name resolution. The system deployment is for rapid online deployment and stable operation, allowing users to remotely access the detection platform through a browser to complete image uploading and detection tasks.

[0234] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-scale feature pavement crack detection method based on a bi-branch attention mechanism, characterized in that, The steps are as follows: S1: Obtain the image of the road surface cracks to be detected; S2: Input the image obtained in step S1 into the crack detection network DBA-UNet; S3: The SVD-IASPP module is used in the bottleneck layer of the crack detection network DBA-UNet; S4: At the encoder-decoder jump connection, DBAM is used to perform channel-space dual-branch attention weighting on low-level features of the same scale; S5: The decoder upsamples step by step and fuses it with the weighted low-level features obtained in step S4, outputting a crack segmentation map with the same resolution as the input image.

2. The multi-scale feature pavement crack detection method based on a dual-branch attention mechanism according to claim 1, characterized in that, In step S1, the crack detection network DBA-UNet is based on UNet, and a dual-branch attention module DBAM is embedded in the skip connections and bottleneck layers.

3. The multi-scale feature pavement crack detection method based on a bi-branch attention mechanism according to claim 2, characterized in that, DBAM module, including The channel branch CB obtains global statistics through global average pooling and global max pooling, generates channel weight vectors through a fully connected layer, and multiplies them channel by channel with the original feature map; The spatial branch SB obtains spatial statistics through average pooling and max pooling in the channel dimension, generates a spatial weight map through 1×1 convolution, and multiplies it pixel by pixel with the original feature map. The outputs of CB and SB are added or concatenated element-wise to obtain the final attention-enhanced feature map.

4. The multi-scale feature pavement crack detection method based on a bi-branch attention mechanism according to claim 1, characterized in that, In step S3, the SVD-IASPP module extracts multi-scale crack features in parallel using multiple convolutions with different hole rates through inverted residual structures and depthwise separable convolutions, and then performs equalization stitching on the feature maps of each scale.

5. The multi-scale feature pavement crack detection method based on a bi-branch attention mechanism according to claim 4, characterized in that, The steps for using the SVD-IASPP module are as follows: Feature extraction was performed sequentially using InvertedDenseConv with porosities of 3, 6, 12, 18, and 24. The output of each InvertedDenseConv stage is concatenated with the input feature map before being fed into the next stage. All scale outputs are dimensionality reduced by 1×1 convolution and the original number of channels is restored to form a balanced multi-scale feature representation.

6. The multi-scale feature pavement crack detection method based on a dual-branch attention mechanism according to claim 5, characterized in that, The steps to build InvertedDenseConv are as follows: The number of channels in a 1×1 convolution is 1.5 times that of the input; 3×3 depth-separable dilated convolution, where the number of groups equals the number of input channels; 1×1 convolution reduces the dimensionality back to the original number of channels.

7. The multi-scale feature pavement crack detection method based on a bi-branch attention mechanism according to claim 1, characterized in that, In step S3, the training steps for the crack detection network DBA-UNet are as follows: Data augmentation was performed using CrackForest, Crack500, or DeepCrack datasets, with rotation, scaling, random cropping, and pixel filtering. Use a joint loss function; The Stochastic Gradient Descent (SGD) optimizer was used with an initial learning rate of 0.001, a decaying step size, a batch size of 8, and a training duration of 100 epochs. The optimal weights are saved on the validation set using MioU as the metric.

8. The multi-scale feature pavement crack detection method based on a bi-branch attention mechanism according to claim 1, characterized in that, It also includes the following steps: The ResNet50 and Swin-Transformer are used in parallel to build the TCencoder to extract local-global complementary features; At the end of each stage of the TCencoder, a TCCF module is introduced. The TCCF module injects the global features of the Swin-Transformer branch into the local features of the ResNet branch through Cross-Attention. TCA-UNet is formed by embedding TCA-UNet into UNet, and SVD-IASPP and DBAM are retained at the bottleneck layer and skip connections. During the training phase, OHEMCross-EntropyLoss and Canny edge enhancement loss are introduced.

9. The multi-scale feature pavement crack detection method based on a bi-branch attention mechanism according to claim 8, characterized in that, The TCCF module includes: Global average pooling is performed on the ResNet branch features to generate key vectors; The Swin-Transformer branch features are convolved with 1×1 to generate the Query matrix; Using the original ResNet features as Values, calculate the Cross-Attention weights of Query and Key and weight the Values ​​to obtain the fused feature map.

Images