Crack image segmentation method based on double encoders in complex environment

Abstract

The application discloses a crack image segmentation method based on double encoders in a complex environment, which improves the overall framework of the double-path encoder-decoder structure in the prior art CN202310525413.0, designs a new Transformer transformer block structure to effectively extract the global semantic information of the image, uses the multi-window high-low frequency mechanism of the Haar wavelet transform to extract the high-frequency features and low-frequency features of the image, and enhances the local information perception ability and the interaction between the image blocks, so as to solve the problems that the existing crack detection algorithm is difficult to adapt to a complex environment and has a large amount of calculation. In addition, a new feature fusion module is designed in the application to better fuse the intermediate features of the two encoders. The method of the application can accurately realize crack detection in various complex environments and reduce the amount of calculation, and improves the environmental versatility of crack detection.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and more specifically, relates to a crack image segmentation method based on dual encoders in complex environments. Background Technology

[0002] Cracks are a crucial indicator of structural health and are essential for inspection. Roads, bridges, and various concrete structures all develop cracks over time, making it one of the most common structural defects. However, due to the complexity of these structures and the significant safety risks involved, manual crack detection is a time-consuming, labor-intensive, and extremely challenging task. With the continuous development of deep learning, many excellent algorithms for crack detection and segmentation have been proposed. Automating crack detection can save considerable manpower and resources, and eliminate the subjective factors inherent in human inspection, thereby improving accuracy.

[0003] Current crack detection methods can be broadly categorized into two types: those employing traditional digital image processing techniques and those utilizing deep learning to build a model for automated crack detection. Traditional digital image processing techniques have been developed for many years and are relatively mature. For example, Salman et al. proposed a crack detection method based on Gabor filtering. Talab et al. used a Sobel filter to remove noise from grayscale images and employed the OTSU method for crack detection. Deep learning methods typically use convolutional neural networks to progressively extract features at various image scales, which are then gradually reconstructed by a decoder to obtain a segmentation mask. For example, Choi et al. proposed SDDNet for real-time crack segmentation, and Jiang et al. proposed HDCB-Net for segmenting concrete cracks in bridges. In recent years, Transformers have achieved considerable success in various computer vision tasks. They possess many features that convolutional neural networks lack, such as input adaptation, parallel processing of the entire input sequence, and excellent long-range dependency modeling capabilities. For example, Wang et al. proposed the crack segmentation network SegCrack, whose encoder is implemented using a Transformer.

[0004] However, while traditional digital image processing methods offer advantages such as high processing speed and low hardware requirements, their robustness and accuracy are low, and detection results are significantly affected by the environment. Therefore, they are difficult to adapt to complex environments in practical applications. Convolutional neural network (CNN) segmentation networks often suffer from excessively deep convolutional layers, leading to an ever-expanding receptive field and the loss of much detailed information. This makes it difficult to recover small regions during subsequent image reconstruction by the decoder. Cracks, in particular, are mostly thin and elongated, which is detrimental to crack segmentation and detection. Transformer-based models often involve enormous computational demands, resulting in a heavy computational burden and cost. Furthermore, an overemphasis on global information can lead to the neglect of local details. In the prior art, the inventors of this invention first applied for Chinese patent application CN202310525413.0 (patent publication number CN 116563544 A, publication date 2023.08.08), which disclosed a road crack segmentation method based on a dual path of convolutional neural network and transformer. Although this method can effectively improve the accuracy and efficiency of segmentation and detection of small cracks, it is mainly aimed at the environment of road surface cracks. In practical applications, it has been found that it is difficult to adapt to more complex environments and has a large amount of computation. Its environmental versatility for crack detection is not strong.

[0005] In order to facilitate accurate crack detection in various complex environments and reduce computational load, it is urgent to improve the design of a new crack image segmentation method to enhance the environmental versatility of crack detection. Summary of the Invention

[0006] (I) Technical Issues

[0007] Based on this, this invention improves upon the design of a crack image segmentation method based on dual encoders in complex environments. It appropriately modifies the overall framework of the dual-path encoder-decoder structure in the existing technology CN202310525413.0, and effectively extracts global semantic information of the image by designing a new Transformer block structure. Simultaneously, it extracts both high-frequency and low-frequency features of the image, and enhances the local information perception capability and the interactivity between image blocks. This effectively addresses the problems of existing crack detection algorithms being difficult to adapt to complex environments and having a large computational load, thus improving the environmental versatility of crack detection. Furthermore, this invention also specifically designs a novel complementary feature fusion module to fuse the intermediate features of the two encoders.

[0008] (II) Technical Solution

[0009] This invention provides a crack image segmentation method based on dual encoders in complex environments. This crack image segmentation method improves the overall framework of the road crack segmentation method based on dual paths of convolutional neural network and transformer in patent number CN202310525413.0 in aspects (a)-(b):

[0010] (a) Dual encoder

[0011] The dual encoder includes a convolutional neural network encoder branch and a Transformer encoder branch. In the Transformer encoder branch, all four transformer blocks are replaced with Transformer blocks. Before the input image is input to the Transformer encoder branch, it is first divided into several image blocks and flattened into an image sequence. After a linear projection, it is input into the Transformer encoder branch structure composed of four Transformer blocks and three image block merging layers. The Transformer blocks in each layer of the dual encoder are executed three times.

[0012] (b) Decoder

[0013] Replace all three transformer blocks in the decoder with the Transformer block, and execute the Transformer block of each layer in the decoder once;

[0014] The Transformer block includes a high- and low-frequency attention mechanism, a first summation and layer normalization, a local enhancement feedforward network, and a second summation and layer normalization, which are connected in series.

[0015] In the high- and low-frequency attention mechanism, the standard transform in the Haar wavelet transform is first used, followed by row filtering and then column filtering, to perform a first-level decomposition, resulting in three detail components and one approximate component. Since the image can be considered a discrete function with values ​​ranging from 0 to 1, it can be decomposed by the scaling function. The scaling function, expressed as the sum of products of its coefficients, is:

[0016]

[0017] wavelet function Then it can be represented by a scaling function, that is The first-order wavelet decomposition is expressed as Here, C0 is the required approximate component; then, C0 is upsampled to the original image size, and the approximate component is subtracted from the original image to obtain the required detail component. For the obtained approximate component and detail component, a set of query Q, key K, and value V are obtained through linear transformation. The query of the approximate component is obtained from the input. Then, the two different sets of Q, K, and V are calculated through scaled dot product attention to obtain the scaled dot product self-attention result SA of a head. h A linear transformation is applied to the result obtained by concatenating multiple heads to obtain the result of multi-head self-attention; when the feature dimension is N h When, the approximate components are divided into αN h The detail components are divided into (1-α)N h The high-frequency and low-frequency features are obtained respectively, and they are connected in the channel direction as the output. Then, they pass through the residual structure that is first added to the input and then normalized by the layer.

[0018] In another aspect, the present invention also discloses a crack image segmentation system based on dual encoders in complex environments, comprising:

[0019] At least one processor; and

[0020] At least one memory communicatively connected to the processor, wherein:

[0021] The memory stores program instructions that can be executed by the processor, which can call the program instructions to execute the crack image segmentation method based on dual encoders in complex environments as described above.

[0022] In another aspect, the present invention also discloses a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform a crack image segmentation method based on dual encoders in a complex environment as described in any of the preceding claims.

[0023] (III) Beneficial Effects

[0024] (1) First, this invention improves the overall architecture of a crack segmentation network model with dual encoder paths, which can adapt to various complex environments and has good computational performance. Convolutional neural network encoder branches are constructed from the convolutional layers of ResNet-50 to extract local detail information of the image, showing excellent extraction of wide and obvious crack information. A Transformer encoder branch is independently designed to extract global semantic information of the image, improving the ability to capture small cracks. The features extracted by the improved dual encoders will be more fully integrated, resulting in fewer multiply-accumulate operations and a higher frame rate, allowing for faster application in practice.

[0025] (2) Secondly, this invention designs a novel Transformer block structure, which can be used as a plug-and-play component to extract image information. The Transformer includes a high- and low-frequency attention mechanism based on Haar wavelet transform, which can simultaneously extract high-frequency and low-frequency features of the image. Furthermore, an improved local enhancement feedforward network is set up, which can effectively enhance the interaction between various image blocks and improve the local information perception capability of the network model.

[0026] (3) Furthermore, this invention also designs a novel feature fusion module to complementaryly fuse the intermediate features of two encoders. First, the two different features are aligned to make their dimensions consistent. Then, channel attention is used to adjust the weights of each channel and suppress unfavorable and redundant channels. Next, their correlation is enhanced and domain fusion is performed. Finally, a feature fusion block consisting of a series of convolutional units, batch normalization, and activation functions is used to aggregate the multi-class features to obtain the fused features.

[0027] (4) Extensive experiments were conducted to verify the performance of this invention. On the two publicly available crack datasets, DeepCrack and Crack3238, comparisons were made with ten state-of-the-art networks, and the overall performance was superior to all other networks. Furthermore, the model's recall rate was significantly improved, and the crack detection rate in complex environments was noticeably enhanced. Ablation experiments were also conducted to verify the effectiveness of the proposed method and modules. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the overall structure of the dual encoder of the present invention.

[0029] Figure 2 for Figure 1 The structure and flowchart of the Transformer block.

[0030] Figure 3 The structure and flowchart of the high- and low-frequency attention mechanism based on Haar transform in the Transformer block are shown.

[0031] Figure 4 This is a diagram showing the structure and flowchart of the local augmentation feedforward network in the Transformer block.

[0032] Figure 5 for Figure 1 The structure and flowchart of the complementary feature fusion module.

[0033] Figure 6 This is a flowchart illustrating the structure and process of the cross-domain fusion block in the feature fusion module.

[0034] Figure 7This is a segmentation example of the present invention on the DeepCrack dataset.

[0035] Figure 8 This is a segmentation example of the present invention on the Crack3238 dataset.

[0036] Figure 9 Examples of feature maps extracted from the intermediate layers of different modules in this invention. Detailed Implementation

[0037] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0038] The prior art CN202310525413.0 discloses a road crack segmentation method based on a dual path of convolutional neural network and transformer. Although this method can effectively improve the accuracy and efficiency of segmentation and detection of small cracks, the inventors found through practical application that it is difficult to adapt to more complex environments and has a large computational load. Therefore, the present invention has made targeted improvements on this basis.

[0039] like Figure 1 As shown, this invention also employs a convolutional neural network and a Transformer to construct a novel dual-encoder path network. The dual encoder includes a convolutional neural network encoder branch and a Transformer encoder branch. It makes certain distinctive modifications to the overall structure of the original dual encoder in CN202310525413.0, enabling the new Transformer encoder branch to extract semantic information from images more comprehensively, thus improving the network's recall rate and enhancing the detection rate and computational efficiency for small cracks in complex environments. This invention uses the widely used ResNet-50 as the convolutional neural network encoder branch to extract local detail information from images, effectively extracting wide and obvious crack information in various environments. The entire network performs downsampling layer by layer from top to bottom, extracting information at different scales. A complementary feature fusion module proposed in this invention is embedded between each layer to aggregate the features extracted by the two encoder branches. The fused features from the last layer are sent to the decoder for decoding, while the fused features from other layers are passed to the corresponding layers of the decoder through skip connections, aiding in the recovery of image information.

[0040] Based on the overall framework of existing technology CN202310525413.0, this study... Figure 1 The dual encoder portion and decoder of the present invention shown have undergone detailed improvements in the following aspects:

[0041] (a) Convolutional neural network encoder branch

[0042] This invention utilizes some layers from the widely used ResNet-50 network, extracting the initialization layer, max pooling layer, and four convolutional layers. The input image first passes through the initialization layer and max pooling layer, and then through the four convolutional layers to extract local features. The intermediate features of each of the four convolutional layers are retained and output to four feature fusion modules, where they are fused with the intermediate features corresponding to the Transformer encoder branch.

[0043] It should be noted that in the existing technology CN202310525413.0, the convolutional layer 1 of the convolutional neural network encoder branch does not input intermediate features to the complementary fusion module, and the features of the depth-separable convolution input connected to the convolutional layer 4 need to be input into a simple fusion module (which has a different structure from the other three complementary fusion modules), and no preprocessing of the input data through the initialization layer and the max pooling layer is performed.

[0044] (b) Transformer encoder branch

[0045] Before the input image is fed into the Transformer encoder branch, it is first divided into several image blocks and flattened into an image sequence. Then, it undergoes a linear projection. The Transformer encoder branch structure is consistent with existing technologies, consisting of four Transformer blocks and three image block merging layers. That is, after the image undergoes a linear projection, it is input into a Transformer block, so that the output of the Transformer block is input in parallel with the output of convolutional layer 1 into a feature fusion module. Subsequently, three transformer decoding layers consisting of image block merging layers and Transformer blocks are connected in sequence. The outputs of the three transformer decoding layers correspond to convolutional layers 2-4 in sequence, and feature fusion is performed through the other three feature fusion modules.

[0046] The newly designed Transformer block consists of a residual structure composed of two parts: a high- and low-frequency attention mechanism based on Haar transform and a local enhancement feedforward network proposed in this invention. Each Transformer block in the encoder is executed three times, meaning that the Transformer block of each layer will be executed three times. The image block merging layer will merge every four adjacent blocks into one, thereby obtaining a feature map of the same scale as each layer of the convolutional neural network.

[0047] (c) Decoder section

[0048] For the decoder section, the skip connections of the four feature fusion modules and the main structure of the decoder in this invention are consistent with the decoder section of the prior art CN202310525413.0. Specifically, the input of the feature fusion module corresponding to the bottom convolutional layer 4 is fed into decoder layer 1 as the basic image recovery signal. The outputs of the three feature fusion modules corresponding to convolutional layers 3-1 are sequentially input into the serially connected decoder layers 1-3. Finally, the result is output through the segmentation head connected to decoder layer 3. In this invention, the three transformer blocks 5-7 in the original decoder layer are replaced with the Transformer blocks of this invention. Unlike the Transformer blocks in the encoder, which are executed three times repeatedly, the Transformer blocks in the decoder are executed only once. Furthermore, the decoder output section specifically uses a segmentation head composed of sub-pixel convolutions and 1×1 convolutions.

[0049] To achieve more accurate crack detection and reduce computational load in various complex environments, Figure 1 Based on the aforementioned dual encoder and decoder, this invention specifically proposes a crack image segmentation method based on a dual encoder in complex environments, which includes the following steps:

[0050] I. Data Input

[0051] Step 1: Data loading, cropping the input image and performing data augmentation.

[0052] Specifically, in step 1, during network training and testing, this invention performs appropriate preprocessing and data augmentation on the input image to enable the network to better segment the image. During the training phase, the input image is cropped and appropriately augmented. The image is sequentially resized, randomly flipped horizontally, randomly flipped vertically, randomly rotated, and center-cropped. During the testing phase, the image is only cropped to the specified input size. Regularization is applied to all three channels of the image. The input image in this invention can be cropped to 512×512 and 256×256.

[0053] II. Convolutional Neural Network Encoder Branch

[0054] Step 2: The encoder branch of the convolutional neural network encodes the input image.

[0055] Specifically, in step 2, this invention uses certain layers from the existing ResNet-50 to construct the convolutional neural network encoder branch. This invention deconstructs ResNet-50, extracting the initialization layer, max-pooling layer, and four convolutional layers. The input image first passes through the initialization layer and max-pooling layer, and local features are extracted during the four convolutional layers. The intermediate features of each layer are retained and output to the feature fusion module, which then fuses them with the intermediate features corresponding to the Transformer encoder branch.

[0056] III. Transformer Encoder Branch

[0057] Step S3: The Transformer encoder branch encodes the input image.

[0058] Specifically, in step 3, before the image is input to the Transformer encoder branch, it is first divided into several image blocks and flattened into an image sequence, and then subjected to a linear projection. The Transformer encoder branch mainly consists of Transformer blocks and an image block merging layer. The Transformer block is composed of a residual structure consisting of a high- and low-frequency attention mechanism based on Haar transform and a local enhancement feedforward network proposed in this invention, as shown below. Figure 2 As shown, each Transformer block in each layer is executed three times. The image block merging layer merges every four adjacent blocks into one, thus obtaining a feature map of the same scale as each layer of the convolutional neural network.

[0059] Step 4: In each Transformer block, the feature map is processed sequentially through the high and low frequency attention mechanism, the first summation and layer normalization, the local enhancement feedforward network, and the second summation and layer normalization.

[0060] like Figure 2 As shown, the Transformer block includes a high- and low-frequency attention mechanism, a first summation and layer normalization, a local enhancement feedforward network, and a second summation and layer normalization connected in series. The main change is that the fully convolutional high- and low-frequency attention module and feedforward network in the original transformer block are replaced with the high- and low-frequency attention mechanism and the local enhancement feedforward network.

[0061] Specifically, in step 4, such as Figure 3 As shown, in the high- and low-frequency attention mechanism, the standard transform in the Haar wavelet transform is first used, that is, row filtering is performed first and then column filtering is performed, and a first-level decomposition is performed to obtain three detail components and one approximate component; if the image can be regarded as a discrete function with values ​​ranging from 0 to 1, then it can be obtained by the scaling function. The scaling function is expressed as the sum of products of its coefficients;

[0062]

[0063] wavelet function Then it can be represented by a scaling function, that is The first-order wavelet decomposition can be expressed as C0 is the approximate component required by this invention. Then, C0 is upsampled to the original image size, and the approximate component is subtracted from the original image to obtain the required detail component. For the obtained approximate and detail components, a set of queries (Q), keys (K), and values ​​(V) are obtained through linear transformations, respectively. To ensure that the size remains unchanged after calculating self-attention, the query for the approximate component is obtained from the input. Then, the two different sets of Q, K, and V are calculated using scaled dot product attention.

[0064]

[0065] Among them, SA h D represents the result of a scaled dot product of a head for self-attention. h The hidden layer dimension is N; then, the result obtained by concatenating multiple heads is subjected to a linear transformation to obtain the multi-head self-attention result. To reduce computational cost, while obtaining the approximate and detail components, the feature dimension is divided according to a certain ratio α. When the feature dimension is N... h When, the approximate components are divided into αN h The detail components are divided into (1-α)N h This method integrates the high- and low-frequency features of both multi-head self-attention and multi-head attention. Furthermore, the attention calculation is divided into several windows, each calculated independently. High-frequency and low-frequency features are then obtained separately and concatenated along the channel direction as the output. The output then passes through a residual structure that undergoes a first addition with the input and layer normalization.

[0066] In step 4, the results of the first summation and layer normalization will be projected into a high-dimensional space through a locally enhanced feedforward network to learn more abstract features that are difficult to capture in the low-dimensional space, and then projected back into the original space.

[0067] Specifically, such as Figure 4As shown, this invention first performs spatial restoration on the input sequence to transform it into the dimensions of an image; then, it enhances the dimensionality using depthwise separable convolution and the GELU activation function; subsequently, in the high-dimensional space, depthwise separable convolution and the GELU activation function are used to capture local features, enabling interaction between blocks and enhancing the network's expressive power. Next, the original dimensions are restored using inverted depthwise separable convolution and the GELU activation function, followed by batch normalization, and flattened into a single sequence. Finally, the residual structure, after a second addition with the input and layer normalization, serves as the output.

[0068] IV. Feature Fusion Module

[0069] Step 5: For the features of each intermediate layer of the convolutional neural network encoder branch and the Transformer encoder branch, fuse them using the unified input feature fusion module.

[0070] Specifically, in step 5, considering the features Cv extracted from the i-th layer of the convolutional neural network encoder... i Features T extracted from the i-th layer of the Transformer encoder i There is inconsistency in the channel dimension. Therefore, as... Figure 5 As shown, the present invention first receives the intermediate feature Cv from the dual encoder input. i and T i Cv i T is an intermediate feature of the i-th layer of the encoder branch of a convolutional neural network. i These are intermediate features from the i-th layer of the Transformer encoder branch, adjusted using a 1×1 convolution. i The channel dimension is then determined; next, channel attention is used to adjust the channel responses of the two features, increasing the weight of favorable channels and reducing the influence of redundant channels. Through this series of processes, the desired result can be obtained. and Then, and Matrix multiplication is performed to enhance the correlation between the two components, resulting in a correlation-enhanced feature. Furthermore, and It will also input a cross-domain fusion module to merge information between different domains, such as Figure 6 As shown, two different features are linearly transformed to obtain two sets of queries, keys, and values. Then, multi-head attention is calculated between each query class and the keys and values ​​of the other class. Finally, the two results are concatenated along the channel direction, and a 1×1 convolution is used to reduce dimensionality and extract effective information, resulting in a cross-domain fusion feature. Finally, this invention connects the relevance enhancement feature, the cross-domain fusion feature, and... and Fourthly, dimensionality is reduced through feature fusion blocks, namely inverted depthwise separable convolution, batch normalization, and GELU activation function; multi-class features are fully fused using depthwise separable convolution, batch normalization, and GELU activation function, as well as 1×1 convolution, batch normalization, and GELU activation function, to extract effective information. Finally, the fused feature F of this layer is output. i .

[0071] V. Decoder

[0072] Step 6: The decoder consists of three cascaded decoder layers and a segmentation head. Each decoder layer consists of subpixel convolutions, inverted depthwise separable convolutions, and Transformer blocks along the channel direction. The corresponding feature fusion module is input between the subpixel convolutions and inverted depthwise separable convolutions of the decoder layer, and each Transformer block in the decoder layer is executed only once.

[0073] The fused features obtained from the last layer of the encoder serve as the initial input to the decoder. Each decoder layer initially undergoes subpixel convolution for upsampling, then concatenates the fused features at the corresponding scale, performs dimensionality reduction through inverted depthwise separable convolution, and finally aggregates and decodes the features using a Transformer block. After three decoder layers, the signal enters the segmentation head. It first upsamples the signal by a factor of 4 to the original size through subpixel convolution, then performs a 1×1 convolution to obtain the final segmentation mask.

[0074] VI. Environmental Setup

[0075] Step 8: The loss function used during model training is the sum of the binary cross-entropy (BCE) loss and the dice loss, i.e., BCE plus Dice:

[0076]

[0077]

[0078] Where N represents the total number of pixels in the image, t i That is the true category of the pixel, p i ε is the pixel category predicted by the network, and ε is the smoothing factor to prevent division by zero.

[0079] During training, the batch size was set to 24, and a total of 70 groups were trained. The optimizer chosen was AdamW with an initial learning rate of 0.001, and the scheduler employed a cosine annealing strategy. The entire model was implemented using Python 3.10 and PyTorch 1.11.0. An Nvidia GTX 3090 graphics card with 24GB of RAM and an Intel(R) Xeon(R) Platinum 8350C CPU were selected.

[0080] It should be noted that step 8 is not a necessary step in the encoder and decoder and steps 2 and 3 can be executed in parallel; it is only to make the trained network more robust.

[0081] As shown in steps 1-8 above, to adapt to crack detection in complex environments and reduce computational load, this invention, compared with existing technologies, uses a convolutional neural network and a Transformer to jointly construct a new dual-encoder path network. This network has fewer multiply-accumulate operations and a higher frame rate. The Transformer encoder branch can extract semantic information from the image more comprehensively, helping to improve the network's recall rate and increase the detection rate of small cracks. This invention uses the widely used ResNet-50 as the convolutional neural network encoder branch to extract local detail information from the image, effectively extracting information about wider and more obvious cracks. The entire network downsamples layer by layer from top to bottom, extracting information at different scales. A complementary feature fusion module with better performance, proposed in this invention, is embedded between each layer to aggregate the features extracted by the two encoder branches. The fused features from the last layer are sent to the decoder for decoding, while the fused features from other layers are passed to the corresponding layers of the decoder through skip connections, aiding in the recovery of image information.

[0082] To illustrate the beneficial effects of the method of the present invention, the following also combines the following: Figure 7-9 The advantages of the dual encoder structure and crack image segmentation method of the present invention are described in detail in the experimental results diagrams, Tables 1-6, and the following Example 1:

[0083] Example 1

[0084] Example 1 was developed entirely using the PyTorch framework. The relevant configuration details are as follows: the batch size during training was set to 24, and a total of 70 groups were trained. The optimizer chosen was AdamW, with an initial learning rate of 0.001, and the scheduler employed a cosine annealing strategy. The entire model was implemented using Python 3.10 and PyTorch 1.11.0. An Nvidia GTX 3090 graphics card with 24GB of RAM and an Intel(R) Xeon(R) Platinum 8350C CPU were selected.

[0085] First, to demonstrate the optimal number of executions for each Transformer block of the encoder, a comparative experiment was conducted on the DeepCrack dataset. This invention compared the results after one, two, and three executions, and the results are shown in Table 1.

[0086] Table 1 Comparison of execution counts for different Transformer blocks

[0087] Execution count accuracy Recall rate F1 score Average crossover ratio ×1 0.8317 0.8945 0.8620 0.7459 ×2 0.8339 0.9081 0.8694 0.7415 ×3 0.8287 0.9270 0.8751 0.7523

[0088] In addition, this invention selected different versions of the ResNet network to form the convolutional neural network encoder branch. This invention found that ResNet-50 had the best overall performance, and the results are shown in Table 2.

[0089] Table 2 Comparison of different encoder branches in convolutional neural networks

[0090] Model accuracy Recall rate F1 score Average crossover ratio ResNet-18 0.6853 0.9178 0.7847 0.6641 ResNet-34 0.7878 0.9008 0.8405 0.7077 ResNet-50 0.8287 0.9270 0.8751 0.7523 ResNet-101 0.8926 0.8258 0.8579 0.7284 ResNet-152 0.8537 0.8772 0.8653 0.7310

[0091] Furthermore, this invention verifies the effectiveness of the dual-path encoder by canceling one of the encoder paths. Experiments show that the dual-path encoder structure effectively improves the network's recall rate, resulting in a significant improvement in overall performance. The results are shown in Table 3.

[0092] Table 3. Comparison of the effectiveness of convolutional neural network encoders and Transformer encoders.

[0093]

[0094]

[0095] To better illustrate the effectiveness of the newly designed feature fusion module, ablation experiments were conducted, as shown in Table 4. The effectiveness of each of the main basic units was tested. It can be seen that the network performs best when all three main operations are executed. Furthermore, channel attention significantly increases the network's recall. Additionally, it can be observed that the cross-domain fusion block has a large computational load; removing this module significantly improves the network's computation and execution speed.

[0096] Table 4 Comparison of the effectiveness of each component unit within the feature fusion module

[0097] Channel attention Enhanced correlation Cross-domain fusion block accuracy Recall rate F1 score Average crossover ratio Frames per second ↑ 0.8786 0.7849 0.8291 0.6863 38.29 √ 0.8439 0.8666 0.8551 0.7264 35.76 √ 0.8496 0.8320 0.8407 0.7076 36.61 √ 0.8686 0.8033 0.8347 0.7027 31.02 √ √ 0.8260 0.8667 0.8459 0.7227 32.68 √ √ 0.8213 0.8835 0.8513 0.7240 30.67 √ √ 0.8748 0.8409 0.8575 0.7318 35.64 √ √ √ 0.8287 0.9270 0.8751 0.7523 30.02

[0098] To verify the advancement of this invention, it was compared with ten current state-of-the-art methods on two publicly available datasets. The results show that the overall performance of the method presented in this invention is superior to the other methods. The results are shown in Tables 5 and 6.

[0099] Table 5 compares the models on the DeepCrack dataset.

[0100] Model accuracy Recall rate F1 score Average crossover ratio UNet-ResNet34 0.8568 0.7912 0.8227 0.6727 Attn-UNet 0.8612 0.7975 0.8282 0.6874 DeepLabv3+ 0.8893 0.7241 0.7982 0.6818 CENet 0.9024 0.7519 0.8203 0.6754 TransFuse 0.9253 0.6624 0.7721 0.6076 UTNet 0.8930 0.8239 0.8570 0.7387 FAT-Net 0.8857 0.8052 0.8436 0.7106 DeepCrack 0.8842 0.8236 0.8528 0.7191 DcsNet 0.9104 0.6961 0.7889 0.6287 DTrC-Net 0.9403 0.6651 0.7791 0.6324 This invention 0.8287 0.9270 0.8751 0.7523

[0101] Table 6 compares the models on the Crack3238 dataset.

[0102] Model accuracy Recall rate F1 score Average crossover ratio UNet-ResNet34 0.7232 0.7314 0.7121 0.5878 Attn-UNet 0.7555 0.7260 0.7278 0.6039 DeepLabv3+ 0.6783 0.7874 0.7155 0.5837 CENet 0.7475 0.7470 0.7332 0.6061 TransFuse 0.7318 0.7256 0.7112 0.5826 UTNet 0.7409 0.7561 0.7368 0.6149 FAT-Net 0.7507 0.7433 0.7347 0.6131 DeepCrack 0.7197 0.7372 0.7138 0.5870 DcsNet 0.7397 0.7488 0.7307 0.6073 DTrC-Net 0.7560 0.7886 0.7644 0.6430 This invention 0.7826 0.7902 0.7864 0.5869

[0103] To demonstrate the practicality and efficiency of this method, it was compared with other models. Although this method is not optimal in the compared metrics, it is still at a mid-level, and its actual network performance is the best, as shown in Table 7. The number of frames per second also meets the practical application requirements, and its segmentation performance is far superior to other models.

[0104] Table 7 Performance Comparison of Each Model

[0105] Model Multiply-accumulate cumulative operand (G) ↓ Parameter quantity (M) ↓ Frames per second ↑ UNet-ResNet34 90.13 69.30 41.07 Attn-UNet 135.34 57.16 28.94 DeepLabv3+ 22.14 59.23 30.64 CENet 8.90 29.00 32.96 TransFuse 63.40 143.39 18.52 UTNet 5.19 3.62 31.59 FAT-Net 42.80 29.62 28.00 DeepCrack 136.80 30.91 27.35 DcsNet 15.79 25.85 38.19 DTrC-Net 123.20 63.45 28.78 This invention 26.08 68.86 30.02

[0106] Additionally, see Figure 7-9 As shown, the crack image segmentation method and structure based on dual encoders in complex environments of the present invention have good performance on datasets in various complex environments, and the features of dual encoders have better complementarity and computational characteristics, making them more versatile and well applicable to crack image segmentation in various environments.

[0107] It is also worth mentioning that the crack image segmentation method of the present invention can be converted into software program instructions, which can be implemented by a software analysis system including a processor and memory, or by computer instructions stored in a non-transitory computer-readable storage medium.

[0108] Finally, the method of this invention is merely a preferred embodiment and is not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A crack image segmentation method based on dual encoders in complex environments, characterized in that, This crack image segmentation method makes improvements (a)-(b) to the original overall framework of the road crack segmentation method based on convolutional neural networks and transformer dual paths: (a) Dual encoder The dual encoder includes a convolutional neural network encoder branch and a Transformer encoder branch; In the Transformer encoder branch, all four Transformer blocks are replaced with Transformer blocks. Before the input image is input into the Transformer encoder branch, it is first divided into several image blocks and flattened into an image sequence. After a linear projection, it is input into the Transformer encoder branch structure composed of four Transformer blocks and three image block merging layers. The Transformer blocks in each layer of the dual encoder are executed three times. (b) Decoder Replace all three transformer blocks in the decoder with the Transformer block, and execute the Transformer block of each layer in the decoder once; The Transformer block includes a high- and low-frequency attention mechanism, a first summation and layer normalization, a local enhancement feedforward network, and a second summation and layer normalization, which are connected in series. In the high- and low-frequency attention mechanism, the standard transform in the Haar wavelet transform is first used, followed by row filtering and then column filtering, to perform a first-level decomposition, resulting in three detail components and one approximate component. Since the image can be considered a discrete function with values ​​ranging from 0 to 1, it can be decomposed by the scaling function. The scaling function, expressed as the sum of products of its coefficients, is: wavelet function Then it can be represented by a scaling function, that is The first-order wavelet decomposition is expressed as Here, C0 is the required approximate component; then, C0 is upsampled to the original image size, and the approximate component is subtracted from the original image to obtain the required detail component. For the obtained approximate component and detail component, a set of query Q, key K, and value V are obtained through linear transformation. The query of the approximate component is obtained from the input. Then, the two different sets of Q, K, and V are calculated through scaled dot product attention to obtain the scaled dot product self-attention result SA of a head. h A linear transformation is applied to the result obtained by concatenating multiple heads to obtain the result of multi-head self-attention; when the feature dimension is N h When, the approximate components are divided into αN h The detail components are divided into (1-α)N h The high-frequency and low-frequency features are obtained respectively, and they are connected in the channel direction as the output. Then, they pass through the residual structure that is first added to the input and then normalized by the layer.

2. The crack image segmentation method based on dual encoders in complex environments according to claim 1, characterized in that, In the convolutional neural network encoder branch of the dual encoder, the initialization layer, the max pooling layer and the four convolutional layers 1-4 in the ResNet-50 network are extracted. The input image first passes through the initialization layer and the max pooling layer, and then passes through the four convolutional layers 1-4 to extract the local features of the image in sequence. The intermediate features of each layer in the convolutional layers 1-4 will be retained and output to the four feature fusion modules, and fused with the intermediate features corresponding to the Transformer encoder branch.

3. The crack image segmentation method based on dual encoders in complex environments according to claim 1, characterized in that, During data loading, the input image is cropped and augmented. The image is then subjected to sequential resizing, random horizontal and vertical flipping, random rotation, and center cropping. During the testing phase, the image is cropped to the specified input size only, and regularization is applied to all three channels of the image.

4. The crack image segmentation method based on dual encoders in complex environments according to claim 2, characterized in that, In the Transformer block, the output of the first summation and layer normalization will be projected into a high-dimensional space through a local enhancement feedforward network and the second summation and layer normalization to learn more abstract features that are difficult to capture in the low-dimensional space, and then projected back into the original space. In the local enhancement feedforward network, the input sequence is first spatially restored to the dimensions of an image; then, the dimensions are increased using depthwise separable convolution and the GELU activation function; in the high-dimensional space, depthwise separable convolution and the GELU activation function are used to capture local features, allowing interaction between blocks and enhancing the network's expressive power; subsequently, the original dimensions are restored using inverted depthwise separable convolution and the GELU activation function, and batch normalization is applied, and the result is flattened into a sequence. Finally, the residual structure, which is added to the input for a second time and normalized by the layer, is used as the output.

5. The crack image segmentation method based on dual encoders in complex environments according to claim 4, characterized in that, The structure of each feature fusion module is as follows: First, the intermediate feature Cv received from the dual encoder input is... i and T i Cv i T is an intermediate feature of the i-th layer of the encoder branch of a convolutional neural network. i These are intermediate features from the i-th layer of the Transformer encoder branch, adjusted using a 1×1 convolution. i The channel dimension is determined; then, channel attention is used to adjust the channel responses of the two features, increasing the weight of favorable channels and reducing the influence of redundant channels, thus obtaining... and Then, and Matrix multiplication is performed to enhance the correlation between the two components, resulting in a correlation-enhanced feature; furthermore... and It also inputs a cross-domain fusion module to fuse information between different domains. The two different features are linearly transformed to obtain two sets of queries, keys and values. Then, each type of query is multi-head attention is calculated with the key and value of the other type. Finally, the two results are connected in the channel direction, and the dimensionality is reduced and effective information is extracted through a 1×1 convolution to obtain the cross-domain fusion feature. Connect the aforementioned correlation enhancement features, cross-domain fusion features, and Fourthly, dimensionality is reduced through inverted depthwise separable convolution, batch normalization, and GELU activation function; multi-class features are fully integrated using depthwise separable convolution, batch normalization, and GELU activation function, as well as 1×1 convolution, batch normalization, and GELU activation function, to extract effective information; finally, the fused feature F of this layer is output. i .

6. The crack image segmentation method based on dual encoders in complex environments according to claim 1, characterized in that, The decoder consists of three cascaded decoder layers and a segmentation head. Each decoder layer consists of subpixel convolutions, inverted depth-separable convolutions, and Transformer blocks along the channel direction. The corresponding feature fusion module is input between the subpixel convolutions and inverted depth-separable convolutions of the decoder layer. The segmentation head consists of subpixel convolutions and 1×1 convolutions.

7. The crack image segmentation method based on dual encoders in complex environments as described in claim 1, characterized in that, The crack image segmentation method uses the sum of binary cross-entropy (BCE) loss and dice loss (Dice) as the loss function during the model training phase, i.e., BCE plus Dice: Where N represents the total number of pixels in the image, t i That is the true category of the pixel, p i ε represents the pixel category predicted by the network, while ε is the smoothing factor.

8. A crack image segmentation system based on dual encoders in complex environments, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the crack image segmentation method based on dual encoders in complex environments as described in any one of claims 1-7 by calling the program instructions.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform a crack image segmentation method based on dual encoders in a complex environment as described in any one of claims 1-7.

Images