Road and bridge deck disease identification method based on deep learning

Abstract

The invention discloses a road and bridge deck disease identification method based on deep learning. The method comprises the following steps: acquiring a road and bridge deck disease image data set; constructing a road and bridge deck disease recognition model, and training the road and bridge deck disease recognition model based on the road and bridge deck disease image data set to obtain a trained road and bridge deck disease recognition model; the road and bridge floor disease identification model is constructed and obtained on the basis of a YOLOv9 network framework in combination with a CARAFE up-sampling module and a Triplet Attention mechanism; and obtaining a to-be-identified road and bridge surface image, and inputting the to-be-identified road and bridge surface image into the trained road and bridge surface disease identification model to obtain a disease identification result of the to-be-identified road and bridge surface image. According to the method provided by the invention, the problem of low recognition precision of an existing algorithm in different application scenes is effectively solved, and multiple disease types in road surface and bridge environments can be effectively and accurately recognized.

Description

Technical Field

[0001] The present invention relates to the field of deep learning combined with road engineering technology, and in particular to a method for identifying road and bridge surface defects based on deep learning. Background Art

[0002] After the highway is built and put into operation, under the influence of long-term repeated automobile loads and external natural environmental factors, especially the increase in traffic volume causing the cumulative axle load to increase, the road is operated under overload, the material properties and structure of the road surface fatigue decay, and finally the performance and safety performance of the road surface are reduced. Roads and bridges provide important guarantees for the operation of urban functions, but each channel has an impact on them. These impacts are common cracks, potholes and other diseases. If not handled in time, it will pose a serious threat to safety. Therefore, the road and bridge surface diseases should be located and repaired in time to maintain a good operating state, which is used to judge the degree of road damage and accurately repair the diseases.

[0003] Traditional detection mainly relies on manual observation and recording. This method relies on inspectors to observe and record the disease conditions manually. As the earliest method, it is easily affected by the subjective judgment of personnel during the detection process, resulting in unstable detection results. When faced with a huge transportation network, manual detection is not only time-consuming and labor-intensive, but also inefficient.

[0004] With the development of science and technology, people have begun to study road disease detection methods based on computer intelligence. These methods are divided into two categories. The first is the traditional detection method based on image processing. This method manually extracts disease features and designs complex rules and processes for detection. Although this method reduces the problems of strong subjectivity and high labor costs of manual detection, its ability to recognize diseases of different sizes and shapes is limited, and the recognition effect is poor under complex backgrounds. For example, edge detection, morphological processing, and feature matching are used for image processing, but these technologies have certain limitations when facing diverse road diseases. The other is a detection method based on deep learning. This method uses neural networks to automatically learn image feature representation and has more powerful feature learning and expression capabilities. Deep learning methods can better adapt to various complex scenes and objects, and can be trained end-to-end on large-scale data sets to improve recognition accuracy and generalization ability. For example, convolutional neural networks (CNNs) perform well in disease detection. They can not only recognize multiple types of diseases, but also maintain efficient detection performance in different environments. Therefore, detection methods based on deep learning have become the current mainstream.

[0005] Previous studies have made significant contributions to the automatic detection of road and bridge surface diseases. However, with the continuous acceleration of urbanization and the increasing development of transportation, the load on roads and bridges has gradually increased, and the diseases have become more and more diverse. Most studies only focus on the single task of long cracks, while ignoring other types of disease conditions, which cannot meet the needs of maintenance and management. Therefore, it is particularly important to conduct a comprehensive study and analysis of different types of road and bridge diseases. At the same time, the data sets used for training in existing studies are shot at an angle perpendicular to the ground, lacking front landscape information that is closer to the real road environment under multi-scenario conditions, which is not conducive to the accurate perception of road conditions by autonomous driving and affects the driving comfort of intelligent networked vehicles.

[0006] Therefore, how to accurately identify various types of defects in a relatively realistic road and bridge surface environment, improve driving comfort and provide timely road maintenance has become a problem that needs to be solved urgently. Summary of the invention

[0007] In view of the shortcomings of the above-mentioned prior art, the present invention provides a road and bridge deck defect identification method based on deep learning, which replaces the bilinear interpolation method with the CARAFE lightweight upsampling operator, captures the cross-dimensional interaction of data through the TripletAttention module, improves the feature representation ability of the model, and introduces the bounding box loss function LIOU, thereby improving the accuracy of road and bridge deck defect type identification.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A road and bridge deck disease identification method based on deep learning includes the following steps:

[0010] Collecting a road and bridge deck damage image dataset, wherein the road and bridge deck damage images include road surface damage images or / and bridge deck damage images for different landscapes, different damage shapes, and different damage sizes;

[0011] A road and bridge surface defect recognition model is constructed, and the road and bridge surface defect recognition model is trained based on a road and bridge surface defect image dataset to obtain a trained road and bridge surface defect recognition model; the road and bridge surface defect recognition model is constructed based on a YOLOv9 network architecture, combined with a CARAFE upsampling module and a TripletAttention mechanism;

[0012] A road bridge surface image to be identified is obtained, and the road bridge surface image to be identified is input into a trained road bridge surface defect identification model to obtain a defect identification result of the road bridge surface image to be identified.

[0013] As a preferred solution, the road and bridge surface disease identification model includes a backbone network, a neck network and a detection head;

[0014] The backbone network includes a convolutional layer, a convolutional fusion unit and three attention fusion feature units connected in sequence, wherein the convolutional fusion unit includes a cascaded convolutional layer and a RepNCSPELAN4 module, and the attention fusion feature unit includes a cascaded convolutional layer, a RepNCSPELAN4 module and a TripletAttention module;

[0015] The neck network includes a pyramid pooling module, two feature fusion upsampling units and two convolutional connection feature fusion units connected in sequence, wherein the feature fusion upsampling unit includes a cascaded CARAFE upsampling module, a connection layer and a RepNCSPELAN4 module, and the convolutional connection feature fusion unit includes a cascaded convolutional layer, a connection layer and a RepNCSPELAN4 module;

[0016] The detection head is used to obtain the road and bridge deck image disease recognition result according to the output of the neck network as the output of the road and bridge deck disease recognition model;

[0017] Among them, the input of the road and bridge surface disease recognition model is used as the input of the backbone network; the output of the RepNCSPELAN4 module in the first attention fusion feature unit and the second attention fusion feature unit in the backbone network are also used as the input of the connection layer in the second feature fusion upsampling unit and the first feature fusion upsampling unit in the neck network respectively, and the output of the RepNCSPELAN4 module in the third attention fusion feature unit in the backbone network is also used as the input of the neck network; the output of the pyramid pooling module in the neck network is also connected to the input of the connection layer in the second convolutional connection feature fusion unit, and the output of the RepNCSPELAN4 module in the first feature fusion upsampling unit in the neck network is also used as the input of the connection layer in the first convolutional connection feature fusion unit.

[0018] As a preferred solution, the road and bridge surface disease recognition model also includes an auxiliary network; the auxiliary network includes a convolution layer, an attention fusion feature unit and three convolution feature fusion extraction units connected in sequence, wherein the convolution feature fusion extraction unit includes a cascaded convolution layer, a fusion layer and a RepNCSPELAN4 module, and the fusion layer is connected to the linear layer;

[0019] Among them, the input of the road and bridge surface disease recognition model is used as the input of the auxiliary network; the output of the RepNCSPELAN4 module in the first attention fusion feature unit in the backbone network passes through the first linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit; the output of the RepNCSPELAN4 module in the second attention fusion feature unit in the backbone network passes through the second linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit and the input of the fusion layer in the second convolutional feature fusion extraction unit respectively; the output of the RepNCSPELAN4 module in the third attention fusion feature unit in the backbone network passes through the third linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit, the input of the fusion layer in the second convolutional feature fusion extraction unit and the input of the fusion layer in the third convolutional feature fusion extraction unit respectively.

[0020] As a preferred solution, the processing process of the CARAFE upsampling module includes:

[0021] For the feature map X with an input shape of H×W×C, the upsampling kernel prediction unit is used to predict the upsampling kernel, and then the upsampling is completed through the feature recombination unit. Finally, the feature map X' with a shape of σH×σW×C is output, where σ is the upsampling ratio.

[0022] As a preferred solution, the upsampling kernel prediction unit includes a channel compression subunit, a content encoding subunit and a kernel normalization subunit;

[0023] The input feature map is reduced by the channel compression subunit, and then the channel-compressed feature map is used as input by the content encoding subunit, using a kernel size of k. up ×k up The convolutional layer generates a reorganized kernel according to the content of the input feature map, and finally uses the kernel normalization subunit to perform a up ×k up The reorganized kernel is normalized spatially using the softmax function to complete the upsampled kernel prediction;

[0024] The upsampling kernel prediction is shown as follows:

[0025]

[0026] In the formula, is the upsampling kernel prediction unit, W l' For X-based l' The domain is the upsampled kernel feature map predicted for each position I, where for any target position I'=(i',j') of the output feature map X', there is a corresponding source position I=(i,j) at the input feature map X, N(Xl' ,K up ) is K centered at position I' up ×K up Sub-region, namely X l' field.

[0027] As a preferred solution, the processing process of the feature recombination unit is:

[0028] Map each pixel position in the output feature map of the upsampling kernel prediction unit back to the input feature map, and use the input feature map to obtain k pixels centered at each pixel position in the output feature map. up ×k up The area of ​​the size is calculated, and the dot product operation is performed with the upsampling kernel prediction unit to predict the upsampling kernel of each pixel position in the output feature map, and finally a feature map with enhanced semantic information is obtained;

[0029] The feature reorganization is shown in the following formula:

[0030]

[0031] Where W l' (n,m) represents the upsampled kernel feature map W output by the sampling kernel prediction unit l' The (n,m) pixel position in the feature map X; X(i+n,j+n) represents the (i+n,j+n) pixel position in the input feature map X; X' represents the semantic information enhanced feature map output by the feature recombination unit; r represents the value limit of the feature recombination pixel position, which is [k up / 2].

[0032] As a preferred solution, the TripletAttention module uses a three-branch structure to capture the interactive relationship between the three dimensions of channel, height and width, calculates the attention weight, and establishes dependencies between different dimensions by rotating the input tensor and applying the residual transformation module.

[0033] As a preferred solution, the residual transformation module includes a Z_pool layer, a convolution layer and a batch normalization layer connected in sequence;

[0034] The processing process of the residual transformation module is as follows: the Z_pool layer reduces the tensor of the channel dimension to 2 dimensions, and then performs convolution processing through a convolution layer with a kernel size of k×k, and finally, after processing through a batch normalization layer, an intermediate output feature map is output, and the intermediate output feature map is constructed with the original input feature map to construct a residual change.

[0035] As a preferred embodiment, the loss function used in the training of the road and bridge deck defect recognition model includes a bounding box loss for describing the degree of match in size between the predicted bounding box of the predicted feature map output by the model and the real bounding box of the training sample road and bridge deck image, and a classification loss for describing the recognition loss between the defect recognition result category of the predicted result output by the model and the real defect category of the training sample road and bridge deck image; wherein the bounding box loss adopts the bounding box loss function LIOU, and the classification loss adopts the binary cross entropy loss function.

[0036] As a preferred solution, the bounding box loss function LIOU is as follows:

[0037]

[0038] Where ρ is the Euclidean distance between the center of the predicted bounding box and the true bounding box, c is the distance between the diagonal of the smallest box enclosing the predicted bounding box and the true bounding box, b and b gt are the center points of the predicted bounding box and the true bounding box, l and l respectively. gt are the lengths of the diagonals of the predicted bounding box and the true bounding box, α is the weight parameter, and v is the aspect ratio of the bounding box.

[0039] Compared with the prior art, the present invention has the following technical effects:

[0040] (1) The present invention replaces the bilinear interpolation method with the CARAFE lightweight upsampling operator and utilizes the content-aware reconstruction feature. CARAFE can aggregate contextual information in a larger receiving field, which not only enhances the semantic information of the feature map, but also improves the detail clarity of the upsampled image, enhances the detail retention capability, and reduces the blur effect and artifact generation.

[0041] (2) The present invention constructs a TripletAttention module through the TripletAttention mechanism, uses the three-branch structure to capture the cross-dimensional interaction of input data, constructs the interdependence between input channels or spatial positions, enhances the feature representation ability of the model, and integrates information on different dimensions by rotating the input tensor and applying residual transformation, so as to better capture the intrinsic characteristics of the input data, enhance the network's understanding and processing capabilities of complex data structures, and solve the imbalance problem of traditional convolutional layers when processing features of different scales.

[0042] (3) The present invention introduces the bounding box loss function LIOU. On the basis of the bounding box loss function in the traditional YOLOv9, the difference in diagonal length between the predicted bounding box and the true bounding box is added. When the aspect ratio is the same, the bounding box loss function LIOU can provide an additional difference measurement, which can better measure the degree of match between the predicted bounding box and the true bounding box in size, thereby improving the accuracy of bounding box determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0044] Figure 1 The present invention discloses a flow chart of a road and bridge deck disease identification method based on deep learning;

[0045] Figure 2 This is a schematic diagram of the structure of the road and bridge surface disease identification model of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the CARAFE upsampling module of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of the TripletAttention mechanism of the present invention; DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0049] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0050] Figure 1 The present invention discloses a method for identifying road and bridge deck defects based on deep learning. Aiming at the problem of low recognition accuracy of various defects in a relatively real road and bridge deck environment, the method comprises the following steps:

[0051] Collecting a road and bridge deck damage image dataset, wherein the road and bridge deck damage images include road surface damage images or / and bridge deck damage images for different landscapes, different damage shapes, and different damage sizes;

[0052] A road and bridge surface defect recognition model is constructed, and the road and bridge surface defect recognition model is trained based on a road and bridge surface defect image dataset to obtain a trained road and bridge surface defect recognition model; the road and bridge surface defect recognition model is constructed based on a YOLOv9 network architecture, combined with a CARAFE upsampling module and a TripletAttention mechanism;

[0053] A road bridge surface image to be identified is obtained, and the road bridge surface image to be identified is input into a trained road bridge surface defect identification model to obtain a defect identification result of the road bridge surface image to be identified.

[0054] In the existing technology, with the continuous acceleration of urbanization and the increasing development of transportation, the load on roads and bridges has gradually increased, and the diseases have become more and more diverse. Most studies only focus on the single task of long cracks, but ignore other types of disease conditions, which cannot meet the needs of maintenance and management. Therefore, it is particularly important to conduct a comprehensive study and analysis of different types of road and bridge diseases. At the same time, the shooting angle of the data set used for training in existing studies is perpendicular to the ground, lacking the front landscape information that is closer to the real road environment under multi-scenario conditions, which is not conducive to the accurate perception of road conditions by autonomous driving and affects the driving comfort of intelligent networked vehicles.

[0055] To solve this problem, the present invention proposes a road and bridge pavement defect recognition method based on deep learning. Aiming at the shortcomings of the bilinear interpolation upsampling method in detail preservation, blurring effect and artifact generation, the present invention proposes a CARAFE upsampling operator, which improves the upsampling effect by utilizing content-aware reorganization features. Secondly, aiming at the shortcomings of the traditional convolutional layer in processing features of different scales, which is unbalanced and unable to pay attention to each important area in a balanced manner, the present invention proposes a TripletAttention module, which uses a three-branch structure to capture the cross-dimensional interaction of input data to improve the feature representation ability of the model. Finally, the present invention introduces a new bounding box loss function LIOU, which enables the model to capture the difference between the predicted bounding box and the real bounding box in a more fine-grained manner during bounding box regression, thereby enhancing the model's stronger discrimination ability when processing similar boxes, overlapping boxes and complex backgrounds.

[0056] The road and bridge deck defect identification method based on deep learning of the present invention is described in more detail below.

[0057] 1. Image preprocessing

[0058] In order to better utilize the method of the present invention to identify road and bridge surface defects, the collected road and bridge surface defect image data set is classified and labeled, and the labeled data set is subjected to data enhancement processing such as random cropping and rotation, thereby improving the generalization ability of the road and bridge surface defect recognition model and making the model more stable in different environments.

[0059] 2. Road and bridge surface disease identification model

[0060] like Figure 2 As shown, the road and bridge surface disease recognition model includes a backbone network, a neck network and a detection head; the backbone network includes sequentially connected convolutional layers, convolutional fusion units and three attention fusion feature units, wherein the convolutional fusion unit includes a cascaded convolutional layer and a RepNCSPELAN4 module, and the attention fusion feature unit includes a cascaded convolutional layer, a RepNCSPELAN4 module and a TripletAttention module; the neck network includes a sequentially connected pyramid pooling module, two feature fusion upsampling units and two convolution connection feature fusion units, wherein the feature fusion upsampling unit includes a cascaded CARAFE upsampling module, a connection layer and a RepNCSPELAN4 module, and the convolution connection feature fusion unit includes a cascaded convolutional layer, a connection layer and a RepNCSPELAN4 module; the detection head is used to detect the damage of the road and bridge surface according to the neck network. The output is the defect recognition result of the road and bridge surface image as the output of the road and bridge surface defect recognition model; wherein, the input of the road and bridge surface defect recognition model is used as the input of the backbone network; the output of the RepNCSPELAN4 module in the first attention fusion feature unit and the second attention fusion feature unit in the backbone network are also used as the input of the connection layer in the second feature fusion upsampling unit and the input of the connection layer in the first feature fusion upsampling unit in the neck network respectively, and the output of the RepNCSPELAN4 module in the third attention fusion feature unit in the backbone network is also used as the input of the neck network; the output of the pyramid pooling module in the neck network is also connected to the input of the connection layer in the second convolution connection feature fusion unit, and the output of the RepNCSPELAN4 module in the first feature fusion upsampling unit in the neck network is also used as the input of the connection layer in the first convolution connection feature fusion unit.

[0061] The road and bridge surface defect recognition model also includes an auxiliary network; the auxiliary network includes a convolution layer, an attention fusion feature unit and three convolution feature fusion extraction units connected in sequence, wherein the convolution feature fusion extraction unit includes a cascaded convolution layer, a fusion layer and a RepNCSPELAN4 module, and the fusion layer is connected to the linear layer; wherein the input of the road and bridge surface defect recognition model is used as the input of the auxiliary network; the output of the RepNCSPELAN4 module in the first attention fusion feature unit in the backbone network is used as the input of the convolution layer in the first convolution feature fusion extraction unit through the first linear layer in the auxiliary network, and the backbone network is used as the input of the convolution layer in the first convolution feature fusion extraction unit. The output of the RepNCSPELAN4 module in the second attention fusion feature unit in the network passes through the second linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit and the input of the fusion layer in the second convolutional feature fusion extraction unit respectively. The output of the RepNCSPELAN4 module in the third attention fusion feature unit in the backbone network passes through the third linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit, the input of the fusion layer in the second convolutional feature fusion extraction unit and the input of the fusion layer in the third convolutional feature fusion extraction unit respectively.

[0062] In order to improve the accuracy of road and bridge pavement identification in various types of disease, the present invention aims at the defects of YOLOv9 in different application scenarios. For the task of road disease identification including the front landscape, the present invention proposes a road and bridge pavement disease recognition model. The CARAFE upsampling module is proposed in the model to improve the upsampling effect by content-aware recombinant features; the TripletAttention mechanism is proposed to improve the feature representation ability of the model; finally, a new bounding box loss function LIOU is designed to capture the difference between the predicted bounding box and the real bounding box in a more fine-grained manner during the model bounding box regression, thereby improving the model's stronger discrimination ability when dealing with similar boxes, overlapping boxes and complex backgrounds.

[0063] The following is a detailed introduction to the CARAFE upsampling module, TripletAttention mechanism and bounding box loss function LIOU.

[0064] 2.1. CARAFE upsampling module

[0065] Considering that the upsampling method currently used is bilinear interpolation, the upsampling kernel is determined only by the spatial position of the pixel point, and the semantic information of the feature map is not used. It can be regarded as a "uniform" upsampling, and the perception domain is usually very small, which will lead to the loss of some image details. Not only that, the upsampling kernel is not calculated by the distance between pixels, but is learned through the network, but the same upsampling kernel is applied to each position of the feature map, which cannot capture the information of the feature map content. And when the upsampling kernel size is large, a large number of parameters and calculations will be introduced.

[0066] To solve the above problems, the present invention adopts the CARAFE operator to replace the bilinear interpolation method, such as Figure 3 As shown in the figure, CARAFE is different from the previous mechanism that only uses sub-pixel neighbors. It can aggregate contextual information in a large receptive field, and it does not use fixed kernels for all samples, but supports instance-specific content-aware processing to generate adaptive convolution kernels in real time. The CARAFE upsampling module is composed of two parts: an upsampling kernel prediction unit and a feature reorganization unit. In the specific processing, the CARAFE upsampling module uses the upsampling kernel prediction unit to predict the upsampling kernel for the feature map X with an input shape of H×W×C, and then completes the upsampling through the feature reorganization unit. Finally, the output is a feature map X' with a shape of σH×σW×C, where σ is the upsampling ratio.

[0067] The upsampling kernel prediction unit includes a channel compression subunit, a content encoding subunit, and a kernel normalization subunit. The channel of the input feature map is reduced by the channel compression subunit, and then the content encoding subunit uses the channel compressed feature map as input, using a kernel size of k. up ×k up The convolutional layer generates a reorganized kernel according to the content of the input feature map, and finally uses the kernel normalization subunit to perform a up ×k up The reorganized kernel is normalized spatially using the softmax function to complete the upsampled kernel prediction.

[0068] The upsampling kernel prediction is shown as follows:

[0069]

[0070] In the formula, is the upsampling kernel prediction unit, W l' For X-based l' The domain is the upsampled kernel feature map predicted for each position I, where for any target position I'=(i',j') of the output feature map X', there is a corresponding source position I=(i,j) at the input feature map X, N(X l' ,Kup ) is K centered at position I' up ×K up Sub-region, namely X l' field.

[0071] The processing process of the feature recombination unit is:

[0072] Map each pixel position in the output feature map of the upsampling kernel prediction unit back to the input feature map, and use the input feature map to obtain k pixels centered at each pixel position in the output feature map. up ×k up The area of ​​the target position l' and the corresponding square area N(X l' ,k up ), the feature reorganization is shown in the following formula:

[0073]

[0074] Where W l' (n,m) represents the upsampled kernel feature map W output by the sampling kernel prediction unit l' The (n,m) pixel position in the feature map X; X(i+n,j+n) represents the (i+n,j+n) pixel position in the input feature map X; X' represents the semantic information enhanced feature map output by the feature recombination unit; r represents the value limit of the feature recombination pixel position, which is [k up / 2].

[0075] Among them, different channels at the same position share the same upsampling kernel, N(X l' ,k up ) region contributes differently to the upsampled pixel l', based on the content of the feature rather than the distance of the location.

[0076] The present invention replaces the bilinear interpolation method with the CARAFE lightweight upsampling operator and utilizes the content-aware reconstruction feature. CARAFE can aggregate contextual information in a larger receiving domain, which not only enhances the semantic information of the feature map, but also improves the detail clarity of the upsampled image, enhances the detail retention capability, and reduces the blurring effect and artifacts.

[0077] 2.2 TripletAttention Mechanism

[0078] In network models, downsampling operations are usually performed through maximum pooling or average pooling to reduce the number of parameters and reduce the complexity of convolutional network calculations, allowing the model to see a wider range of input information. However, the downsampling module will also lead to the loss of some important information, especially for small-sized targets, downsampling may make their features less obvious, increase the difficulty of detection, and affect the expressiveness of the model. The traditional method of calculating channel attention involves calculating a singular weight, usually a scalar for each channel in the input tensor, and then uniformly scaling these feature maps using the singular weights. However, in order to calculate the singular weights of the channel, the input tensor space is decomposed into a pixel for each channel through global average pooling. This leads to a significant loss of spatial information, so when calculating the attention on these single-pixel channels, the interdependence between the channel dimension and the spatial dimension is missing.

[0079] The present invention is directed to the above-mentioned problems. Figure 4 As shown in the figure, a TripletAttention module that applies the TripletAttention mechanism is proposed. The three-branch structure is used to capture the interaction between the three dimensions of channel, height and width, and the attention weight is calculated. The dependency between different dimensions is established by rotating the input tensor and applying the residual transformation module.

[0080] The residual transformation module includes a Z_pool layer, a convolution layer, and a batch normalization layer connected in sequence, wherein the Z_pool layer reduces the tensor of the channel dimension to 2 dimensions, and then performs convolution processing through a convolution layer with a kernel size of k×k, and finally, after processing through a batch normalization layer, an intermediate output feature map is output, and the intermediate output feature map is constructed with the original input feature map to form a residual change.

[0081] In the Z_pool layer, the average pooled features and the maximum pooled features on the C-dimensional Tensor are concatenated by reducing it to 2 dimensions. This enables the layer to retain the rich representation of the actual tensor while reducing its depth to make further computations lighter, as shown in the following formula:

[0082] Z_pool(x)=[MaxPool 0d (x),AvgPool 0d (x)];

[0083] Where 0d is the 0th dimension where the maximum pooling operation and the average pooling operation occur. For example, a tensor of shape Z_pool(C×H×W) produces a tensor of shape (2×H×W).

[0084] Then, the intermediate output is obtained through a standard convolutional layer with a kernel size of k×k and a batch normalization layer, and a residual connection is constructed with the input feature. From the perspective of forward and backward information propagation, assuming that the input and output dimensions of the nonlinear unit of the neural network are consistent, the function H(·) to be fitted in the neural network unit can be split into two parts, as shown in the following formula:

[0085] Z (l) =H(a (l-1) )=a (l-1) +F(a (l-1) );

[0086] In the formula, F(·) is the residual function. At the high level of the network, an identity mapping H(a (l-1) )→a (l-1) , consider a neural network composed of residual blocks, assuming that no activation function is used, that is, a (l) =Z (l) , there are any two layers l2>l1, recursively expand to get the following formula:

[0087]

[0088] In this way, the gradient of the final loss ∈ to a certain low-level output can be expanded to obtain the following formula:

[0089]

[0090] During back propagation, the gradient of the loss output is decomposed into two terms, so that the error signal can be directly propagated to the lower layers without any intermediate weight matrix transformation, which can alleviate the gradient diffusion problem to a certain extent.

[0091] The three structures are introduced in detail below.

[0092] In the first branch, an interaction is established between the H dimension and the W dimension, similar to the channel attention module used in CBAM, by using the following formula:

[0093] ω=σ(W1ReLU(W0g(x))+W1ReLU(W0δ(x)));

[0094] Where ω is the learned channel attention weight and is applied to the input X, ω∈R C×1×1 , σ is the sigmoid function, W0 and W1 are weight matrices, shared in CBAM, g(x) is the global average pooling (GAP) function, and δ(χ) is the global maximum pooling (GMP) function;

[0095] The global average pooling (GAP) function g(x) is shown below:

[0096]

[0097] The global maximum pooling (GMP) function δ(χ) is shown below:

[0098]

[0099] In the second branch, an interaction is established between the C dimension and the W dimension, and the input tensor is rotated 90 degrees counterclockwise along the W axis. The shape of this rotated tensor is (H×C×W). After the Residual Transformation, the final output is rotated 90 degrees clockwise along the W axis to keep the same shape as the input.

[0100] In the last branch, an interaction is established between the C latitude and the W latitude by rotating the input tensor 90 degrees counterclockwise along the H axis. The shape of this rotated tensor is (W×H×C). After the Residual Transformation, the final output is rotated 90 degrees clockwise along the H axis to keep the same shape as the input.

[0101] Finally, the three branches are aggregated together. This structure can integrate information from different dimensions through different rotation and permutation operations, better capture the intrinsic characteristics of the data, and enhance the network's ability to understand and process complex data structures.

[0102] The three branches are aggregated as shown in the following formula:

[0103]

[0104] Where ψ1, ψ2 and ψ3 are the three branches of triple attention, and σ(·) are the three cross-dimensional weights calculated in the attention.

[0105] By utilizing the three-branch structure and different rotation and permutation operations, it is possible to integrate information from different dimensions, better capture the intrinsic characteristics of the data, enhance the network's understanding and processing capabilities of complex data structures, and solve the imbalance problem of traditional convolutional layers when processing features of different scales.

[0106] 2.3 Loss function for training the road and bridge surface disease identification model

[0107] The loss functions used in the training of the road and bridge deck defect recognition model include the bounding box loss used to describe the degree of match in size between the predicted bounding box of the predicted feature map output by the model and the real bounding box of the training sample road and bridge deck image, and the classification loss used to describe the recognition loss between the defect recognition result category of the predicted result output by the model and the real defect category of the training sample road and bridge deck image.

[0108] In YOLOv9, the bounding box loss function uses CIOU, which takes into account the overlapping area, aspect ratio and center point distance. However, if there are two predicted bounding boxes with the same aspect ratio, and their midpoints have the same distance to the center of the bounding box, but the physical distance is still different; secondly, if the aspect ratio of the predicted bounding box and the true bounding box is the same, then the aspect ratio penalty term is always 0, which is obviously unreasonable.

[0109] In view of the above problems, in order to accurately determine the size of the bounding box, the loss function must be able to uniquely represent the position and size of the box. Aiming at the bounding box loss in the training of the road and bridge surface defect recognition model, the present invention proposes a bounding box loss function LIOU. On the basis of the original penalty, the difference in diagonal length between the predicted bounding box and the true bounding box is added. Even when the aspect ratio is the same, this item can provide an additional difference measurement, which can better measure the matching degree of the predicted bounding box and the true bounding box in size.

[0110] The bounding box loss function LIOU is as follows:

[0111]

[0112] Where ρ is the Euclidean distance between the center of the predicted bounding box and the true bounding box, c is the distance between the diagonal of the smallest box enclosing the predicted bounding box and the true bounding box, b and b gt are the center points of the predicted bounding box and the true bounding box, l and l respectively. gt are the lengths of the diagonals of the predicted bounding box and the true bounding box, α is the weight parameter, and v is the aspect ratio of the bounding box.

[0113] For the classification loss in the training of the road and bridge surface disease recognition model, the binary cross entropy loss function BCE-LOSS can be used, which is conducive to training convergence and obtaining better classification results.

[0114] 3. Example

[0115] In order to further verify the effectiveness of the road and bridge deck disease recognition model proposed in this embodiment, some mainstream target detection algorithms, such as Yolov3-tiny, Yolov5x, Yolov7, Faster R-CNN, Mask R-CNN, etc., are compared. Experiments are carried out using this dataset under the same environment. The experimental results are shown in Table 1.

[0116] Table 1 Experimental results of different target detection algorithms

[0117]

[0118] As can be seen from Table 1, the road and bridge surface defect recognition model proposed in this embodiment is superior to other target detection models in terms of detection accuracy, recall rate, F1 score and other indicators. The Precision value of the road and bridge surface disease recognition model reached 0.751, which was 1 percentage point higher than the baseline model, and 0.3, 19.7, 4.2, 3.5, 13.1, 22.2, and 27.2 percentage points higher than Yolov3-tiny, Yolov5x, Yolov7, Yolov8n, SSD, Faster R-CNN, and Mask R-CNN algorithms, respectively. This shows that the model of this embodiment has a low error rate when predicting the positive class. The Recall value reached 0.691, which was 2.2 percentage points higher than the baseline model, and 4.2, 17.8, 3.1, 20.4, 16.6, 20.8, and 25.04 percentage points higher than Yolov3-tiny, Yolov5x, Yolov7, Yolov8n, SSD, Faster R-CNN, and Mask R-CNN algorithms, respectively. This shows that the model of this embodiment can identify more real positive objects and reduce missed detections. The Score value reached 0.7137, which was 1.06 percentage points higher than the baseline model, and 1.87, 18.1, 3.01, 21.26, 14.51, 20.87, and 25.21 percentage points higher than Yolov3-tiny, Yolov5x, Yolov7, Yolov8n, SSD, Faster R-CNN, and Mask R-CNN algorithms, respectively. This shows that the model in this embodiment has achieved a good balance between precision and recall. The Map@0.5 value reached 0.757, which was 1.7 percentage points higher than the baseline model, and 1.87, 18.1, 3.01, 21.26, 14.51, 20.87, and 25.21 percentage points higher than Yolov3-tiny, Yolov5x, Yolov7, Yolov8n, SSD, Faster R-CNN, and Mask R-CNN algorithms, respectively. Algorithms such as R-CNN and MaskR-CNN improved by 4.1, 24.8, 4.02, 26.7, 22.5, 28.9, and 34.5 percentage points, respectively, indicating that the model in this embodiment has good detection performance under different categories and different IoU thresholds.

[0119] 4. Overview

[0120] In summary, the method of the present invention has the following technical advantages:

[0121] (1) The present invention replaces the bilinear interpolation method with the CARAFE lightweight upsampling operator and utilizes the content-aware reconstruction feature. CARAFE can aggregate contextual information in a larger receiving field, which not only enhances the semantic information of the feature map, but also improves the detail clarity of the upsampled image, enhances the detail retention capability, and reduces the blur effect and artifact generation.

[0122] (2) The present invention constructs a TripletAttention module through the TripletAttention mechanism, uses the three-branch structure to capture the cross-dimensional interaction of input data, constructs the interdependence between input channels or spatial positions, enhances the feature representation ability of the model, and integrates information on different dimensions by rotating the input tensor and applying residual transformation, so as to better capture the intrinsic characteristics of the input data, enhance the network's understanding and processing capabilities of complex data structures, and solve the imbalance problem of traditional convolutional layers when processing features of different scales.

[0123] (3) The present invention introduces the bounding box loss function LIOU. On the basis of the bounding box loss function in the traditional YOLOv9, the difference in diagonal length between the predicted bounding box and the true bounding box is added. When the aspect ratio is the same, the bounding box loss function LIOU can provide an additional difference measurement, which can better measure the degree of match between the predicted bounding box and the true bounding box in size, thereby improving the accuracy of bounding box determination.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A road and bridge deck disease identification method based on deep learning, characterized in that: The steps include: Collecting a road and bridge deck damage image dataset, wherein the road and bridge deck damage images include road surface damage images or / and bridge deck damage images for different landscapes, different damage shapes, and different damage sizes; A road and bridge surface defect recognition model is constructed, and the road and bridge surface defect recognition model is trained based on a road and bridge surface defect image dataset to obtain a trained road and bridge surface defect recognition model; the road and bridge surface defect recognition model is constructed based on a YOLOv9 network architecture, combined with a CARAFE upsampling module and a TripletAttention mechanism; A road bridge surface image to be identified is obtained, and the road bridge surface image to be identified is input into a trained road bridge surface defect identification model to obtain a defect identification result of the road bridge surface image to be identified.

2. The method for identifying road and bridge surface defects based on deep learning according to claim 1, characterized in that: The road and bridge surface disease recognition model includes a backbone network, a neck network and a detection head; The backbone network includes a convolutional layer, a convolutional fusion unit and three attention fusion feature units connected in sequence, wherein the convolutional fusion unit includes a cascaded convolutional layer and a RepNCSPELAN4 module, and the attention fusion feature unit includes a cascaded convolutional layer, a RepNCSPELAN4 module and a TripletAttention module; The neck network includes a pyramid pooling module, two feature fusion upsampling units and two convolutional connection feature fusion units connected in sequence, wherein the feature fusion upsampling unit includes a cascaded CARAFE upsampling module, a connection layer and a RepNCSPELAN4 module, and the convolutional connection feature fusion unit includes a cascaded convolutional layer, a connection layer and a RepNCSPELAN4 module; The detection head is used to obtain the road and bridge deck image disease recognition result according to the output of the neck network as the output of the road and bridge deck disease recognition model; Among them, the input of the road and bridge surface disease recognition model is used as the input of the backbone network; the output of the RepNCSPELAN4 module in the first attention fusion feature unit and the second attention fusion feature unit in the backbone network are also used as the input of the connection layer in the second feature fusion upsampling unit and the first feature fusion upsampling unit in the neck network respectively, and the output of the RepNCSPELAN4 module in the third attention fusion feature unit in the backbone network is also used as the input of the neck network; the output of the pyramid pooling module in the neck network is also connected to the input of the connection layer in the second convolutional connection feature fusion unit, and the output of the RepNCSPELAN4 module in the first feature fusion upsampling unit in the neck network is also used as the input of the connection layer in the first convolutional connection feature fusion unit.

3. The method for identifying road and bridge surface defects based on deep learning according to claim 2 is characterized in that: The road and bridge surface disease recognition model also includes an auxiliary network; the auxiliary network includes a convolution layer, an attention fusion feature unit and three convolution feature fusion extraction units connected in sequence, wherein the convolution feature fusion extraction unit includes a cascaded convolution layer, a fusion layer and a RepNCSPELAN4 module, and the fusion layer is connected to the linear layer; Among them, the input of the road and bridge surface disease recognition model is used as the input of the auxiliary network; the output of the RepNCSPELAN4 module in the first attention fusion feature unit in the backbone network passes through the first linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit; the output of the RepNCSPELAN4 module in the second attention fusion feature unit in the backbone network passes through the second linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit and the input of the fusion layer in the second convolutional feature fusion extraction unit respectively; the output of the RepNCSPELAN4 module in the third attention fusion feature unit in the backbone network passes through the third linear layer in the auxiliary network and is used as the input of the convolutional layer in the first convolutional feature fusion extraction unit, the input of the fusion layer in the second convolutional feature fusion extraction unit and the input of the fusion layer in the third convolutional feature fusion extraction unit respectively.

4. The method for identifying road and bridge surface defects based on deep learning according to claim 2 is characterized in that: The processing process of the CARAFE upsampling module includes: For the feature map X with an input shape of H×W×C, the upsampling kernel prediction unit is used to predict the upsampling kernel, and then the upsampling is completed through the feature recombination unit. Finally, the feature map X' with a shape of σH×σW×C is output, where σ is the upsampling ratio.

5. The method for identifying road and bridge surface defects based on deep learning according to claim 4 is characterized in that: The upsampling kernel prediction unit includes a channel compression subunit, a content encoding subunit and a kernel normalization subunit; The input feature map is reduced by the channel compression subunit, and then the channel-compressed feature map is used as input by the content encoding subunit, using a kernel size of k. up ×k up The convolutional layer generates a reorganized kernel according to the content of the input feature map, and finally uses the kernel normalization subunit to perform a up ×k up The reorganized kernel is normalized spatially using the softmax function to complete the upsampled kernel prediction; The upsampling kernel prediction is shown as follows: In the formula, is the upsampling kernel prediction unit, W l' For X-based l' The domain is the upsampled kernel feature map predicted for each position I, where for any target position I'=(i',j') of the output feature map X', there is a corresponding source position I=(i,j) at the input feature map X, N(X l' ,K up ) is K centered at position I' up ×K up Sub-region, namely X l' field.

6. The method for identifying road and bridge surface defects based on deep learning according to claim 4 is characterized in that: The processing process of the feature recombination unit is: Map each pixel position in the output feature map of the upsampling kernel prediction unit back to the input feature map, and use the input feature map to obtain k pixels centered at each pixel position in the output feature map. up ×k up The area of ​​the size is calculated, and the dot product operation is performed with the upsampling kernel prediction unit to predict the upsampling kernel of each pixel position in the output feature map, and finally a feature map with enhanced semantic information is obtained; The feature reorganization is shown in the following formula: Where W l' (n,m) represents the upsampled kernel feature map W output by the sampling kernel prediction unit l' The (n,m) pixel position in the feature map X; X(i+n,j+n) represents the (i+n,j+n) pixel position in the input feature map X; X' represents the semantic information enhanced feature map output by the feature recombination unit; r represents the value limit of the feature recombination pixel position, which is [k up / 2].

7. The method for identifying road and bridge surface defects based on deep learning according to claim 2, characterized in that: The TripletAttention module uses a three-branch structure to capture the interactive relationship between the three dimensions of channel, height and width, calculates the attention weight, and establishes dependencies between different dimensions by rotating the input tensor and applying the residual transformation module.

8. The method for identifying road and bridge surface defects based on deep learning according to claim 7, characterized in that: The residual transformation module includes a Z_pool layer, a convolution layer and a batch normalization layer connected in sequence; The processing process of the residual transformation module is as follows: the Z_pool layer reduces the tensor of the channel dimension to 2 dimensions, and then performs convolution processing through a convolution layer with a kernel size of k×k, and finally, after processing through a batch normalization layer, an intermediate output feature map is output, and the intermediate output feature map is constructed with the original input feature map to construct a residual change.

9. The method for identifying road and bridge surface defects based on deep learning according to claim 1, characterized in that: The loss function used in the training of the road and bridge deck defect recognition model includes a bounding box loss used to describe the degree of match in size between the predicted bounding box of the predicted feature map output by the model and the real bounding box of the training sample road and bridge deck image, and a classification loss used to describe the recognition loss between the defect recognition result category of the prediction result output by the model and the real defect category of the training sample road and bridge deck image; wherein the bounding box loss adopts the bounding box loss function LIOU, and the classification loss adopts the binary cross entropy loss function.

10. The method for identifying road and bridge surface defects based on deep learning according to claim 9, characterized in that: The bounding box loss function LIOU is as follows: Where ρ is the Euclidean distance between the center of the predicted bounding box and the true bounding box, c is the distance between the diagonal of the smallest box enclosing the predicted bounding box and the true bounding box, b and b gt are the center points of the predicted bounding box and the true bounding box, l and l respectively. gt are the lengths of the diagonals of the predicted bounding box and the true bounding box, α is the weight parameter, and v is the aspect ratio of the bounding box.

Images