Bridge deck data cleaning method and system based on deep learning

Abstract

The present invention provides a bridge deck data cleaning method and system based on deep learning. The cleaning method uses a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions to train data in a database of bridge pavement. The specific steps are as follows: Step 1: Use a drone equipped with a high-definition camera to photograph the bridge pavement. The captured images are divided into three categories: the bridge deck, other bridge structures other than the bridge deck structure, and non-bridge structures; a database of bridge pavement is constructed using all the captured images; Step 2: Construct a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions, using the framework of the Resnet and Transformer models; Step 3: Use the database constructed in Step 1 to train the model, and use the trained model for cleaning and classification of bridge deck data. By adding a MultKAN module with multiplication and activation functions to the Resnet structure and Transformer, the method can solve the problems of inaccurate data cleaning and classification for bridge pavement, poor timeliness, and huge workload in traditional data processing methods.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge deck data cleaning, and in particular to a bridge deck data cleaning method and system based on deep learning. Background Art

[0002] With the rapid development of the modern economy, bridges have become crucial transportation infrastructure, generating vast amounts of data during their design, construction, operation, and maintenance. This data includes, but is not limited to, bridge pavement data, as well as other bridge structures and images unrelated to bridges. Due to the sheer volume, diversity, and complexity of this data, traditional manual processing methods often struggle to quickly identify specific bridge components, consuming significant manpower and resources. Therefore, cleaning and classifying databases of bridge pavement data is crucial.

[0003] Traditional manual processing methods are difficult for bridge and pavement databases, significantly impacting defect detection. However, with the continued maturity of deep learning network model technology, bridge and pavement database cleaning and classification are developing in this direction. Therefore, combining deep learning-based network models with bridge and pavement data cleaning to quickly and accurately clean and classify bridge deck data is a pressing technical challenge. Summary of the Invention

[0004] The purpose of the present invention is to provide a bridge deck data cleaning method and system based on deep learning, which can quickly and accurately clean and classify bridge pavement data.

[0005] To achieve the above object, the technical solution of the present invention is:

[0006] In a first aspect, the present invention provides a bridge deck data cleaning method based on deep learning. The cleaning method uses a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions to train data in a bridge pavement database. The specific steps are:

[0007] Step 1: Use a drone equipped with a high-definition camera to photograph the bridge pavement. During the construction process, the drone's position constantly changes, and the drone shoots in the height direction of the object being photographed. The captured images will contain three categories: the first is the bridge deck, the second is other bridge structures except the bridge deck structure, and the third is non-bridge structures. A database of bridge pavement is constructed using all the captured images.

[0008] Step 2: Build a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions:

[0009] The Resnet-MultKAN-Transformer model based on multiplication and ReLU functions adopts the framework of Resnet and Transformer models, including a 7×7 convolutional layer, a 3×3 maximum pooling downsampling layer, 3 ReLU-MultKAN modules, 6 3×3 convolutional layers, a ReLU-MultKAN-Transformer converter, an average pooling layer and a fully connected layer;

[0010] The ReLU-MultKAN-Transformer converter includes a position encoding module, a ReLU-MultKAN-Transformer encoder and a ReLU-MultKAN-Transformer decoder, both of which are provided with a ReLU-MultKAN module;

[0011] The ReLU-MultKAN module includes a 1×1 convolutional layer, a batch normalization layer, a ReLU-MultKAN structure, a 3×3 convolutional layer, a batch normalization layer, and a 1×1 convolutional layer connected in sequence;

[0012] In the ReLU-MultKAN structure, each layer has a series of nodes, and multiplication operations are performed on some nodes using the activation function ω(x), while identity transformation operations are performed on other nodes. The activation function ω(x) is the product of the basic function b(x) and the function R i The sum of (x) is expressed as:

[0013] ω(x)=w(b(x)+R i (x)),

[0014] Wherein, w is used to control the overall size of the activation function; the basis function b(x) is well known in the art;

[0015] Function R i (x) is expressed as:

[0016]

[0017] Where, ReLU(x)=max(0,x); s i and e i is the domain of the independent variable x of the function to be fitted, that is, x∈[s i , e i ];

[0018] Step 3: Use the database constructed in step 1 to train the Resnet-MultKAN-Transformer model based on multiplication and ReLU functions, and use the trained Resnet-MultKAN-Transformer model based on multiplication and ReLU functions for cleaning and classification of bridge deck data.

[0019] Furthermore, the cleaning and classification results are divided into three categories of images, namely, the bridge deck structure, other bridge structures other than the bridge deck structure, and non-bridge structures; and according to the classified data images, they are classified as positive samples and negative samples;

[0020] Data images of bridge deck structures classified as bridges are positive samples;

[0021] Data images classified as other bridge structures except bridge deck structures and non-bridge structures are negative samples.

[0022] Furthermore, during the training process, the learning rate is set to: The formula for the change of the learning rate of the i-th generation is:

[0023] Lr=V min +(LV min )*(1+cos(π*Le / T max )) / 2.

[0024] Where Lr is the result of learning rate; T max is the total number of training cycles, used to define the period of the cosine function; V min is the minimum value of the learning rate; L e is the current number of training cycles; L is the basic learning rate;

[0025] The loss function is the weighted cross entropy loss function Loss, and the formula is:

[0026]

[0027] Where z is the true distribution of each pixel, is the output distribution of the model.

[0028] Furthermore, in the ReLU-MultKAN-Transformer encoder, the feature sequence is input into the multi-head mask layer together with the output of the position encoding module, and then input into the ReLU-MultKAN module. After passing through the ReLU-MultKAN module, its result is residually connected with the feature sequence input into the ReLU-MultKAN-Transformer encoder, and the result is output to the normalization layer of the lower layer, and then input into the ReLU-MultKAN feedforward network module. The output of the ReLU-MultKAN feedforward network module is then residually connected with the input, and the result of the residual connection is then input into the normalization layer to obtain the output of the ReLU-MultKAN-Transformer encoder.

[0029] The output of the ReLU-MultKAN-Transformer encoder is input into the first multi-head mask layer in the ReLU-MultKAN-Transformer decoder together with the positional encoding output by the positional encoding module, and then input into the ReLU-MultKAN module of the lower layer. The output of the ReLU-MultKAN module is residually connected with the output of the ReLU-MultKAN-Transformer encoder and then input into the normalization layer. The output of the normalization layer is then input into the second multi-head mask layer together with the positional encoding, and then input into the ReLU-MultKAN module. Its output is residually connected with the feature sequence of the second multi-head mask layer, and then input into the normalization layer of the lower layer. Then, it is input into the ReLU-MultKAN feedforward network module. The output of the ReLU-MultKAN feedforward network module is residually connected with the input, and the result is output into the normalization layer to obtain the output of the ReLU-MultKAN-Transformer converter.

[0030] Furthermore, the ReLU-MultKAN feedforward network module includes a reshaping and upgrading module, a ReLU-MultKAN module and a CBL module.

[0031] In a second aspect, the present invention provides a bridge deck data cleaning system based on deep learning, which performs the above steps.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention creatively adds a MultKAN module with multiplication and activation functions to the Resnet structure and Transformer in the Resnet-MultKAN-Transformer model, constructing a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions. This model can solve the problems of inaccurate data cleaning and classification for bridge and pavement, poor timeliness, and huge workload in traditional data processing methods.

[0034] The cleaning method of the present invention is used for cleaning and classifying bridge and pavement data images, and is particularly suitable for cleaning and classifying image data of large data sets (with tens of thousands or hundreds of thousands of images). The effect is more obvious. For the processing of large amounts of data in the bridge engineering construction cycle, the resource consumption is reduced, the classification speed is accelerated, and the accuracy of the classification results is improved. It can also accurately and efficiently complete the cleaning and classification problems of the bridge and pavement database, providing accurate protection for the subsequent use of the bridge and pavement database.

[0035] The present invention defines a ReLU function to improve the MultKAN structure, thereby accelerating the progress of model training, reducing the complexity of cleaning and classification of the bridge pavement database, reducing the amount of data, and ensuring the accuracy of data for defect detection and analysis of bridge pavements. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 2 is a schematic structural diagram of a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions in one embodiment of the present invention.

[0037] Figure 2 It is a structural diagram of the ReLU-MultKAN-Transformer converter in the present invention.

[0038] Figure 3 It is a structural diagram of the ReLU-MultKAN structure in the present invention.

[0039] Figure 4 It is a structural diagram of the ReLU-MultKAN module in the present invention.

[0040] Figure 5 Schematic diagram of the structure of the ReLU-MultKAN feedforward network module (ReLU-MultKAN-FFN) in the present invention. DETAILED DESCRIPTION

[0041] The present invention is further explained below with reference to the embodiments and drawings, but they are not intended to limit the scope of protection of the present application.

[0042] Example 1

[0043] The bridge deck data cleaning method based on deep learning in this embodiment is as follows: obtaining data from a bridge pavement database, training the data in the bridge pavement database using a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions to obtain appropriate parameter information, and performing cleaning and classification using the trained model to obtain classification results. The method specifically includes the following steps:

[0044] Step 1: Get the data from the bridge pavement database:

[0045] A drone equipped with a high-definition camera is used to photograph the road surface of a bridge. There is a certain distance between the drone and the road surface of the bridge being photographed. The position of the drone is constantly changing during the construction of the bridge. The drone shoots in the direction of the front height of the object being photographed. The captured images will contain three categories: the first is the bridge deck, the second is other bridge structures except the bridge deck structure, and the third is non-bridge structures.

[0046] The camera in this embodiment uses a high-precision camera. The drone is kept 20 meters from the bridge surface, capturing the bridge deck, other structures, and non-bridge structures. The drone takes images every 10 minutes and stores them in memory, capturing 48 images per day. The construction period for this bridge in this embodiment is approximately three months, resulting in a total of 12,960 images, all of which are 512×512 pixel PNG images.

[0047] The captured images were randomly shuffled and divided into a training set and a validation set in a ratio of 8:2, with the number of images in each set being 10,368 and 2,592 respectively.

[0048] Step 2: Build a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions:

[0049] The Resnet-MultKAN-Transformer model based on multiplication and ReLU functions adopts the framework of Resnet and Transformer models. The overall structure is as follows Figure 1 As shown, it includes 14 modules, namely 7×7 convolution layer, 3×3 maximum pooling downsampling layer, 3 ReLU-MultKAN modules, 6 3×3 convolution layers, ReLU-multiplication Kolmogorov-Arnold transformer (ReLU-MultKAN-Transformer transformer), average pooling layer, and fully connected layer.

[0050] The ReLU-MultKAN module mainly includes a 1×1 convolution layer, a batch normalization layer, a ReLU-MultKAN structure, a 3×3 convolution layer, a batch normalization layer and a 1×1 convolution layer.

[0051] The image is input into the 7×7 convolutional layer of the first layer, then into the 3×3 maximum pooling downsampling layer of the second layer, and then into the ReLU-MultKAN module of the third layer.

[0052] In the ReLU-MultKAN module of the third layer, the feature sequence is first input into the 1×1 convolution layer, then into the batch normalization layer, followed by the ReLU-MultKAN structure, the 3×3 convolution layer, the batch normalization layer and the 1×1 convolution layer. Then the output of the 1×1 convolution layer is residually connected to the feature sequence of the input ReLU-MultKAN module, and then the result is output to the 3×3 convolution layer of the fourth layer. Then its output is output to the 3×3 convolution layer of the fifth layer, and its output is residually connected to the feature sequence of the input ReLU-MultKAN module of the third layer, and then input to the ReLU-MultKAN module of the sixth layer. The output of the eLU-MultKAN module is input into the 3×3 convolutional layer of the seventh layer, and then the output of this layer is input into the 3×3 convolutional layer of the eighth layer. Then its output is residually connected with the feature sequence of the ReLU-MultKAN module input into the sixth layer. The result of the connection is input into the ReLU-MultKAN module of the ninth layer. The feature sequence of this module is input into the 3×3 convolutional layer of the tenth layer, and then its output is input into the 3×3 convolutional layer of the eleventh layer. Then its output is residually connected with the input of the ReLU-MultKAN module of the ninth layer, and the result of the connection is input into the ReLU-Multiplication Kolmogorov-Arnold Converter of the twelfth layer.

[0053] The ReLU-MultKAN-Transformer converter includes a position encoding module, a ReLU-MultKAN-Transformer encoder and a ReLU-MultKAN-Transformer decoder.

[0054] In the ReLU-MultKAN-Transformer encoder, the feature sequence is input into the multi-head mask layer together with the output of the position encoding module, and then input into the ReLU-MultKAN module. After passing through the ReLU-MultKAN module, its result is residually connected with the feature sequence input into the ReLU-MultKAN-Transformer encoder, and the result is output to the normalization layer of the lower layer, and then input into the ReLU-MultKAN feedforward network module. The output and input of the ReLU-MultKAN feedforward network module are residually connected, and the result of the residual connection is input into the normalization layer to obtain the output of the ReLU-MultKAN-Transformer encoder.

[0055] The output of the ReLU-MultKAN-Transformer encoder is input into the first multi-head mask layer in the ReLU-MultKAN-Transformer decoder together with the positional encoding output of the positional encoding module (the positional encoding module here is the same as the positional encoding module in the encoder module), and then input into the ReLU-MultKAN module in the lower layer. The output of the ReLU-MultKAN module is residually connected with the output of the ReLU-MultKAN-Transformer encoder, and the result is input into the normalization layer, and then The output of the normalization layer and the position code are input into the second multi-head mask layer together, and then into the ReLU-MultKAN module. The output and the feature sequence of the second multi-head mask layer are residually connected, and the result is input into the normalization layer of the lower layer, and then input into the ReLU-MultKAN feedforward network module. The output and input of the ReLU-MultKAN feedforward network module are residually connected, and the result is output into the normalization layer, and then input into the average pooling layer of the thirteenth layer. Finally, the output of the average pooling layer of the thirteenth layer is input into the fully connected layer of the fourteenth layer, and finally the classification result is output.

[0056] Step 3: Use the database constructed in step 1 to train the Resnet-MultKAN-Transformer model based on multiplication and ReLU functions, and use the trained Resnet-MultKAN-Transformer model based on multiplication and ReLU functions for cleaning and classification of bridge deck data.

[0057] First, we randomly initialized the hyperparameters (initial weights, learning rate, etc.) of the Resnet-MultKAN-Transformer model, which uses multiplication and ReLU functions, and trained the model for 50 epochs using the training set. After each epoch, we validated the model using the validation set to understand the generalization ability of the deep learning model and prevent overfitting.

[0058] The learning rate is set as follows: a cosine decay strategy is used, and a smooth adjustment strategy is used during training, that is, the learning rate is gradually adjusted to improve the convergence and performance of the model. The formula for the change of the learning rate for the i-th generation is as follows:

[0059] Lr=V min +(LV min )*(1+cos(π*Le / T max )) / 2.

[0060] Where Lr is the result of learning rate. max is the total number of training cycles (epochs), which is used to define the period of the cosine function. In this embodiment, T max =50, V min is the minimum learning rate. During training, the learning rate will not fall below this value. Le is the current training cycle number. L is a basic learning rate.

[0061] Secondly, the loss function is used to adjust the parameter model. The loss function of the model of the present invention is the weighted cross entropy loss function Loss, and the formula is as follows:

[0062]

[0063] Where z is the true distribution of each pixel, is the output distribution of the model.

[0064] Set up a visualization chart to display the learning rate and loss function curves in real time. During training, the current accuracy and time consumption are output after each epoch. By observing the changing trends of the curves, analyze whether the model convergence and accuracy meet the expected requirements. If the model fails to converge or the accuracy is low after convergence, adjust the hyperparameters. When the loss function value is less than 0.001, it indicates that the model has reached convergence and training stops.

[0065] Data images from a database of classified bridge pavement to be cleaned were input into a trained Resnet-MultKAN-Transformer model based on multiplication and ReLU functions for recognition. Three categories of images were obtained: bridge deck structure, other bridge structures other than the bridge deck structure, and non-bridge structures. Based on these classifications, the images were classified as positive or negative samples.

[0066] The data images of the bridge deck structure classified as a bridge are positive samples and can be used as data in the database for defect detection of the bridge deck.

[0067] Data images classified as other bridge structures and non-bridge structures other than bridge deck structures are negative samples and are needed for other detections.

[0068] Through the data cleaning method of the present invention, the collected image data can be quickly cleaned and classified, and the classified data can be used to establish and update the bridge pavement data set in the future. After this classification processing, different samples can be labeled to facilitate later maintenance, verification and inspection, etc.

[0069] Example 2

[0070] The structures of the various parts of the Resnet-MultKAN-Transformer model based on multiplication and ReLU functions in this embodiment are as follows:

[0071] First, image I0 is fed into the 7×7 convolutional block of the first layer. After the convolution operation of this module, it is passed through the 3×3 max pooling downsampling layer of the second layer. The output feature sequence I1 after the 3×3 max pooling downsampling layer is 1 / 4 of the size of the original image. It is then fed into the ReLU-MultKAN module of the third layer.

[0072] Next, the 1 / 4-sized feature sequence is fed into the third-layer ReLU-MultKAN module. Within the ReLU-MultKAN module, it is first fed into the first 1×1 convolutional layer. After convolution, it is fed into the batch normalization layer, which calculates the mean and variance of the previous layer's output and uses these statistics to normalize the output of that layer. This layer also reduces internal covariate shift, stabilizing the model's training process and allowing it to converge to the optimal solution more quickly. This is then fed into the lower-layer ReLU-MultKAN structure, where each layer has a series of nodes with a specific set of functions within them to process the input data and output it to the next layer.

[0073] The standard KAN layer Φ is expressed as:

[0074]

[0075] Among them, φ l,1,1 (·) represents a one-dimensional function, l represents the depth, n l belongs to the integer array [n1, n2, ..., n l ], x l Represents the input vector, and The first KAN layer has n l Input dimensions and n l+1 Output dimension, the input vector x l Convert to x l+1 .

[0076] The entire KAN(x) network is composed of standard KAN layers, namely

[0077]

[0078] Among them, Φ l is a standard KAN layer. The number of addition and multiplication operations in layer l are expressed as and in Φ l Accepts an input vector And Φ l Convert to output z l ,Right now

[0079]

[0080] Multiplication layer M l Including identity transformation operation and multiplication operation, for some child nodes z l Use the activation function ω(x) to perform a multiplication operation (such as Figure 3 The black multiplication sign in the figure shows the same transformation operation is performed on the other child nodes ( Figure 3 The black solid circle in the middle layer). The multiplication layer M of the ReLU-MultKAN structure l The activation function ω(x) of the node in the multiplication operation is the basis function b(x) and the function R i The sum of (x), ω(x) can be expressed as:

[0081] ω(x)=w(b(x)+R i (x)),

[0082] Among them, w is used to control the overall size of the activation function. The basic function b(x) is expressed as:

[0083]

[0084] Function Ri (x) is expressed as:

[0085]

[0086] Where ReLU(x)=max(0,x). i and e i is the domain of the independent variable x of the function to be fitted, that is, x∈[s i , e i ].

[0087] Then the ReLU-MultKAN layer represents the multiplication layer M l Compared with the standard KAN layer Φ l The calculation can be concisely expressed as follows:

[0088]

[0089] The entire ReLU-MultKAN structure is expressed as follows:

[0090]

[0091] The present invention introduces the ReLU-MultKAN module into the model, which can achieve performance comparable to or even better than that of traditional convolutional networks with fewer parameters, has a small memory requirement, and has a learnable nonlinear activation function. It can significantly improve computational efficiency and can flexibly adapt to different application scenarios. Here, the original B-spline function is replaced by the function R i (x), so that the model can fully utilize the parallel capabilities of the GPU, reduce the time of model training, and reduce resource consumption. The ReLU-MultKAN structure can fit the nonlinear transformation that best suits the data at each degree. Each input variable is embedded through a set of independent one-dimensional nonlinear learnable functions φ p，q (x i ) processing, where p represents the index of the input variable and q represents the index of the output dimension. The ReLU-MultKAN structure is based on the Kolmogorov-Arnold representation theorem, which states that a multivariate continuous function can be represented as a finite combination of single-variable continuous functions, which can be expressed as:

[0092]

[0093] Among them, φ q，p (x p) is a one-dimensional function, p is the index of the input dimension, q is the index of the output dimension, and n is the number of parameters referenced. The activation functions within the network nodes of the ReLU-MultKAN structure avoid errors in sensitive data input, thereby reducing the model's operations such as convolution and pooling on sensitive data. This allows the ReLU-MultKAN structure to further improve the accuracy of the output image and enable the ReLU-MultKAN structure to implement complex multidimensional input to multidimensional output mapping at each layer.

[0094] After passing through the ReLU-MultKAN structure, the feature sequence will be input into the 3×3 convolutional layer of the lower layer. After the convolution operation, it will be input into the batch normalization layer of the lower layer. After the batch normalization operation, it will be input into the 1×1 convolutional layer. Its output will be residually connected with the input of the ReLU-MultKAN module. The result of this residual connection is the output of the ReLU-MultKAN module.

[0095] Then the output of the ReLU-MultKAN module in the third layer is input into the 3×3 convolution layer in the fourth layer. The output of this layer is input into the 3×3 convolution layer in the fifth layer. After the convolution operation in this layer, the feature sequence at this time is I2, and its size is 1 / 8 of the original image. Then the feature sequence is residually connected with the input of the ReLU-MultKAN module in the third layer, and the result of the connection is output to the ReLU-MultKAN module in the sixth layer. It is then input into the 3×3 convolution layers in the seventh and eighth layers. After the feature sequence after the eighth 3×3 convolution layer, it is residually connected with the feature sequence I2 input into the ReLU-MultKAN module in the sixth layer. The result of the connection is I3, and the size is 1 / 16 of the original image. Next, I3 is fed into the ReLU-MultKAN module in the ninth layer, and then into the 3×3 convolutional layer in the tenth and eleventh layers. After the convolution, the output of the feature sequence is residually connected with the feature sequence I3. The result of the connection is I4, which is 1 / 32 the size of the original image. Feature sequence I4 is then fed into the ReLU-MultKAN-Transformer in the twelfth layer.

[0096] Before entering the ReLU multiplication Kolmogorov-Arnold encoder, feature sequence I4 undergoes an embedding operation, which maps feature sequence I4 into a vector. This embedding operation consists of two parts. The first is the input embedding operation, which maps feature sequence I4 into a vector a. In this case, the size of vector a is 1 / 32 of the original image. The second part is the positional encoding operation, which generates a set of vectors b with the same dimensions as the vector after the input embedding operation, used to provide position information. The encoding rules for the positional encoding operation are as follows:

[0097]

[0098]

[0099] Among them, PE represents the position encoder, pos x,y Indicates pos x or POS y ,(pos x ,pos y ) is a position in the feature sequence, and i represents the dimension of the position. x By introducing two encoding formulas, we can calculate a 16-dimensional vector representing pos x Position encoding; pos y Substitute the last two encoding formulas and we can also calculate a 16-dimensional vector, representing pos y Position encoding; by concatenating these two vectors, we can get a vector with the same dimension as the input feature sequence, representing (pos x ,pos y ) positional encoding. By calculating all the positional encodings, we get a vector with the same dimension as the input feature sequence, which represents the positional encoding of this batch.

[0100] Next, we need to add vectors a and b to get vector c, which is the key K and query Q in the multi-head mask layer in the ReLU-Multiplication Kolmogorov-Arnold (ReLU-MultKAN-Transformer) encoder. This multi-head mask layer can focus on different parts of the input vector c at the same time and map the query, key, and value to multiple different linear spaces. Vector a also serves as the value V of the multi-head mask layer. For each input set (query, key, value) of vector c and vector a, four steps are performed. First, the query Q, key K, and value V are obtained, which can be expressed as follows:

[0101] Q=W q ×query,

[0102] K=W k ×key,

[0103] V=W v ×value.

[0104] Among them, W q , W k and W v is the weight matrix, corresponding to different heads.

[0105] The second step is for each head to calculate the similarity between the query and the key and use it as the attention score. It can be expressed as:

[0106]

[0107] Among them, Attention_scores[i] represents the attention score of the i-th position in the feature sequence, softmax is the function, Q i represents the query vector at position i in the feature sequence, K i Represents the key vector of the i-th position in the feature sequence, d_k represents the dimension of the key vector. The dimension under the square root is divided here to stabilize the gradient and prevent the value from being too large.

[0108] The third step is to apply the attention score to the Value. Each head has its own score, so we sum each head, which can be expressed as:

[0109] Attention_output[i]=Attention_scores[i]×V,

[0110] Among them, Attention_output[i] represents the weighted summation result of the i-th position, and V represents the value Value.

[0111] Finally, the results from all heads are concatenated and integrated into the final output vector through another linear transformation. The multi-head mask layer mechanism captures richer dependencies. Each head processes a portion of the input vector c and vector a. Combining the results from all heads enhances understanding of the global context. The final output is combined with the input vector a to form the feature sequence I5, which is then fed into the ReLU-MultKAN module.

[0112] In the ReLU-MultKAN module, feature sequence I5 undergoes a 1×1 convolutional layer, a batch normalization layer, a ReLU-MultKAN structure, a 3×3 convolutional layer, a batch normalization layer, and a 1×1 convolutional layer. The output is then residually connected to feature sequence I5. The resulting residual connection is then applied to feature sequence I4 and fed into the normalization layer below. In this layer, the feature sequence prevents vanishing gradients and improves model stability and generalization. The output of the normalization layer, I6, is fed into the ReLU-MultKAN feed-forward network module below.

[0113] The ReLU-MultKAN feedforward network module includes a reshape and upgrade module, a ReLU-MultKAN module, and a CBL module. First, the reshape and upgrade module reshapes the feature sequence I6 and refines it. After that, it passes through a ReLU-MultKAN module, performs further convolution operations on the feature sequence I6, and finally enters the CBL module. The CBL module contains three convolution blocks (Cnovolution Block), where the convolution kernel size of the first two convolution blocks is 3×3, and the convolution kernel size of the third convolution block is 1×1. The CBL module is expressed in mathematical formula as follows:

[0114]

[0115] Among them, d1 is the output of the ReLU-MultKAN module in the ReLU-MultKAN feedforward network module, is the output of the first convolutional block in the CBL module, is the output of the second convolutional block in the CBL module, C h1 and C h2 are the hidden layers of the first and second convolution blocks, G is the third convolution block, Leaky_ReLU is the Leaky_ReLU activation function, Batch is the batch normalization operation, and C b The output of the ReLU-MultKAN feedforward network module is residually connected to the feature sequence I6, and the result of the residual connection is input to the normalization layer in the next layer. After the normalization layer, the output is the feature sequence I7, which is then input to the ReLU-MultKAN-Transformer decoder in the next layer.

[0116] Before inputting the multi-head mask layer, the feature sequence I7 undergoes an embedding operation, mapping the output feature sequence I7 to a vector d. At this time, the size of vector d is 1 / 32 of the original image. This is followed by a position encoding operation to generate a vector e. Next, vector d and vector e need to be added together to obtain vector f, which is the key K and query Q in the multi-head mask layer of the first layer in the ReLU-Kolmogorov-Arnold decoder. The multi-head mask layer maps the query, key, and value to multiple different linear spaces. Vector d also serves as the value V of the multi-head mask layer. For each set of inputs (query, key, value) of vector f and vector d, the same four-step operation is performed as in the multi-head mask layer in the encoder. Finally, the output result and the input vector d are merged into the feature sequence I8 and output to the ReLU-MultKAN module. After feature sequence I8 undergoes the 1×1 convolutional layer, batch normalization layer, ReLU-MultKAN structure, 3×3 convolutional layer, batch normalization layer, and 1×1 convolutional layer in the ReLU-MultKAN module, a new feature sequence is generated. This new feature sequence is then residually connected with feature sequence I7. The result of the residual connection is input to the normalization layer in the next layer. After normalization, a new feature sequence I9 is ​​generated. Before being input to the multi-head masking layer in the next layer, feature sequence I9 undergoes an embedding operation, mapping the output feature sequence I9 into a vector g. The size of vector g is 1 / 32 of the original image. Next, a positional encoding operation is performed to generate vector h. Next, vectors g and h are added to obtain vector k, which represents the key K and query Q in the multi-head masking layer in the fourth layer of the ReLU-multiplication Kolmogorov-Arnold decoder. This multi-head masking layer maps the query, key, and value into multiple different linear spaces. Vector g also serves as the value V in the multi-head masking layer. For each input set (query, key, value) of vector k and vector g, the same four-step operation is performed as the first multi-head mask layer in the decoder, and the final output result is merged with the input vector g into the feature sequence I 10 . Then the characteristic sequence I 10 The ReLU-MultKAN module will be input. After the operation of this module, a new feature sequence will be generated. This feature sequence will be residually connected with the feature series I9. The result of the residual connection will be input to the normalization layer of the lower layer for normalization operation to generate a new feature sequence I 11 . Then the characteristic sequence I 11 Input the ReLU-MultKAN feedforward network module of the lower layer. The sequence will undergo the operations of the reshaping and upgrading module, the ReLU-MultKAN module and the CBL module to generate a new feature sequence. The sequence is finally input into the normalization layer of the lower layer for normalization operation to generate a new feature sequence I 12 .

[0117] Feature Sequence I 12 The input is fed into the average pooling layer, which averages the feature maps of each channel into a single value, resulting in a one-dimensional vector with the same length as the number of channels. This layer reduces the dimensionality of the feature map while preserving important feature information in the image. The one-dimensional vector output by the pooling layer is then fed into the fully connected layer below. The weight matrix of this layer maps the one-dimensional vector into the category space, outputting the classified image.

[0118] The present invention adopts the framework of Resnet and Transformer, and uses the ReLU-MultKAN module therein, which can not only avoid the input of sensitive data, but also further improve the accuracy of the output classification image. The ReLU-MultKAN module is added to the traditional Resnet and Transformer model, which not only replaces the B-spline function in the MultKAN structural point with the ReLU activation function, so that the model can better and faster capture relevant features in the image, speeding up the training time of the model, but also emphasizes local attention, which helps to maintain the model's attention to the feature details in the image, dynamically switches between local and global information, increases the model's adaptability to different features of the input image, helps to improve the model's extraction of features in complex images, enhances the model's ability to understand features in images, and also adjusts the self-attention mechanism in the model, helping the model to focus more effectively on key areas in the image, improves the model's accuracy in extracting image features, and enhances the model's ability to classify images.

[0119] Table 1 below compares the results of training different models using the example dataset. F1 represents an indicator for evaluating model classification performance, which is the harmonic mean of precision and recall. T refers to the time it takes for the model to be trained.

[0120] Table 1

[0121]

[0122] The Resnet-MultKAN-Transformer model based on multiplication and ReLU functions of the present invention aims to solve the problems of existing image classification models in image classification, such as long training time, large number of required parameters, high computational complexity, unintuitive feature fusion, strong data dependence, poor interpretability, and high hardware requirements. By appropriately adding a method with a multiplication structure and ReLU activation function, the strong expressiveness and interpretability of the ReLU-MultKAN structure can be combined with the feature extraction capabilities of Resnet and Transformer, thereby improving the accuracy of image classification. At the same time, the residual structure and self-attention mechanism of the two can be combined with the recognition-related feature capability of the ReLU-MultKAN structure, optimizing the feature extraction and prediction process, reducing the model training time, improving the model convergence speed and the efficiency of input image classification, and improving the adaptability and precise fitting of the model. At the same time, the generalization ability of the model is improved, and the accuracy of data cleaning and classification of bridge pavement database is improved.

[0123] The present invention adopts the Resnet-MultKAN-Transformer model based on multiplication and ReLU functions, and adds the ReLU-MultKAN module, which greatly improves the accuracy of image feature extraction and classification, ensures the accuracy and timeliness of image feature extraction, and makes the model perform better in image classification tasks. Due to the innovative ReLU-MultKAN structure, the convergence speed and fitting ability of the model are significantly improved, which helps the model achieve the best effect more quickly and can better capture the key features in the image, thereby improving the speed and accuracy of image classification. It can clean and classify the database of bridge pavement in a timely and accurate manner, greatly reducing the time for data cleaning and classification of the bridge pavement database, while ensuring the accuracy of the data for crack detection on the bridge pavement.

[0124] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A bridge deck data cleaning method based on deep learning, characterized in that: The cleaning method uses the Resnet-MultKAN-Transformer model based on multiplication and ReLU functions to train the data in the bridge pavement database. The specific steps are: Step 1: Use a drone equipped with a high-definition camera to photograph the bridge pavement. During the construction process, the drone's position constantly changes, and the drone shoots in the direction of the front height of the object being photographed. The captured images are divided into three types: the first is the bridge deck, the second is other bridge structures except the bridge deck structure, and the third is non-bridge structures. A database of bridge pavement is constructed using all the captured images. Step 2: Build a Resnet-MultKAN-Transformer model based on multiplication and ReLU functions: The Resnet-MultKAN-Transformer model based on multiplication and ReLU functions adopts the framework of Resnet and Transformer models, including a 7×7 convolutional layer, a 3×3 maximum pooling downsampling layer, 3 ReLU-MultKAN modules, 6 3×3 convolutional layers, a ReLU-MultKAN-Transformer converter, an average pooling layer and a fully connected layer; The ReLU-MultKAN-Transformer converter includes a position encoding module, a ReLU-MultKAN-Transformer encoder and a ReLU-MultKAN-Transformer decoder, both of which are provided with a ReLU-MultKAN module; The ReLU-MultKAN module includes a 1×1 convolutional layer, a batch normalization layer, a ReLU-MultKAN structure, a 3×3 convolutional layer, a batch normalization layer, and a 1×1 convolutional layer connected in sequence; In the ReLU-MultKAN structure, each layer has a series of nodes, and some nodes are activated using the activation function ω. x Perform a multiplication operation, and perform an identity transformation operation on another part of the nodes, the activation function ω x is the basis function b(x) and function R i The sum of (x) is expressed as: ω x =w(b(x)+R i (x)), Among them, w is used to control the overall size of the activation function; Function R i (x) is expressed as: Among them, ReLU(x)=max(0,x);s i and e i is the domain of the independent variable x of the function to be fitted, that is, x∈[s i ,e i ]; Step 3: Use the database constructed in step 1 to train the Resnet-MultKAN-Transformer model based on multiplication and ReLU functions, and use the trained Resnet-MultKAN-Transformer model based on multiplication and ReLU functions for cleaning and classification of bridge deck data.

2. The cleaning method according to claim 1, wherein The cleaning and classification results are divided into three categories of images, namely, the bridge deck structure, other bridge structures other than the bridge deck structure, and non-bridge structures; and according to the classified data images, they are classified as positive samples and negative samples; Data images of bridge deck structures classified as bridges are positive samples; Data images classified as other bridge structures except bridge deck structures and non-bridge structures are negative samples.

3. The cleaning method according to claim 1, wherein During the training process, the learning rate is set to: The formula for the change of the learning rate of the i-th generation is: Lr6V min +(LV min )*(f+cos(π*Le / T max )) / 2 Among them, Lr is the result of learning rate; T max is the total number of training cycles, used to define the period of the cosine function; V min is the minimum value of the learning rate; Le is the current number of training cycles; L is the basic learning rate; The loss function is the weighted cross entropy loss function Loss, and the formula is: Where z is the true distribution of each pixel, is the output distribution of the model.

4. The cleaning method according to claim 1, wherein In the ReLU-MultKAN-Transformer encoder, the feature sequence is input into the multi-head mask layer together with the output of the position encoding module, and then into the ReLU-MultKAN module. After passing through the ReLU-MultKAN module, the result is residually connected with the feature sequence input into the ReLU-MultKAN-Transformer encoder, and the result is output to the normalization layer of the lower layer, and then input into the ReLU-MultKAN feedforward network module. The output of the ReLU-MultKAN feedforward network module is residually connected with the input, and the result of the residual connection is input into the normalization layer to obtain the output of the ReLU-MultKAN-Transformer encoder. The output of the ReLU-MultKAN-Transformer encoder is input into the first multi-head mask layer in the ReLU-MultKAN-Transformer decoder together with the positional encoding output by the positional encoding module, and then input into the ReLU-MultKAN module of the lower layer. The output of the ReLU-MultKAN module is residually connected with the output of the ReLU-MultKAN-Transformer encoder and then input into the normalization layer. The output of the normalization layer is then input into the second multi-head mask layer together with the positional encoding, and then input into the ReLU-MultKAN module. Its output is residually connected with the feature sequence of the second multi-head mask layer, and then input into the normalization layer of the lower layer. Then, it is input into the ReLU-MultKAN feedforward network module. The output of the ReLU-MultKAN feedforward network module is residually connected with the input, and the result is output into the normalization layer to obtain the output of the ReLU-MultKAN-Transformer converter.

5. The cleaning method according to claim 4, wherein: The ReLU-MultKAN feedforward network module includes a reshaping and upgrading module, a ReLU-MultKAN module and a CBL module.

6. A bridge deck data cleaning system based on deep learning, characterized in that: The system executes the steps of the deep learning-based bridge deck data cleaning method described in any one of claims 1-5.

Images