A bridge concrete crack detection method and device based on lightweight Transformer

Abstract

The present invention discloses a bridge concrete crack detection method and device based on a lightweight Transformer, comprising: using a drone to collect bridge concrete crack images and transmitting the collected data back; preprocessing the images and converting them into a fixed input format to establish a data set; constructing a lightweight Transformer network, training and testing the images to obtain a recognition model, wherein the lightweight Transformer adopts a two-stage patch transformation method, discarding part of the input sequence of the Transformer during training, alleviating the overfitting problem and enhancing the diversity between patches; deploying the recognition model to carry out recognition of the surface image of the bridge to be identified. The present invention reduces the computational load of the bridge concrete crack detection method through the proposed lightweight Transformer, so that it can be deployed on a platform with limited computing resources.

Description

Technical Field

[0001] The present invention belongs to the field of crack detection and computer vision, and specifically relates to a bridge concrete crack detection method and device based on a lightweight Transformer. Background Art

[0002] Cracks are one of the common surface damages of concrete. The formation of cracks is usually related to natural disasters, long-term daily overload use, structural corrosion and other factors. Cracks that appear on the surface of structures such as bridges will make the structure fragile and reduce its load capacity, thereby affecting its service life and traffic safety. Once the surface structure collapses, it will cause significant loss of life and property. Therefore, the sooner crack detection and repair work are carried out in a timely manner, the sooner these safety issues can be eliminated.

[0003] Early crack detection mainly relies on human eye recognition, which is greatly affected by the external environment and has low efficiency and accuracy. With the application of non-destructive testing methods such as optics and acoustics and the update and upgrade of various monitoring equipment, the accuracy and speed of crack detection are constantly improving. Computer vision technology uses non-contact means such as cameras to collect images of the surface of the structure and perform automated analysis. It has the advantages of high accuracy, high flexibility, and low cost, and has been increasingly widely used in the field of crack detection. Among them, the crack detection method based on deep learning uses deep learning methods such as convolutional neural networks to construct a deep model suitable for crack detection, which effectively solves the problem of low recognition accuracy and poor image quality affecting the effect.

[0004] The emerging Vision Transformer model abandons the convolutional architecture by using a complete attention mechanism, achieving performance that is equal to or even exceeds that of convolutional neural networks. However, the Vision Transformer is a data-hungry model that requires a sufficiently large data set to demonstrate its superior performance, resulting in high requirements for computing resources. In addition, it lacks some of the inductive biases inherent in convolutional neural networks, so it cannot generalize well when trained in the absence of sufficient data, limiting its ability to be applied in the field of crack detection. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing bridge concrete crack detection method requires too much computing resources, and to provide a bridge concrete crack detection method and device based on a lightweight Transformer.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] On the one hand, a bridge concrete crack detection method based on a lightweight Transformer is provided, comprising:

[0008] Step S1: Use a drone to collect images of bridge concrete cracks and transmit the collected data back; the drone first performs flight preparation, including weather detection, flight plan design, and pre-flight testing, then takes off to perform the image collection task, returns after completion, and checks the collected data; if the image quality is unqualified, it is necessary to make up for the shooting area corresponding to the image;

[0009] Step S2: pre-process the image and convert it into a fixed input format to establish a data set; import the images taken by the drone into the local computer, perform image annotation, image noise reduction and quality enhancement, geometric correction, and auxiliary information extraction; then select a certain number of images with and without cracks to establish a data set of bridge surface images;

[0010] Step S3: Construct a lightweight Transformer network, train and test the image to obtain a recognition model; the network includes two parts: patch-based segmentation and Transformer with sequence pooling, wherein the patch-based segmentation includes block segmentation, two-stage Patch transformation and linear projection, and the Transformer with sequence pooling includes Transformer encoder, sequence pooling layer and linear layer; position encoding needs to be inserted between the two parts; the bridge surface image needs to be segmented into 8*8 patches before inputting the network, and then a two-stage Patch transformation is performed, and the input sequence of the Transformer encoder is formed after linear projection, wherein the embedding dimension of each patch is 256; the Transformer encoder has a total of 6 blocks, each of which includes a multi-head attention layer (4 heads) and a position-based feedforward network (FFN), wherein the first layer dimension of the FFN is 512, the second layer dimension is 256, and the activation function is GELU; the outputs of the multi-head attention layer and the feedforward network are respectively sent to the "add and The first layer is a two-layer perceptron, the first layer has an output dimension of 1024, and the second layer is used for classification, with an output dimension of 2, representing images without cracks and images with cracks.

[0011] Step S4: Deploy the recognition model to carry out recognition of the bridge surface image to be recognized; integrate the trained lightweight Transformer model into the bridge concrete crack detection software; the user transfers the image to be recognized to the local computer and uses the software to detect cracks; during detection, the user can upload a single or multiple pictures, the software calls the model for recognition, and returns the recognition results to the software interface.

[0012] Optionally, in a possible implementation, the two-stage Patch transformation method is as follows:

[0013] In the first stage, grid-shaped Patch dropping is used. Among all the 8*8 patches generated by the image, a certain number of patches are discarded at intervals in a grid shape. Let Patches be a Patch matrix with m rows and n columns. The formula for grid-shaped Patch dropping is as follows:

[0014] Drop=Reshape(Cycle(0,1,p))

[0015] Patches=Patches*Drop

[0016] First, calculate the Drop matrix, which is a binary matrix containing only 0 and 1. 0 represents the patch to be discarded, and 1 represents the patch to be retained. The Cycle function is used to generate an alternating vector of 0 and 1, where p = m*n represents the total number of patches. Then, take a continuous vector of length p from the alternating matrix and reconstruct it into a matrix of the same size as Patches. Multiply Patches by the Drop matrix to obtain the final retained Patches matrix.

[0017] In the second stage, the patches discarded in the first stage are converted into 4*4 thumbnails and randomly mixed with the retained input patches. The formula is as follows:

[0018] x′=Mx 1 +Φ(T(x 2 ))

[0019] where x′ is a new mixed sample (Patch), M is a binary mask indicating the area of ​​the patch to be cropped and retained, and Φ represents a padding operation that generates a 2 For patches of the same size, T(x 2 ) is a thumbnail operation;

[0020] The two-stage patch transformation method discards part of the input sequence of the Transformer during training, forcing the Transformer to use an incomplete sequence for training. This method is not used during testing, and the complete input sequence is used for calculation.

[0021] On the other hand, a bridge concrete crack detection device based on a lightweight Transformer is provided, comprising:

[0022] The acquisition module uses drones to collect images of bridge concrete cracks and transmits the collected data back;

[0023] The dataset module preprocesses the images and converts them into a fixed input format to establish a dataset.

[0024] The training module builds a lightweight Transformer network to train and test images to obtain a recognition model;

[0025] The evaluation module deploys the recognition model and carries out the recognition of the surface images of the bridge to be identified.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] In the two-stage Patch transformation method used in model training, the first stage of transformation alleviates the problem of Transformer overfitting for small data sets and improves the operating efficiency of Transformer. In the second stage of transformation, the patches deleted in the first stage are used as thumbnails and randomly overlaid on the retained patches, which enhances the diversity between patches. The thumbnails of patches with cracks can enhance the model's learning of subtle cracks, and the thumbnails of patches without cracks can be used as occlusions to improve the generalization ability of the model. In addition, the lightweight Transformer reduces the computational load of the bridge concrete crack detection method, allowing it to be deployed on platforms with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the bridge concrete crack detection method based on lightweight Transformer;

[0029] Figure 2 Flowchart for drone data collection;

[0030] Figure 3 This is the architecture diagram of the lightweight Transformer;

[0031] Figure 4 This is a flow chart of the bridge concrete crack detection device based on lightweight Transformer;

[0032] Figure 5 The operation process of the bridge concrete crack detection software;

[0033] Figure 6 It is the main interface of the bridge concrete crack detection software;

[0034] Figure 7 It is the recognition interface of bridge concrete crack detection software;

[0035] Figure 8 This is the recognition result diagram of the bridge concrete crack detection software. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application are described below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0037] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0038] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] The present invention provides a bridge concrete crack detection method based on a lightweight Transformer, such as Figure 1 As shown in the flowchart, the specific process includes:

[0041] Step S1: Use a drone to collect images of bridge concrete cracks and transmit the collected data back;

[0042] The flowchart of step S1 is as follows Figure 2 As shown, including:

[0043] S101, UAV flight preparation;

[0044] Conduct weather detection, flight plan design, and pre-flight tests. Weather detection is to grasp the specific weather conditions, observe light, air visibility, etc., to prevent the impact of strong winds, cloudy weather, etc. on the drone. Flight plan design includes planning the flight route, altitude, flight speed, overlap, and finding a suitable take-off point. Before the official flight, a pre-flight test should be conducted to check whether the various functions of the drone are operating normally and whether it can fully respond to operating instructions, and record the weather conditions of the day, take-off and landing locations, and other information for future reference and analysis.

[0045] S102, the UAV performs flight and collects images;

[0046] Two technicians are required for drone flight, a pilot and an observer. The pilot controls the movement of the aircraft, and the observer observes the flight status of the drone and collects data. During takeoff, the pilot controls the aircraft in real time through the remote control, and the observer observes the aircraft status through the parameters transmitted back by the aircraft. According to the specific flight situation, the manual mode can be switched at any time to adjust the aircraft status to ensure flight safety. The drone flies along the inspected surface of the bridge and completes the image capture of the structural surfaces such as the bottom surface, column surface and beam of the bridge.

[0047] S103, UAV return and data inspection;

[0048] After completing the flight mission, the drone will land at the predetermined location. During landing, the landing location should be safe to avoid unauthorized persons approaching. In case of special circumstances, another suitable and safe location should be selected for landing. After landing, the image data can be viewed online through the application software installed on the mobile terminal (such as a mobile phone or tablet) that matches the drone to check whether the image data and other related information in the drone's onboard camera are complete. If the image quality is not up to standard, it is necessary to re-fly the area it has captured.

[0049] Step S2: preprocess the image and convert it into a fixed input format to establish a data set;

[0050] S201, image export and processing;

[0051] Take out the memory card carried by the drone, put it into the card reader, and import the photos taken by the drone into the local computer. First, process the captured pictures, including image annotation (with cracks and without cracks), image noise reduction and quality enhancement, geometric correction, and auxiliary information extraction. If the quality of the processed pictures is still low, such as blurry pictures, exposure errors, etc., these pictures can be deleted.

[0052] S202, data set construction;

[0053] Establish a data set of bridge surface images, select a certain number of pictures with cracks and without cracks, and try to keep the two types of pictures equal. Then filter out pictures with too high overlap. If the overlap of two pictures reaches 90%, the two pictures are considered to be highly similar, and one of them should be selected to join the data set. Convert all the two types of pictures into RGB images in PNG format and put them into two folders to complete the data set construction. In order to ensure the accuracy of the training model, the number of pictures in the data set should be greater than 500.

[0054] Step S3: Build a lightweight Transformer network, train and test the image to obtain a recognition model;

[0055] S301, lightweight Transformer network;

[0056] The structure of the lightweight neural network is as follows Figure 3 , including two parts: patch-based tokenization and Transformer with sequence pooling, wherein the patch-based tokenization includes patching, two-stage patch transformation and linear projection, and the Transformer with sequence pooling includes a Transformer encoder, a sequence pooling layer and a linear layer; positional encoding needs to be inserted between the two parts;

[0057] The block division is to divide the input image into 8*8 patches; the two-stage patch transformation is a method used in the model training stage, which transforms the input patch during training, discards part of the input information, and allows the Transformer to use an incomplete input sequence for training, and use the complete input sequence for calculation during inference. The first stage of transformation is a patch-level transformation, using a grid-like patch dropping, and let Patches be an m-row n-column patch matrix (usually m=n). The grid-like patch dropping formula is as follows:

[0058] Drop=Reshape(Cycle(0,1,p))

[0059] Patches=Patches*Drop

[0060] First, calculate the Drop matrix, which is a binary matrix containing only 0 and 1, where 0 represents the patch to be discarded and 1 represents the patch to be retained. The Cycle function is used to generate an alternating vector of 0 and 1, where p = m*n represents the total number of patches. Then, a continuous vector of length p is taken from the alternating matrix, and finally reconstructed into a matrix of the same size as Patches. Multiply Patches by the Drop matrix to get the retained patches.

[0061] The second stage of transformation is the transformation within the patch. The patches discarded in the first stage are converted into 4*4 thumbnails, which are randomly mixed with the remaining input patches. Suppose two samples x 1 and x 2 , use thumbnails To replace x 1 A random area of ​​​​the thumbnail T(x 2 ) is usually set to half the width and height of the original patch. The transformation formula is as follows:

[0062] x′=Mx 1 +Φ(T(x 2 ))

[0063] where x′ is the mixed sample, M is a binary mask indicating the area of ​​the patch to be cropped and retained, and Φ is a padding operation that generates a 2 To obtain the binary mask M, we need to 1 The bounding box coordinates of the upper cropping area B = (r x ,r y ,r w ,r h ) to sample, and then remove x 1 area B in the image and take a thumbnail T(x 2 ) is filled. The coordinates of the bounding box are uniformly sampled as shown in the following formula:

[0064] r x ~Uniform(0,W),r w =w

[0065] r y ~Uniform(0,H),r h =h

[0066] Where W and H are the width and height of the original patch, respectively. This hybrid strategy enables the network to learn the same image at different scales.

[0067] After the two-stage patch transformation, the input sequence of the Transformer encoder is formed by linear projection, and the embedding dimension of each patch is 256.

[0068] The Transformer encoder has a total of 6 blocks, each of which includes a multi-head attention layer (Multi-headAttention Layers, 4 heads) and a position-based feedforward neural network (FFN). The first layer dimension of the FFN is 512, the second layer dimension is 256, and the activation function is GELU. The outputs of the multi-head attention layer and the feedforward network are sent to the "addand norm" layer for processing, which contains a residual structure and layer normalization; the sequence pooling layer aggregates the output of the Transformer encoder and maps the sequence output to a single class index; the last linear layer is a two-layer perceptron, the first layer output dimension is 1024, and the second layer is used for classification, and the output dimension is the number of categories 2, representing crack-free images and cracked images.

[0069] The Transformer encoder architecture consists of two parts: a self-attention layer (multi-head attention mechanism) and a feed-forward neural network. In the self-attention layer, the input sequence is first converted into three different vectors: a query vector Q, a key vector K, and a value vector V, with dimensions d q ,d k and d v The attention function between the input vectors is calculated as follows:

[0070]

[0071] First, we calculate the score QK between each pair of input vectors T , when encoding the data at the current position, the score determines how much attention should be paid to other input data. Then the score is multiplied by Normalization is performed to enhance gradient stability. The scores are then converted to probabilities using the Softmax function. Finally, the probabilities are multiplied by the value vector, so that vectors with larger probability values ​​will receive additional attention. In addition, the self-attention layer lacks the ability to capture position information in the input sequence. To solve this problem, additional position encoding needs to be added to the original input embedding.

[0072] The self-attention layer used in the Transformer architecture is a multi-head self-attention mechanism. The multi-head self-attention mechanism achieves the ability to focus on multiple specific locations by giving the attention layer different subspace representations. The vectors generated by multiple heads are concatenated and mapped to the final output. In this mechanism, the input vector is mapped into three different matrices: and The process of the multi-head self-attention mechanism can be expressed as:

[0073] head i =Attention(Q i ,K i ,V i),i=1...h

[0074] MultiHead(Q′,K′,V′)=Concat(head 1 ,head 2 ,...head h )W O

[0075] Among them, h is the number of attention heads, Q′, K′, and V′ are and The series connection, Represents the linear transformation matrix (d is the embedding dimension). Similar to the sparse connection of convolution, multi-head attention separates the input into h independent attention heads with d / h dimensional vectors and integrates the features of each head, enriching the diversity of the feature subspace without additional computational cost.

[0076] Afterwards, the output of the self-attention layer is fed into a feed-forward neural network (FFN), which consists of two linear transformation layers and a nonlinear activation function, which can be expressed as the following function:

[0077] FFN(x)=W 2 σ(W 1 x)

[0078] Where W 1 and W 2 are the two parameter matrices of the two linear transformation layers, and σ represents the nonlinear activation function GELU. This position-based feedforward layer can be viewed as a point-by-point convolution, which transforms the representation of all positions in the input sequence using the same multi-layer perceptron.

[0079] The Transformer encoder is followed by a sequence pooling layer. Sequence pooling is a method proposed in the paper "Escaping the BigData Paradigm with Compact Transformers" that aggregates the output of the Transformer backbone throughout the sequence and maps the sequence output to a single class index. Vision Transformer usually adds an additional learnable [class] tag to the sequence of embedded patches, which represents the classification parameters of the entire image and contains potential information for classification. Its output after the Transformer encoder can be used for classification. Sequence pooling increases the flexibility of the model and can include relevant information from different parts of the input image, eliminating the mandatory need for the [class] tag and making the model compact. This operation can be seen as a mapping transformation:

[0080]

[0081] Given where x L or f(x 0 ) is the output of the L-layer Transformer encoder, b is the mini-batch size, n is the length of the input sequence, and d is the dimension of the embedding. L Input a linear layer And apply the Softmax function:

[0082]

[0083] Then calculate:

[0084]

[0085] By pooling the second dimension we get This output can then be fed into a linear classifier for classification.

[0086] The final linear layer is a two-layer perceptron (MLP) with a first layer of 1024 and a second layer for classification. The output dimension is the number of categories 2, representing no cracks and cracks.

[0087] S302, training and testing;

[0088] First, pre-train on the ImageNet-1K dataset, with the input image resolution of 224*224. Then use the pre-trained model to fine-tune on the dataset obtained in step S2.

[0089] The dataset is divided into a training set and a test set in a ratio of 8:2. The model parameters are set as follows: batch_size is set to 32, the learning rate lr is set to 5e-4, the minimum learning rate min_lr is set to 0.00001, the learning rate decay method sched is set to cosine, the warmup learning rate warmup_lr is set to 1e-6, the warmup learning rate iteration number warmup_epochs is set to 10, the weight decay coefficient weight_decay is set to 5e-2, and the number of iterations is set to 100. During model training, the training set images are randomly flipped left and right, randomly added with noise, randomly adjusted in brightness, randomly adjusted in contrast, and normalized, while the test set images are only normalized.

[0090] The PyTorch deep learning framework is used for training. The images of the training set are input into the lightweight Transformer network. In an iterative optimization process, 32 training samples are randomly sampled to form a batch. The network is optimized and the network parameters are updated in combination with the back propagation algorithm. After the training is completed, the model with the highest accuracy on the test set is saved to obtain the crack detection model.

[0091] Step S4: deploying a recognition model to carry out recognition of the surface image of the bridge to be recognized;

[0092] Integrate the trained model into the bridge concrete crack detection software. This embodiment uses the Electron open source framework to build the bridge concrete crack detection software for PC. Electron is a cross-platform desktop application framework based on Chrominum and Node.js. Desktop applications can be developed using Web technologies such as HTML, CSS and JavaScript. Applications developed on this framework can run on Windows, macOS and Linux operating systems. Electron combines Chrominum's v8 engine and Node runtime, adopts a front-end and back-end separation architecture, the front-end is developed in a Web manner, and the back-end communicates between processes. Put the model file and the program written to call the model in the same file directory in the Electron project to complete the deployment.

[0093] The user transfers the image to be identified to the local computer and uses the bridge concrete crack detection software to detect cracks. During the detection, the user can upload a single or multiple pictures, the software calls the model for identification, and returns the identification results to the software interface.

[0094] The present invention provides a bridge concrete crack detection device based on a lightweight Transformer, such as Figure 4 As shown, including:

[0095] The acquisition module uses drones to collect images of bridge concrete cracks and transmits the collected data back;

[0096] The dataset module preprocesses the images and converts them into a fixed input format to establish a dataset.

[0097] The training module builds a lightweight Transformer network to train and test images to obtain a recognition model;

[0098] The evaluation module deploys the recognition model and carries out the recognition of the surface images of the bridge to be identified.

[0099] Furthermore, the acquisition module includes: UAV flight preparation, including weather detection, flight plan design and pre-flight testing; UAV flight execution and image acquisition; UAV return and data inspection.

[0100] Weather detection is to grasp the specific weather conditions, observe the light, air visibility, etc., to prevent the impact of strong winds, cloudy weather and other weather on the drone. Flight plan design includes planning the flight route, altitude, flight speed, overlap and finding a suitable take-off point. Before the official flight, a pre-flight test should be carried out to check whether the various functions of the drone can operate normally and whether the operation instructions can be completed. The wind speed, weather, take-off and landing coordinates and other information of the day should be recorded, and the data should be saved for future reference and analysis.

[0101] Two technicians are required for drone flight, a pilot and an observer. The pilot is responsible for controlling the movement of the fuselage, and the observer is responsible for observing the flight status of the drone and collecting data. The drone flies along the inspected surface of the bridge, and uses control devices such as remote controllers to capture images of the bottom, columns, and beams of the bridge. After completing the flight mission, the drone lands at the predetermined location. When landing, the landing site should be ensured to be safe and avoid unauthorized personnel approaching. After landing, check whether the image data and other related information in the drone's onboard camera are complete. If the image quality is unqualified, it is necessary to re-fly the area it has captured.

[0102] Furthermore, the dataset module includes: 1) image export and processing, taking out the memory card carried by the drone, placing it in the card reader, and importing the photos taken by the drone into the local computer; processing the taken pictures, including image annotation (with cracks and without cracks), image noise reduction and quality enhancement, geometric correction, and auxiliary information extraction; if the quality of the processed pictures is still low, such as blurry pictures, exposure errors, etc., these pictures can be deleted. 2) Dataset construction, establishing a data set of bridge surface images, selecting a certain number of pictures with cracks and without cracks, and keeping the two types of pictures as equal as possible. Afterwards, screen the pictures with too high overlap. If the overlap of the two pictures reaches 90%, it is considered that the two pictures are highly similar, and one of the pictures should be selected to join the data set. Convert all the two types of pictures into RGB images in PNG format and put them into two folders to complete the dataset construction. In order to ensure the accuracy of the training model, the number of pictures in the data set should be greater than 500.

[0103] Furthermore, the training module includes a lightweight Transformer network and its training and testing, wherein the lightweight Transformer includes two parts: patch-based segmentation and Transformer with sequence pooling, wherein the patch-based segmentation includes blocking, two-stage Patch transformation and linear projection, and the Transformer with sequence pooling includes a Transformer encoder, a sequence pooling layer and a linear layer; position encoding needs to be inserted between the two parts; the two-stage Patch transformation method discards part of the Transformer input sequence during training, forcing the Transformer to use an incomplete sequence for training, and this method is not used during testing, and the complete input sequence is used for calculation.

[0104] Furthermore, the evaluation module includes a deployment unit, which is used to deploy the lightweight Transformer model to the bridge concrete crack detection software; and an identification unit, which is used to detect bridge concrete cracks.

[0105] The front end of the bridge concrete crack detection software is responsible for file selection, recognition and result return, and the back end is responsible for the main process and communication. The front end is developed using the desktop component library Element Plus based on Vue 3. The file selection function is used to select the image to be recognized on the local computer. The user can select a single file or a folder. A single file is used to recognize a single bridge surface image. A folder is used to recognize multiple bridge surface images. Multiple images need to be placed under its path; it supports common image formats such as jpg, png, bmp, and tiff. The recognition function uses the child_process module of Electron to create a child process, and then runs an external executable program in the child process to call the recognition model; taking the Windows system as an example, the python file will be packaged to generate an exe program. When used, child_process can call the cmd execution command of the Windows system. The result return function is used to capture and display the results output by the recognition unit; the recognition results output by the child process unit are captured using JavaScript regular expressions, and are listed in a table in the interface after parsing. The back end is responsible for controlling the life cycle of the crack detection program, creating and managing software windows, and controlling desktop function modules such as menus and dialog boxes; providing the entrance and closing logic of each menu.

[0106] The deployment unit also includes a calling program for calling the lightweight Transformer model by the bridge concrete crack detection software, which can receive path parameters (user input of the file selection unit, the path of a single file or folder), locate the image to be identified according to the path parameters, and then call the recognition model to identify the image, obtain the recognition result (including whether it is a crack and the accuracy of the recognition), and output it in the form of a string. The calling program is packaged through the third-party python library PyInstaller to generate an executable program. PyInstaller can package python source files into independent executable programs, so that it can be executed independently in an environment where python is not installed, and it can also be used as an independent file for easy delivery and management. After the executable program is generated, this program and the model file are placed in the same file directory in the Electron project to complete the deployment. After that, the software is packaged using the electron-builder module, and the generated installation program can be installed on other users' computers.

[0107] The user uses the generated installation program to install it on the local computer where crack detection is required, transfers the bridge surface image to be identified to this computer, and uses this software to perform crack detection. The software operation process is as follows: Figure 5 As shown;

[0108] The main interface is as follows Figure 6 As shown in the figure, click the "Crack Identification" button in the main interface and select the file (single image detection) or folder (batch image detection) to be detected. After the selection is completed, enter the recognition interface, as shown in Figure 7 As shown in the figure, click the "Start Detection" button, the software calls the trained crack detection model and inputs the user-selected image into the model for detection. The user waits for the system to complete the detection, and then the software interface returns the file name, detection results and detection probability of all selected images, as shown in the figure below. Figure 8 As shown; for each image table, two columns are listed: "File Name" and "Result and Probability": "File Name" refers to the file name of the uploaded image; "Result and Probability" refers to the detection result of whether there is a crack and its confidence, which is limited to "with cracks" and "without cracks";

[0109] For batch crack detection, the software will generate a list of txt files with crack images, including the names of the image files identified as cracks. The txt file can be generated based on the probability threshold, which is 50% by default, that is, when the model recognizes the crack with a probability greater than 50%, the image path will be added to the txt file. In addition, you can choose the probability threshold of 50%, 60%, 70%, 80%, 90% or a custom threshold. The generated txt file can be used for subsequent detection.

[0110] The user transfers the image to be identified to the local computer and uses the software to perform crack detection. During detection, the user can upload a single or multiple images, the software calls the model for identification, and returns the identification results to the software interface.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bridge concrete crack detection method based on lightweight Transformer, characterized in that: The steps include: Step S1: Use a drone to collect images of bridge concrete cracks and transmit the collected data back; the drone first performs flight preparation, including weather detection, flight plan design, and pre-flight testing, then takes off to perform the image collection task, returns after completion, and checks the collected data; if the image quality is unqualified, it is necessary to make up for the shooting area corresponding to the image; Step S2: pre-process the image and convert it into a fixed input format to establish a data set; import the pictures taken by the drone into the local computer, perform image annotation, image noise reduction and quality enhancement, geometric correction, and auxiliary information extraction; then select a certain number of images with and without cracks to establish a data set of bridge surface images; Step S3: Construct a lightweight Transformer network, train and test the image to obtain a recognition model; the network includes two parts: patch-based segmentation and Transformer with sequence pooling, wherein the patch-based segmentation includes block segmentation, two-stage Patch transformation and linear projection, and the Transformer with sequence pooling includes Transformer encoder, sequence pooling layer and linear layer; position encoding needs to be inserted between the two parts; the bridge surface image needs to be segmented into 8*8 patches before inputting the network, and then a two-stage Patch transformation is performed, and the input sequence of the Transformer encoder is formed after linear projection, wherein the embedding dimension of each patch is 256; the Transformer encoder has a total of 6 blocks, each block includes a multi-head attention layer and a position-based feedforward network FFN, wherein the multi-head attention layer has 4 heads, the first layer dimension of the FFN is 512, the second layer dimension is 256, and the activation function is GELU; the outputs of the multi-head attention layer and the feedforward network are respectively sent to the "add and norm" layer, which contains the residual structure and layer normalization; the sequence pooling layer aggregates the output of the Transformer encoder and maps the sequence output to a single class index; the final linear layer is a two-layer perceptron, the first layer output dimension is 1024, the second layer is used for classification, and the output dimension is the number of categories 2, representing crack-free images and cracked images; wherein the two-stage Patch transformation method is as follows: In the first stage, grid-shaped Patch dropping is used. Among all 8*8 patches generated by the image, a certain number of patches are discarded at intervals in a grid shape. Let Patches be a patch matrix with m rows and n columns. The formula for grid-shaped Patch dropping is as follows: Drop=Reshape(Cycle(0,1,p)) Patches=Patches*Drop First, calculate the Drop matrix, which is a binary matrix containing only 0 and 1. 0 represents the patch to be discarded, and 1 represents the patch to be retained. The Cycle function is used to generate an alternating vector of 0 and 1, where p = m*n represents the total number of patches. Then, take a continuous vector of length p from the alternating matrix and reconstruct it into a matrix of the same size as Patches. Multiply Patches by the Drop matrix to obtain the final retained Patches matrix. In the second stage, the patches discarded in the first stage are converted into 4*4 thumbnails and randomly mixed with the retained input patches. The formula is as follows: x′=Mx1+Φ(T(x2)) Where x1 and x2 are two input patches, x′ is the new hybrid patch, M is a binary mask indicating the area of ​​the patch to be cropped and retained, Φ is a padding operation that generates a patch of the same size as x2, and T(x2) is a thumbnail operation; The two-stage patch transformation method discards part of the input sequence of the Transformer during training, forcing the Transformer to use an incomplete sequence for training. This method is not used during testing, and the complete input sequence is used for calculation; Step S4: Deploy the recognition model to carry out recognition of the bridge surface image to be recognized; integrate the trained lightweight Transformer model into the bridge concrete crack detection software; the user transfers the image to be recognized to the local computer and uses the software to detect cracks; during detection, the user can upload a single or multiple pictures, the software calls the model for recognition, and returns the recognition results to the software interface.

2. A bridge concrete crack detection device based on lightweight Transformer, characterized in that: It includes the following modules: A collection module that uses drones to collect images of bridge concrete cracks and transmits the collected data back; The dataset module preprocesses the images and converts them into a fixed input format to build a dataset; The training module builds a lightweight Transformer network to train and test images to obtain a recognition model. The lightweight Transformer network includes two parts: patch-based segmentation and Transformer with sequence pooling. The patch-based segmentation includes block segmentation, two-stage patch transformation and linear projection, and the Transformer with sequence pooling includes a Transformer encoder, a sequence pooling layer and a linear layer. Position coding needs to be inserted between the two parts. The two-stage Patch transformation method comprises: In the first stage, a grid-shaped patch dropping is used to discard a certain number of patches of all 8*8 patches generated by the image according to the grid shape interval; In the second stage, the patches discarded in the first stage are converted into 4*4 thumbnails and randomly mixed with the input patches retained in the first stage; The two-stage patch transformation method discards part of the input sequence of the Transformer during training, forcing the Transformer to use an incomplete sequence for training, while this method is not used during testing and the complete input sequence is used for calculation; The evaluation module deploys the recognition model and carries out the recognition of the surface image of the bridge to be recognized, including: a deployment unit, which is used to deploy the lightweight Transformer model; a recognition unit, which is used to detect cracks in the bridge concrete, calls the software to detect cracks in the image to be recognized transmitted by the user to the local computer, and returns the recognition results to the software interface.

Images