Welding quality inspection method for assembled bridges based on MIU-Transformer

Abstract

The present invention is a method for detecting the welding quality of prefabricated bridges based on MIU-Transformer, comprising the following steps: obtaining a dataset of weld defect images of prefabricated bridges, annotating the images at the pixel level, and assigning corresponding semantic labels to each pixel; using the MIU-Transformer neural network for semantic segmentation defect measurement, the MIU-Transformer neural network includes a framework integrating mask image uniformity and unsupervised domain adaptation, a main network consisting of a Transformer encoder and a context-aware feature fusion decoder; using the trained MIU-Transformer neural network to obtain the semantically segmented image, and calculating the area of ​​the defect. The method can accurately and efficiently detect and quantify the weld defects of steel materials, providing an effective means for quality assurance and maintenance of prefabricated bridges.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge welding quality detection, and in particular to a method for detecting welding quality of an assembled bridge based on an MIU-Transformer. Background Art

[0002] As an important part of modern transportation infrastructure, prefabricated bridges carry the safe passage of vehicles and pedestrians. However, due to various reasons, various defects may occur in bridge welds, which not only affect their performance, but also pose a threat to traffic safety. In general, the impact of poor weld quality on steel is comprehensive, involving the safety, reliability, durability, economy, construction, aesthetics, standardization and environmental impact of the structure. Therefore, ensuring the quality of welds is an important link that cannot be ignored in steel structure engineering. Therefore, the large-scale normalized measurement of prefabricated bridge welding detection is of vital importance.

[0003] In recent years, with the widespread application of ultrasonic technology, ultrasonic testing has become one of the main methods for measuring welding defects in prefabricated bridges. The transmitter generates a high-frequency ultrasonic signal and transmits the signal to the surface of the prefabricated bridge through a coupling agent; the receiver receives the ultrasonic signal reflected by the defect; the signal processing system filters and amplifies the received signal; the display and recording equipment displays the processed signal in the form of waveforms, images, etc. for analysis by the inspectors. However, the type and properties of the materials of the prefabricated bridge have a direct impact on the effect of ultrasonic testing. For example, some alloy elements and impurities may cause sound wave scattering and reduce the detection accuracy. In addition, the purchase and maintenance costs of ultrasonic testing equipment are high, which may be difficult to afford for some regions or projects with limited economic conditions. In addition, this method is not real-time during the welding process, and the maintenance and calibration of the equipment also requires professional knowledge and skills, which increases the difficulty of use and maintenance.

[0004] The traditional method is to use manual inspection methods, allowing surveyors to find welding defects in prefabricated bridges through visual inspection or tapping inspection, and classify and mark them. However, it is far from enough to analyze welding defects in prefabricated bridges by manual methods alone. The method of manual screening is time-consuming and labor-intensive, and an efficient welding detection method is urgently needed. At present, with the rapid development of intelligent technology, its application in the measurement of welding defects of steel structure prefabricated bridges is becoming more and more extensive. Intelligent theory and technology mainly include artificial intelligence, machine learning, image processing, etc. By using these technologies, automatic identification, classification and evaluation of bridge welding defects can be achieved, and detection efficiency and accuracy can be improved. The development of image analysis methods based on deep learning provides a new way to solve this problem. The present invention proposes a new method for welding quality detection of prefabricated bridges based on MIU-Transformer. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above technologies and provide a stable, real-time, high-precision, intelligent, prefabricated bridge welding quality detection method based on MIU-Transformer. It can realize the quantitative measurement and analysis of welding defects of prefabricated bridges, accurately and efficiently detect and quantify the defects of steel material welding seams, and provide an effective means for quality assurance and maintenance of prefabricated bridges.

[0006] To achieve the above object, the technical solution of the present invention is:

[0007] In a first aspect, the present invention provides a method for detecting welding quality of an assembled bridge based on MIU-Transformer, the method comprising the following steps:

[0008] Obtain a dataset of weld defect images of prefabricated bridges, including bubble defect images containing pores or slag inclusion defects, weld defect images containing welding cracks or incomplete fusion defects, and defect-free images. Annotate the images at the pixel level and assign corresponding semantic labels to each pixel.

[0009] The MIU-Transformer neural network is used for semantic segmentation defect measurement. The MIU-Transformer neural network includes a framework that integrates mask image uniformity and unsupervised domain adaptation, a main network consisting of a Transformer encoder and a context-aware feature fusion decoder;

[0010] The framework of integrated mask image uniformity and unsupervised domain adaptation is referred to as MIU framework, including teacher network, source image and a corresponding one-hot label For training the main network f θ With the supervision loss L S ; After that, the source image is used to generate the source prediction image through the main network, and the supervision loss between the source prediction image and the real image is calculated using the following formula;

[0011]

[0012] Among them, the superscript s represents the corresponding data of the source image, Ns is the number of samples for each label, and the subscript k represents the kth element;

[0013] At the same time, the target image is input into the main network, mask operation, and teacher network respectively. The teacher network and the main network update the weight of the teacher network through the exponential moving average EMA to transfer knowledge from the source domain to the target domain; the unlabeled image of the target domain Input the main network and calculate the adaptation loss by the following formula

[0014]

[0015] Among them, H and W represent the height and width of the target image, H×W is the image pixel area, and C represents the number of categories. represents the quality estimate of the pseudo-label corresponding to the target image, represents the true label of the target image, Represents the main network output; the superscript T represents the corresponding data of the target image;

[0016] Adaptation loss is used to improve the ability to effectively generalize to unseen data. The model parameters θ of the main network are optimized with adaptation loss, so that the model's prediction in the target domain is closer to the true label.

[0017] When the target image is input into the mask operation, the MIU framework trains the network to predict the semantic segmentation of the entire image by masking random patches of the target image. The MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain a mask. The masked target image is obtained by element-by-element multiplication of the mask and the target image. The masked target image is input into the main network to obtain the mask prediction result;

[0018] When the target image is input into the teacher network, according to Obtain pseudo labels for target domain data;

[0019] in Pseudo labels representing target domain data; represents the output of the teacher network; c is the category label, which is an integer between 1 and C; Indicates the category label with the maximum probability value among all category labels c';

[0020] Use the teacher network to generate pseudo labels for the target domain data, and use local and contextual information to improve the quality of pseudo labels in the target domain, thereby generating more reliable labels;

[0021] The pseudo-labels and their quality estimates are used for further training on the target domain; the mask predictions are input into the MIU loss calculation together with the pseudo-labels, and the MIU loss calculation formula is:

[0022] L M =q T H(y M ,p T )

[0023] Among them, p T represents the pseudo label of the target image, q T is the quality estimation of the target image; y Mis the predicted probability of the main network for the masked target image; H(.) represents the maximum entropy function;

[0024] After constrained training with three loss functions, the final output of the MIU-Transformer neural network is obtained, and the semantically segmented image is obtained, and then the area of ​​the defect is calculated.

[0025] Further, the encoder includes an overlapping patch embedding module, four converter modules, and the context-aware feature fusion decoder includes a multi-layer perceptron and a context-aware fusion module;

[0026] The overlapping patch embedding module in the image input encoder cuts the input image into tiles, and then passes through the first converter module, the second converter module, the third converter module, and the fourth converter module in sequence to obtain multi-scale feature maps, denoted as F1, F2, F3, and F4, respectively. The multi-scale feature map output is connected to the context-aware feature fusion decoder, and the multi-level features output by the multi-layer perceptron are concatenated and concatenated to generate a heap input to the context-aware fusion module to obtain the output of the decoder;

[0027] The context-aware fusion module includes two 1×1 convolutions and multiple 3×3 depth-separable convolutions with different dilation rates. After the input image is processed by a structure in which multiple parallel 3×3 depth-separable convolutions with different dilation rates and one 1×1 convolution are set in parallel, the output of the decoder is obtained through a splicing operation and a 1×1 convolution.

[0028] Furthermore, the first converter module, the second converter module, the third converter module, and the fourth converter module have the same structure, including a self-attention mechanism module SWEfficientSelf-Attention based on sliding window technology, a Mix-feedforward network module, and an overlapping patch fusion module. Four feature maps of the same size but different resolutions are obtained through the four converter modules.

[0029] Furthermore, the process of the self-attention mechanism module SWEfficientSelf-Attention based on sliding window technology is: the output of the previous layer is recorded as Z l-1 , Z l-1 It is processed by layer normalization LN operation and window-based multi-head self-attention W-MSA module in turn, and then the output of the W-MSA module is compared with Z l-1 Perform dropout and residual connection to obtain After that, it is processed by an LN layer and an FC layer before being combined with After dropout and residual connection, we get Z l ;

[0030] Z l After being processed by an LN layer and a sliding window-based multi-head self-attention mechanism SW-EMSA module, it is combined with Z l Perform dropout and residual connection to obtain After that, it is processed by an LN layer and an FC layer before being combined with After dropout and residual connection, we get Z l+1 , and finally processed by the patch merging layer to obtain the output of the self-attention mechanism module SW Efficient Self-Attention based on the sliding window technology.

[0031] Furthermore, the convolution kernel size of the overlapping patch fusion module (Overlap Patch Merging) of the first converter module is kernel=7, stride=4, and padding=3; the convolution kernel size of the overlapping patch fusion module (Overlap Patch Merging) of the second converter module, the third converter module, and the fourth converter module is kernel=3, stride=2, and padding=1; the pixel size of the moving window is 4*4, and the moving step stride=2.

[0032] Furthermore, the image to be identified is input into the trained MIU-Transformer neural network to detect defect targets, thereby obtaining the pixel area and actual area of ​​each defect, and setting the measurement parameter threshold of the bubble defect;

[0033] If it is identified as a welding crack or unfused image, an alarm will be issued and defect information will be displayed;

[0034] If it is identified as a pore or slag inclusion defect image, the ratio of the defect pixel area to the lens detection area is calculated to see whether it exceeds the measurement parameter threshold of the bubble defect. The measurement parameter threshold of the bubble defect is 2%. If it does not exceed the measurement parameter threshold of the bubble defect, the bridge welding output is qualified, and no alarm or mark is given if it is qualified; if the ratio of the defect pixel area to the lens detection area exceeds the measurement parameter threshold of the bubble defect, the ratio of the defect pixel area to the lens detection area and the defect type are passed to the prompt processing module to notify relevant personnel to process the weld according to the alarm information; then the processed weld is re-inspected until all areas do not exceed the measurement parameter threshold, and the quality inspection of the prefabricated bridge is completed.

[0035] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which can implement the steps of the method when executed by a processor.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention applies the MIU-Transformer neural network to the welding quality inspection of prefabricated bridges, which has the function of finely identifying image information, can identify and quantify welding defects, overcome the difficulty of manual monitoring of welding quality, avoid potential harm to operators and the need for equipment maintenance, realize real-time monitoring and quality control, improve construction efficiency and quality, reduce dependence on professional and technical personnel, reduce labor costs, and reduce the pressure of workers working in harsh environments, and provide a more reliable and safe solution for prefabricated bridge construction. Compared with existing algorithms, it shows obvious advantages in mRecall, mF1 and mIoU indicators.

[0038] The system of the present invention provides the possibility of realizing welding quality inspection on a larger scale. The large amount of data collected can be used for further analysis to provide data support for improving welding process and quality control. After being trained, the algorithm can adapt to different production environments, different welding parameters and various types of prefabricated bridge structures. It further promotes the development of this field and promotes the transformation of traditional manufacturing to intelligent manufacturing. It has great significance and broad application prospects, promotes the innovation and application of related technologies, and also provides strong support and guarantee for the development of welding quality inspection of prefabricated bridges. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall structure of the MIU-Transformer neural network in the present invention;

[0040] Figure 2 It is a schematic diagram of the structure of the main network of the MIU-Transformer neural network in the present invention;

[0041] Figure 3 It is a schematic diagram of the internal structure of the transformer block in the MIU-Transformer main network of the present invention;

[0042] Figure 4 It is a schematic diagram of the internal structure of the self-attention mechanism module SW Efficient Self-Attention based on the sliding window technology in the present invention. DETAILED DESCRIPTION

[0043] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0044] The present invention provides a method for detecting welding quality of assembled bridges based on MIU-Transformer, comprising the following steps:

[0045] Step 1: Obtain a dataset of weld defect images of prefabricated bridges, including bubble defect images containing pores or slag inclusion defects, weld defect images containing welding cracks or unfused defects, and defect-free images, and annotate the images at the pixel level, assigning corresponding semantic labels to each pixel; obtain source domain data (annotated images) and target domain data (unannotated images);

[0046] Step 2: Use MIU-Transformer neural network to measure semantic segmentation defects

[0047] For the semantic segmentation defect measurement model, the MIU-Transformer neural network is used, including: a framework integrating masked image uniformity and unsupervised domain adaptation (UDA), referred to as the MIU framework; a main network consisting of a Transformer encoder and a context-aware feature fusion decoder.

[0048] The framework of integrating mask image uniformity and unsupervised domain adaptation includes a teacher network, a source image and a corresponding one-hot label For training the main network f θ With the supervision loss L S ; After that, the source image is used to generate the source prediction image through the main network, and the supervision loss between the source prediction image and the real image is calculated using the following formula;

[0049]

[0050] Among them, the superscript s represents the corresponding data of the source image, Ns is the number of samples for each label, and the subscript k represents the kth element;

[0051] At the same time, the target image is input into the main network, mask operation, and teacher network respectively. The teacher network and the main network update the weight of the teacher network through the exponential moving average EMA to transfer knowledge from the source domain to the target domain; the unlabeled image of the target domain Input the main network and calculate the adaptation loss by the following formula

[0052]

[0053] Among them, H and W represent the height and width of the target image, H×W is the image pixel area, and C represents the number of categories. represents the quality estimate of the pseudo-label corresponding to the target image, represents the true label of the target image, Represents the main network output; the superscript T represents the corresponding data of the target image;

[0054] Adaptation loss is used to improve the ability to effectively generalize to unseen data. The model parameters θ of the main network are optimized with adaptation loss so that the model's prediction on the target domain is closer to the true label.

[0055] When the target image is input into the mask operation, the MIU framework trains the network to predict the semantic segmentation of the entire image by masking random patches of the target image. The MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain a mask. The masked target image is obtained by element-by-element multiplication of the mask and the target image. The masked target image is input into the main network to obtain the mask prediction result;

[0056] When the target image is input into the teacher network, according to the formula Obtain pseudo labels for target domain data;

[0057] in Pseudo labels representing target domain data; represents the output of the teacher network, that is, the probability distribution of the feature vector of the target domain data updated based on the source domain; c is the category label, which takes an integer value of 1-C; Indicates the category label with the maximum probability value among all category labels c';

[0058] Use the teacher network to generate pseudo labels for the target domain data, and use local and contextual information to improve the quality of pseudo labels in the target domain, thereby generating more reliable labels;

[0059] The pseudo-labels of the target images and their quality estimates are used for further training on the target domain; the mask predictions are input into the MIU loss calculation together with the pseudo-labels, and the MIU loss calculation formula is:

[0060] L M =q T H(y M ,p T )

[0061] Among them, p T represents the pseudo label, q T is the quality estimation of the target image; y M is the predicted probability of the main network for the masked target image; H(.) represents the maximum entropy function;

[0062] After constrained training of three loss functions, the final output of the neural network is obtained, and the image after semantic segmentation is obtained, and then the area of ​​the defect is calculated.

[0063] In the main network, the encoder includes an overlapping patch embedding module and four converter modules, and the context-aware feature fusion decoder includes a multi-layer perceptron and a context-aware fusion module; the overlapping patch embedding module in the image input encoder cuts the input image into blocks, and then passes through the first converter module, the second converter module, the third converter module, and the fourth converter module in sequence, and the multi-scale feature map is output, denoted as F1, F2, F3, and F4, and the multi-scale feature map output is connected to the context-aware feature fusion decoder, and the multi-level features output by the multi-layer perceptron are concatenated and Concat is performed. After concatenation, a heap is generated and input into the context-aware fusion module to obtain the output of the decoder.

[0064] The first converter module, the second converter module, the third converter module, and the fourth converter module have the same structure, including a self-attention mechanism module (SWEfficientSelf-Attention) based on sliding window technology, a Mix-feedforward network module, and an overlapping patch fusion module (OverlapPatchMerging). Four feature maps of the same size but different resolutions are obtained after the four converter modules.

[0065] The self-attention mechanism module (SWEfficientSelf-Attention) based on the sliding window technology first performs a layer normalization operation on the block output by the previous layer, and then enters the window-based multi-head self-attention W-MSA module, cuts the block into windows to calculate the attention weight, merges the attention weight back to the corresponding position in the original feature map, and then performs dropout and residual connection; after layer normalization operation processing, enters the multi-head self-attention mechanism SW-EMSA module, sets the moving window and the moving step, slides the moving window on each block according to the moving step, calculates the attention weight of each window, merges the attention weight back to the corresponding position in the original feature map, and finally obtains the output through a layer normalization operation LN, a fully connected layer FC, dropout and a residual connection in turn, and finally obtains the feature Z l+1 , after being processed by the patch merging layer PatchMerging, the output of the self-attention mechanism module SWEfficientSelf-Attention based on the sliding window technology is obtained.

[0066] The training set is used to train the MIU-Transformer neural network, and the trained MIU-Transformer neural network is used to perform semantic segmentation and defect detection on the prefabricated bridge welding images to be inspected, and the pixel area of ​​welding defects and bubble defects is obtained. The actual area of ​​the defect is then obtained through the area conversion module for later detection and early warning.

[0067] In this embodiment, the convolution kernel size of the overlapping patch fusion module (OverlapPatchMerging) of the first converter module is kernel=7, stride=4, and padding=3; the convolution kernel size of the overlapping patch fusion module (Overlap Patch Merging) of the second converter module, the third converter module, and the fourth converter module is kernel=3, stride=2, and padding=1; the pixel size of the moving window in the four converter modules is 4*4, and the moving step stride=2.

[0068] The area conversion module is used to obtain the actual area of ​​the defect according to the pixel area of ​​the defect using the pixel scale S; the specific process is:

[0069] Step 1: Place a ruler in the image. The ruler should be on the same plane as the weld. The ruler length is L. The pixel length of the ruler in the image is measured to be P. The pixel scale S is:

[0070] S=L / P

[0071] Step 2: Multiply the pixel area of ​​the defect by the square of the pixel scale S to obtain the actual area B of the defect. The actual area B of the defect is calculated by the following formula:

[0072] B=A*S 2

[0073] Where A represents the pixel area of ​​the defect, in pixels squared.

[0074] The test set data is input into the MIU-Transformer neural network trained with the training set, and the calibration algorithm is used to correct the pixel scale S in the area conversion module to obtain the optimal pixel scale S after calibration. * ; The calibration algorithm is:

[0075] Step 1 uses the following error function E(S) to measure the difference between the pixel area and the actual area:

[0076]

[0077] Where Bi represents the actual area of ​​the i-th defect, obtained by annotation; S represents the pixel scale; A i represents the pixel area of ​​the i-th defect, obtained by the output of the MIU-Transformer neural network trained on the training set; N is the total number of predicted defects in all samples in the test set;

[0078] Step 2: Minimize the error function E(S), take the derivative of E(S) and set the derivative equal to 0 to obtain the optimal pixel scale S* ,

[0079]

[0080] The area conversion module is updated with the optimal pixel scale S* to obtain a calibrated area conversion module for area conversion of welding defects in prefabricated bridges.

[0081] The image to be identified is input into the trained MIU-Transformer neural network to detect defect targets, obtain the actual area of ​​each defect, and set the measurement parameter threshold of the bubble defect;

[0082] If it is identified as a welding crack or unfused image, an alarm will be issued and defect information will be displayed;

[0083] If it is identified as a pore or slag inclusion defect image, the ratio of the defect pixel area to the lens detection area is calculated to see whether it exceeds the measurement parameter threshold of the bubble defect. The measurement parameter threshold of the bubble defect is 2%. If it does not exceed the measurement parameter threshold of the bubble defect, the bridge welding output is qualified, and no alarm or mark is given if it is qualified; if the ratio of the defect pixel area to the lens detection area exceeds the measurement parameter threshold of the bubble defect, the ratio of the defect pixel area to the lens detection area and the defect type are passed to the prompt processing module to notify relevant personnel to process the weld according to the alarm information; then the processed weld is re-inspected until all areas do not exceed the measurement parameter threshold, and the quality inspection of the prefabricated bridge is completed.

[0084] Embodiment 1:

[0085] The present embodiment provides a method for detecting welding quality of assembled bridges based on MIU-Transformer, including:

[0086] An image acquisition module, used to obtain images of welding defects of assembled bridges;

[0087] An image processing module is used to segment and annotate the defect images of the image acquisition module at the pixel level to obtain a data set, which includes bubble defect images containing pores or slag inclusion defects, welding defect images containing welding cracks or lack of fusion defects, and defect-free images;

[0088] MIU-Transformer neural network, used to predict and classify defect areas;

[0089] The area conversion module is used to obtain the actual area of ​​the defect based on the predicted defect pixel area using the pixel scale S for later defect analysis;

[0090] The calibration algorithm module uses the error function E(S) to measure the difference between the pixel area and the actual area, takes the derivative of E(S) and sets the derivative equal to 0 to obtain the optimal pixel scale S*;

[0091]

[0092] Where Bi represents the actual area of ​​the i-th defect, obtained by annotation; S represents the pixel scale; Ai represents the pixel area of ​​the i-th defect, obtained by the output of the MIU-Transformer neural network; N is the total number of predicted defects in all samples in the test set;

[0093] The area conversion module is updated with the optimal pixel scale S* to obtain a calibrated area conversion module for the area conversion of weld defects in prefabricated bridges;

[0094] The image to be identified is input into the trained MIU-Transformer neural network, and the actual area of ​​each defect is obtained using the calibrated area conversion module. For bubble defects, the pixel area of ​​the bubble defect is compared with the lens detection area (the size of the picture taken by the camera) and the corresponding measurement parameter threshold to determine whether to alarm. If the measurement parameter threshold is exceeded, the target image is marked and an alarm is issued to notify relevant personnel to process the weld according to the alarm information; the processed weld is then re-inspected until all areas do not exceed the measurement parameter threshold, completing the quality inspection of the prefabricated bridge.

[0095] Example 2

[0096] Step 1

[0097] Step 1.1 The welding defect images and bubble defect images of the printing area obtained during the construction of the 3D printed bridge deck are annotated at the pixel level using the Labelme tool. Each pixel is assigned a corresponding semantic label to form a data set with annotation information. After the annotation is completed, a binary image is generated. The black pixel value is 0, representing the background, and the white pixel value is 255, representing the target defect. The annotation information indicates the category to which each pixel in the image belongs at the pixel level.

[0098] Step 1.2: Divide the data set into a training set, a validation set, and a test set in a ratio of 8:2.

[0099] Step 2

[0100] Read the welding defect images in the training set into the MIU-Transformer neural network, and train to obtain the trained MIU-Transformer neural network;

[0101] In this embodiment, the data in the training set is divided into 10 parts, and one of the 10 parts is used as a validation set each time, and the rest are used as training. After 10 trainings, 10 different models are obtained. The effects of the 10 models are evaluated, and the hyperparameters with the best mIOU are selected. The optimal hyperparameters are used, and then all 10 parts of data are used as training sets to retrain the model to obtain the trained MIU-Transformer neural network.

[0102] Measure the actual area of ​​the defects marked in the test set and the pixel area in the image as the actual reference group;

[0103] The calculation method of the pixel area in the picture is as follows: the binary mask of the defect is obtained by the image segmentation algorithm, that is, the defective part is marked as the foreground and the non-defective part is marked as the background. The number of pixels in the defective part, that is, the number of foreground pixels, is calculated using the pixel counting method, which is represented by N. The number of pixels N is multiplied by the square of the pixel size to obtain the area A of the defective part, in pixels squared.

[0104] Binarization mask refers to the process of converting a certain area of ​​an image into a binary image. In the semantic segmentation task, the binary mask is used to represent the category information of each pixel, that is, to mark each pixel as foreground (target) or background (non-target).

[0105] The area conversion module is directly connected to the output of the MIU-Transformer neural network to convert the pixel area into the actual area.

[0106] It should be noted that the calculation of pixel scale requires that the length of the ruler in the image be accurately measured and that the ruler and the crack defect be on the same plane.

[0107] The process of the prefabricated bridge welding quality inspection method based on MIU-Transformer is mainly divided into two parts: training and prediction. In the training part, the network parameters of the MIU-Transformer neural network are first randomly initialized, and then the defective images of the prefabricated bridge welding area and the corresponding annotated images and transformed images (the images after random cropping of the original images are spliced ​​together with the original images into new input information) are extracted from the training set; secondly, the MIU-Transformer neural network is used to extract the features of each dimension of the image, and the prediction of each pixel category of the image is completed based on the fused features. All images are iterated and trained in sequence. After 160,000 iterations, the training model reaches convergence, that is, when the model training gradient is close to 0 (less than 0.01 can be considered close to 0), the training is stopped, and the optimal network parameters are extracted for prediction. If the training gradient is not close to 0, the weight parameters are adjusted. During prediction, the network first loads the trained parameters and loads the input image from the test set. The category of each pixel is calculated through the trained MIU-Transformer neural network, thereby realizing semantic segmentation.

[0108] Threshold parameter threshold is the threshold obtained by a large number of experiments on the training model. Pixel area refers to the number of pixels of each bubble defect in the segmentation result, which is used to measure the relative size of the bubble defect in the image.

[0109] In the early warning and feedback module, the system will follow the following sub-processes:

[0110] 1) Pass defect information to the background;

[0111] 2) The background analyzes the defect information, where the defect information is the defect type. If the defect type is welding crack or lack of fusion, it is a serious defect and has a significant impact on the structural function, and an alarm is directly issued;

[0112] 3) If the defect type is pores or slag inclusions, the severity level of the defect is determined according to the ratio of the pixel area of ​​the pores or slag inclusions to the lens detection area, wherein the steps for determining the severity level of the pores or slag inclusions defect are: if the ratio of the pixel area of ​​the bubble to the lens detection area is less than 2%, it is a minor defect; if the ratio of the pixel area of ​​the pores or slag inclusions to the lens detection area exceeds 2%, it is a serious defect;

[0113] 4) The background sends warning notifications to relevant personnel based on the severity of the defect;

[0114] 5) Relevant personnel conduct further inspection and take appropriate measures to resolve the defects, such as repair welding, grinding, and re-welding;

[0115] 6) The processed welds are then retested until all areas do not exceed the measurement parameter threshold, completing the inspection of the welding quality of the prefabricated bridge.

[0116] The present invention is mainly used for welding inspection of prefabricated bridges. It uses industrial cameras to automatically identify defects in the welding area of ​​prefabricated bridges, conduct quantitative analysis of the defects, automatically alarm when the area of ​​pores or slag inclusions is larger than the set measurement parameter threshold, and perform material processing, thereby realizing welding inspection of prefabricated bridges.

[0117] Embodiment 3:

[0118] The present embodiment is based on the MIU-Transformer fabricated bridge welding detection method, which uses the MIU-Transformer neural network for semantic segmentation defect measurement. The MIU-Transformer neural network includes a framework integrating mask image uniformity and unsupervised domain adaptation, a main network consisting of a Transformer encoder and a context-aware feature fusion decoder;

[0119] like Figure 2 In the main network shown, the image is input into the Overlap Patch Embeddings module in the Transformer encoder, the input image is cut into patches, and then passed through the first transformer block, the second transformer block, the third transformer block, and the fourth transformer block in sequence to obtain multi-scale feature maps, and the multi-scale feature maps are output to the multi-layer perceptron layer (MLP Layer) of the decoder, and the multi-level features output by the multi-layer perceptron layer (MLPLayer) are concatenated (concat) and the stack generated after concatenation is output.

[0120] The context-aware fusion module uses multiple parallel 3×3 depth-separable convolutions with different dilation rates and one 1×1 convolution. In this embodiment, two parallel 3×3 depth-separable convolutions with different dilation rates and one 1×1 convolution are used to process the result after the stack output, and then the results on all parallel branches are spliced, and then a 1×1 convolution is performed to obtain the output prediction image.

[0121] The main network of the present invention can be used in bottleneck characteristics, can fuse all stacked multi-level characteristics, and can reduce the overfitting phenomenon of the source domain.

[0122] The internal structures of the first converter module, the second converter module, the third converter module and the fourth converter module are as follows: Figure 3As shown in the figure, all of them are processed by the self-attention mechanism module (SW Efficient Self-Attention) based on the sliding window technology, the Mix-feedforward network module, and the overlap patch fusion module (Overlap Patch Merging). The four blocks of the first converter module, the second converter module, the third converter module, and the fourth converter module obtain feature maps with resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 respectively. The feature images are fused into the same size and output through the overlap patch fusion module (Overlap Patch Merging). The difference is that the convolution kernel size of the overlap patch fusion module (Overlap Patch Merging) of the first converter module is kernel=7, stride=4, and padding=3, and the convolution kernel size of the last three overlap patch fusion modules (Overlap Patch Merging) is kernel=3, stride=2, and padding=1. The dimension of the feature map output by the first converter module is

[0123] The feature map output by the second transformer module has a dimension of The feature map output by the third transformer module has a dimension of The feature map dimension of the fourth converter module output is

[0124] The self-attention mechanism module based on sliding window technology (SW Efficient Self-Attention) first normalizes the tiles, slides a 4*4 moving window (window size) on each tile with a step size of stride=2, then cuts the feature map into windows to calculate the attention weights, merges the self-attention output back to the corresponding position in the original feature map (the input of the LN in front of W-MSA or SW-EMSA), and then performs dropout and residual connection, and then passes through layer normalization operation, fully connected layer, dropout and residual connection in sequence.

[0125] The specific process is as follows Figure 4 As shown, the output from the previous layer is recorded as input Z l-1 , processed by layer normalization (LN) and window-based multi-head self-attention (W-MSA) modules, and then obtained by residual processing After being processed by an LN layer and an FC layer, Perform residual connection to obtain Z l . Z l After being processed by the LN layer and the sliding window-based multi-head self-attention mechanism (SW-EMSA) module, it is combined with Z l Residual connection to obtain after After being processed by an LN layer and an FC layer, Residual connection to obtain Z l+1 . Then, through Z l+1 After being processed by the patch merging layer and output, the calculation process of the entire module is shown in the following formula:

[0126]

[0127]

[0128]

[0129]

[0130] Among them, LN represents layer normalization processing, MLP represents fully connected layer processing, W-MSA represents W-MSA module processing, and SW-EMSA represents SW-EMSA module processing.

[0131] The present invention introduces a self-attention mechanism module based on sliding window technology in the encoder, divides the image segmentation task into a series of smaller patches for processing, and applies the self-attention mechanism to each patch, performs local self-attention calculation within the patch, instead of considering the pixels of the entire block, limits the attention mechanism to a small window, and then slides within the block. For each window position, only the relationship between the pixels within the window is considered when calculating the attention weight, which can reduce calculation and memory overhead, improve prediction speed and save training time, improve prediction accuracy and real-time performance, and effectively expand the receptive field of the model.

[0132] Mix-feedforward network module such as Figure 3 As shown in the figure, a 1×1 convolution kernel, a 3×3 convolution kernel and a multi-layer perceptron (MLP) are mixed in each feed-forward network (FFN), and the features output by the self-attention mechanism module (SWEfficient Self-Attention) based on the sliding window technology are used as input, and after a multi-layer perceptron (MLP), a 1×1 convolution kernel and a 3×3 convolution kernel are used for convolution operation, and then a SiLU activation function is used to output the result E through the multi-layer perceptron (MLP) operation. Finally, the result E is added to the original input of the Mix-feedforward network module to obtain the output of the Mix-feedforward network module.

[0133] The structure used in the present invention enables the Mix-feedforward network to effectively encode position information while maintaining low computational complexity. The same Mix-feedforward network is used in all converter modules, which can process features from different levels and reduce information loss of the image, which helps the model capture information of different scales from local details to global structures. The Overlap Patch merging technology is used to enhance local continuity and reduce the consumption of computing resources. In addition, the Mix-feedforward network of the present invention is affected by the target domain image during the training process, and the mask uniformity is taken into account, which improves the segmentation accuracy of the main network.

[0134] In the overall network structure of MIU-Transformer, the framework of integrating mask image uniformity and unsupervised domain adaptation, source image and a corresponding one-hot label For training the main network f θ With the supervision loss L S After that, the source image is passed through the main network to generate the source prediction image, which is then compared with the real image through the supervision loss.

[0135]

[0136] At the same time, the target image is input into the main network and MIU framework respectively. In the MIU framework, the exponential moving average EMA is used to update the teacher network and correct the main network parameters.

[0137] First, the unlabeled images of the target domain Input to the main network, and then adapt the loss to improve the ability to generalize effectively to unseen data:

[0138]

[0139] Among them, H and W represent the height and width of the target image, H×W is the image pixel area, and C represents the number of categories. represents the quality estimate of the pseudo-label corresponding to the target image, represents the true label of the target image, Represents the main network output; the superscript T represents the corresponding data of the target image;

[0140] The purpose of this loss function is to optimize the main network model parameters θ so that the prediction of the main network in the target domain is closer to the true label.

[0141] When the target image is input into the MIU framework, the MIU framework trains the network to predict the semantic segmentation of the entire image by masking random patches of the target image. The MIU loss is L M:First, randomly sample a mask M from the uniform distribution to obtain the mask, and obtain the masked target image by element-by-element multiplication of the mask and the target image. The occluded masked target image is input into the main network to obtain the mask prediction result to predict the semantic segmentation of the entire image, and then use the MIU loss L M =q T H(y M ,p T ) is optimized. Among them, p T represents the pseudo label of the target image, q T represents the quality estimate of the corresponding target image, y M is the predicted probability of the main network for the masked target image.

[0142] The teacher network uses the masked target image to update the moving average EMA processed by the main network. When the target image is input to the teacher network, the teacher network output is obtained. Then, the pseudo label of the target domain data is obtained according to the following formula:

[0143]

[0144] in, represents the probability distribution of the feature vector of the target domain data updated based on the source domain, c is the category label, Represents the category label with the maximum probability value among all category labels c'.

[0145] The teacher network is used to generate pseudo labels for the target domain data, and the local and contextual information is used in the pseudo label generation process to improve the quality of the pseudo labels in the target domain, thereby generating more reliable labels. This pseudo label represents the transfer of knowledge from the source domain to the target domain. In addition, the pseudo label and its quality estimation are used for further training on the target domain to calculate the MIU loss. The mask prediction is input into the MIU loss calculation together with the pseudo label. The MIU loss calculation formula is:

[0146] L M =q T H(y M ,p T )

[0147] Among them, p T represents the pseudo label of the target image, q T is the quality estimation of the target image; y M is the predicted probability of the main network for the masked target image; H(.) represents the maximum entropy function.

[0148] Finally, the supervision loss, adaptation loss and MIU loss jointly constrain the training to obtain the final output of the MIU-Transformer neural network.

[0149] Mask image uniformity is one of the unsupervised domain adaptation methods. The goal of UDA is to adapt the model from the source domain to the target domain without labeled data in the target domain. Since the data in the target domain is not labeled, traditional supervised learning methods are difficult to apply directly. Mask image uniformity enhances the prediction uniformity of the MIU-Transformer neural network when processing occluded images by learning the spatial contextual relationship of the target domain.

[0150] The MIU-Transformer neural network used in the present invention greatly improves the accuracy of segmentation and prediction, enables the system to achieve real-time and high-precision measurement, successfully improves the adaptability and performance of the model on unlabeled target data, and enhances the model's ability to capture instance-level features. The MIU framework improves the segmentation accuracy of the model in complex scenes by clarifying the dependencies between modeling instances (mainly referring to the spatial context, predicting the content of the occluded area based on the area not occluded by the mask). The MIU framework provides additional information by learning the spatial context relationship of the target domain, which allows the teacher network to generate more robust pseudo-labels, utilizing local and contextual clues. These improvements not only improve the robustness of the system when processing complex scenes, but also reduce the dependence on computing resources, making the model more suitable for actual application scenarios. The system is enabled to achieve real-time and high-precision monitoring and the quality of prefabricated bridge welding.

[0151] The experimental data were collected in the bridge steel structure processing factory. The bridge welding defects were collected through an industrial camera, which was fixed at a distance of 0.5m from the bridge steel structure processing platform.

[0152] The experimental data set consists of 5,000 images of prefabricated bridges with welding defects and 3,000 images of prefabricated bridges without defects. Each image in the data set is an RGB image of 512 x 512 pixels. The images of prefabricated bridges without defects are added to improve the accuracy and robustness of training, so that the model can more accurately identify bridge images with defects.

[0153] The areas to be identified in the welding defect images and bubble defect images obtained in the quality inspection of prefabricated bridges are annotated at the pixel level using the Labelme tool, and a corresponding semantic label is assigned to each pixel to form a data set with annotation information. After the annotation is completed, a binary image is generated, with a black pixel value of 0, representing the background, and a white pixel value of 255, representing the target defect; the annotation information indicates the category to which each pixel in the image belongs at the pixel level;

[0154] The labeled images are used as source domain data, and the corresponding unlabeled images are used as target domain data. The source domain data and the target domain data are a corresponding sample, and all corresponding samples constitute a data set. The images in the data set are allocated as training sets and test sets in a ratio of 8:2.

[0155] Different training parameters and learning rate scheduling are used in the present invention to conduct super-supervised semantic segmentation experiments: these parameters include a learning rate scheduling of 40,000 iterations in DeeplabV3+, a learning rate scheduling of 80,000 iterations in DAFormer, a learning rate scheduling of 160,000 iterations in MIU-Transformer, and a learning rate scheduling of 80,000 iterations in PSPNET. When the loss value decreases and stabilizes, while evaluation indicators such as mIoU (mean intersection over union: calculates the ratio of the intersection and union between the predicted output and the true label to measure the accuracy of the model prediction.) and mF1 (the average F1 score combines precision and recall using a harmonic mean calculation, which is particularly suitable for processing unbalanced data sets.) improve and stabilize, the model is considered to have converged. And it is evaluated on the data set. And the performance of models such as MIU-Transformer, DAFormer, PSPNet and DeepLabV3+ is compared, and the comparison results are as follows:

[0156]

[0157] On the dataset, the loss values ​​of the MIU-Transformer model on different indicators tend to be stable. Specifically, MIU-Transformer outperforms other models on the mRecall, mF1, and mIoU indicators.

[0158] mRecall indicator: The mRecall value of MIU-Transformer on the dataset is 83.9%, which is 8.94%, 2.75% and 6.37% higher than DeeplabV3+, DAFormer and PSPNET.

[0159] mF1 indicator: The mF1 value of MIU-Transformer on the dataset is 83.87%, which is 7.42%, 0.84% ​​and 6.23% higher than DeeplabV3+, DAFormer and PSPNET.

[0160] mIoU indicator: The mIoU value of MIU-Transformer on the dataset is 75.38%, which is 7.78%, 0.93% and 6.66% higher than DeeplabV3+, DAFormer and PSPNET.

[0161] Therefore, on the dataset, the loss values ​​of the MIU-Transformer model on the mRecall, mF1, and mIoU indicators tend to be stable, and its performance is better than other models.

[0162] The detection method of the present invention aims to solve the problem that the current welding detection process of prefabricated bridges relies on manual monitoring or the existing intelligent algorithm has low detection accuracy, which is prone to omissions or misjudgments. It can monitor places beyond the human field of vision and realize full-process quality monitoring and early warning of the welding area of ​​prefabricated bridges. Defects can be quantitatively analyzed and automatically alarmed when the defect size exceeds the set measurement parameter threshold. Compared with the existing technology, this technical solution has the following advantages and application prospects: reducing labor costs, improving detection efficiency, and ensuring material quality. This is of great significance to the welding detection industry and has broad application prospects.

[0163] It should be understood that each part of the present invention can be implemented by hardware, software or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or hardware stored in a memory and executed by a suitable instruction execution system.

[0164] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0165] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A method for detecting welding quality of assembled bridges based on MIU-Transformer, characterized in that: The method comprises the following steps: Obtain a dataset of weld defect images of prefabricated bridges, including bubble defect images containing pores or slag inclusion defects, weld defect images containing welding cracks or incomplete fusion defects, and defect-free images. Annotate the images at the pixel level and assign corresponding semantic labels to each pixel. The MIU-Transformer neural network is used for semantic segmentation defect measurement. The MIU-Transformer neural network includes a framework that integrates mask image uniformity and unsupervised domain adaptation, a main network consisting of a Transformer encoder and a context-aware feature fusion decoder; The context-aware feature fusion decoder includes a multi-layer perceptron and a context-aware fusion module. The context-aware fusion module includes two 1×1 convolutions and multiple 3×3 depth-separable convolutions with different dilation rates. After the input image is processed by a structure in which multiple parallel 3×3 depth-separable convolutions with different dilation rates and one 1×1 convolution are set in parallel, the output of the decoder is obtained by a splicing operation and a 1×1 convolution. The framework of integrated mask image uniformity and unsupervised domain adaptation is referred to as MIU framework, including teacher network, source image and a corresponding one-hot label For training the main network f θ With the supervision loss L S ; After that, the source image is used to generate the source prediction image through the main network, and the supervision loss between the source prediction image and the real image is calculated using the following formula; Among them, the superscript s represents the corresponding data of the source image, Ns is the number of samples for each label, and the subscript k represents the kth element; At the same time, the target image is input into the main network, mask operation, and teacher network respectively. The teacher network and the main network update the weight of the teacher network through the exponential moving average EMA to transfer knowledge from the source domain to the target domain; the unlabeled image of the target domain Input the main network and calculate the adaptation loss by the following formula Among them, H and W represent the height and width of the target image, H×W is the image pixel area, and C represents the number of categories. represents the quality estimate of the pseudo-label corresponding to the target image, represents the true label of the target image, Represents the main network output; the superscript T represents the corresponding data of the target image; Adaptation loss is used to improve the ability to effectively generalize to unseen data. The model parameters θ of the main network are optimized with adaptation loss, so that the model's prediction in the target domain is closer to the true label. When the target image is input into the mask operation, the MIU framework trains the network to predict the semantic segmentation of the entire image by masking random patches of the target image. The MIU loss is L M : First, a mask M is randomly sampled from a uniform distribution to obtain a mask. The masked target image is obtained by element-by-element multiplication of the mask and the target image. The masked target image is input into the main network to obtain the mask prediction result; When the target image is input into the teacher network, according to Obtain pseudo labels for target domain data; in Pseudo labels representing target domain data; represents the output of the teacher network; c is the category label, which is an integer between 1 and C; Indicates the category label with the maximum probability value among all category labels c'; Use the teacher network to generate pseudo labels for the target domain data, and use local and contextual information to improve the quality of pseudo labels in the target domain, thereby generating more reliable labels; The pseudo-labels and their quality estimates are used for further training on the target domain; the mask predictions are input into the MIU loss calculation together with the pseudo-labels, and the MIU loss calculation formula is: L M =q T H(y M ,p T ) Among them, p T represents the pseudo label of the target image, q T is the quality estimation of the target image; y M is the predicted probability of the main network for the masked target image; H(·) represents the maximum entropy function; After constrained training with three loss functions, the final output of the MIU-Transformer neural network is obtained, and the semantically segmented image is obtained, and then the area of ​​the defect is calculated.

2. The method according to claim 1, characterized in that The encoder includes an overlapping patch embedding module, four transformer modules, The overlapping patch embedding module in the image input encoder cuts the input image into tiles, and then passes through the first converter module, the second converter module, the third converter module, and the fourth converter module in sequence to obtain multi-scale feature maps, denoted as F1, F2, F3, and F4. The multi-scale feature map output is connected to the context-aware feature fusion decoder, and the multi-level features output by the multi-layer perceptron are concatenated and Concat is performed. After concatenation, a heap is generated and input into the context-aware fusion module to obtain the output of the decoder.

3. The method according to claim 2, characterized in that The first converter module, the second converter module, the third converter module, and the fourth converter module have the same structure, including a self-attention mechanism module SWEfficientSelf-Attention based on sliding window technology, a Mix-feedforward network module, and an overlapping patch fusion module. Four feature maps of the same size but different resolutions are obtained through the four converter modules.

4. The method according to claim 3, characterized in that The process of the self-attention mechanism module SWEfficientSelf-Attention based on sliding window technology is: the output of the previous layer is recorded as Z l-1 , Z l-1 It is processed by layer normalization LN operation and window-based multi-head self-attention W-MSA module in turn, and then the output of the W-MSA module is compared with Z l-1 Perform dropout and residual connection to obtain After that, it is processed by an LN layer and an FC layer before being combined with After dropout and residual connection, we get Z l ; Z l After being processed by an LN layer and a sliding window-based multi-head self-attention mechanism SW-EMSA module, it is combined with Z l Perform dropout and residual connection to obtain After that, it is processed by an LN layer and an FC layer before being combined with After dropout and residual connection, we get Z l+1 , and finally processed by the patch merging layer to obtain the output of the self-attention mechanism module SW Efficient Self-Attention based on the sliding window technology.

5. The method according to claim 3, characterized in that: The convolution kernel size of the overlapping patch fusion module (Overlap Patch Merging) of the first converter module is kernel=7, stride=4, and padding=3; the convolution kernel size of the overlapping patch fusion module (Overlap Patch Merging) of the second converter module, the third converter module, and the fourth converter module is kernel=3, stride=2, and padding=1; the pixel size of the moving window is 4*4, and the moving step stride=2.

6. The method according to claim 1, characterized in that The image to be identified is input into the trained MIU-Transformer neural network to detect defect targets, thereby obtaining the pixel area and actual area of ​​each defect, and setting the measurement parameter threshold of the bubble defect; If it is identified as a welding crack or unfused image, an alarm will be issued and defect information will be displayed; If the image is identified as a pore or slag defect, the ratio of the defect pixel area to the lens detection area is calculated to see if it exceeds the measurement parameter threshold of the bubble defect. The measurement parameter threshold of the bubble defect is 2%. If it does not exceed the measurement parameter threshold of the bubble defect, the bridge welding output is qualified. If it is qualified, no alarm or mark will be made. If the ratio of the defective pixel area to the lens detection area exceeds the measurement parameter threshold of the bubble defect, the ratio of the defective pixel area to the lens detection area and the defect type will be passed to the prompt processing module, and the relevant personnel will be notified to handle the weld according to the alarm information; the processed weld will then be re-inspected until all areas do not exceed the measurement parameter threshold, and the quality inspection of the prefabricated bridge will be completed.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of any method described in claims 1-6 can be implemented.

Images