Steel structure welding quality detection method and storage medium

Abstract

The invention provides a steel structure welding quality detection method and a storage medium, and the method comprises the steps: generating a composite image # imgabs1 # marked with the type and position of a welding defect based on an unmarked real welding region image # imgabs0 # of a steel structure; the real steel structure welding image # imgabs2 # and the composite image # imgabs3 # marked with the welding defect type and position are combined into a steel structure welding image set # imgabs4 #; performing datamation processing on the steel structure welding image set # imgabs5 # to generate a deformable grid # imgabs6 # matched with the type of the welding defect and a preprocessing image set # imgabs7 #; and inputting the preprocessed image set # imgabs8 # into a pre-trained building engineering steel structure welding defect detection model so as to output welding defect characteristics of a welding seam area. The steel structure welding quality detection method is small in data dependence, high in welding defect recognition robustness and high in intelligent degree, and the intelligent level of constructional engineering steel structure welding quality detection can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of welding quality inspection for steel structures in construction engineering, and particularly relates to a method for inspecting the welding quality of steel structures and a storage medium. Background Art

[0002] The welding area is a potential failure location of steel structures in construction engineering. During the welding process, due to local heating and cooling of the material, non-uniform thermal expansion and cooling shrinkage exist in the welded part, which may cause welding defects at the weld, and further lead to problems such as a decline in the mechanical properties of the welded area of the steel structure. Therefore, it is necessary to inspect the welding quality of steel structures in construction engineering. Traditional inspection of the welding quality of steel structures mainly relies on manual visual inspection and experience judgment, with deficiencies such as strong subjectivity, low inspection efficiency, and limited evaluation accuracy. Currently, there are already some technical solutions for inspecting the welding quality of steel structures in construction engineering. For example, magnetic particle, ultrasonic and other steel structure weld flaw detection systems have the characteristics of strong intuitiveness, high sensitivity, and non-destructive detection. However, the inspection process still requires the full participation of operators, and the problems of labor and equipment costs have not been solved.

[0003] In addition, there are also weld quality inspection methods based on image processing and traditional computer vision technology. Images of the weld area are obtained through devices such as cameras, and the images are processed such as gray scale and grid division. Subsequently, computer vision technology is used to compare the optimized gray scale image or unit image block with a preset standard image, so as to realize the inspection of the welding quality of steel structures. However, this technical means still has the following deficiencies: it relies on high-quality real steel structure welding defect data, and it is difficult to establish a preset standard image library; the feature extraction methods used perform poorly when facing complex welding defect morphologies, and the feature description robustness is poor; the welding quality assessment uses static algorithm tuning, and the overall scalability and applicability of the technology are limited. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for inspecting the welding quality of steel structures and a storage medium.

[0005] To solve the above problems, the present invention provides a method for inspecting the welding quality of steel structures, the method comprising: Obtaining images of a plurality of unlabeled real steel structure welding areas from an image acquisition device ; Based on the images of the unlabeled real welding areas of the steel structure , generating a synthetic image annotating the type and position of welding defects ; Combining the real steel structure welding images and the synthetic image annotating the type and position of welding defects into a steel structure welding image set ; For the steel structure welding image set Perform data processing to generate a deformable grid matching the type of welding defect , and a preprocessed image set ; Input the preprocessed image set into a pre-trained building engineering steel structure welding defect detection model to output the welding defect characteristics in the weld area; By comparing the welding defect characteristics in the weld area with a preset standard defect database, annotate the information on the type, location, and physical size of the welding defects output by the detection model on the real steel structure welding image .

[0006] Furthermore, in the above method, based on the unannotated real welding area image of the steel structure , generate a synthetic image with the type and location of the annotated welding defects , including: Perform semantic segmentation on the obtained unannotated real welding area image of the steel structure to generate a weld part mask to accurately distinguish the weld area and the base metal part in the steel structure welding image, calculate the mask coverage rate based on the weld area ; evaluate the recognition degree of the weld area based on the mask coverage rate , and when the mask coverage rate is less than the preset weld recognition threshold , reject the image ; Among them, W is the pixel width of the unannotated real welding area image of the steel structure , H is the pixel height of the unannotated real welding area image of the steel structure ; For the unannotated real welding area image of the steel structure that is greater than or equal to the preset weld recognition threshold , use a multi-modal pre-trained model to perform prompt reverse inference on the weld area to obtain the result of prompt reverse inference; Based on the result of prompt reverse inference, associate the type label of the welding defect with the corresponding area of the weld part mask ; Based on the weld part mask associated with the defect category label ​The corresponding area, and insert the rank into the generative model LoRA layer of, training objective function ; Use the objective function To control and adjust the image generated by training the generative model; At the same time, use the conditional control loss function and the weld part mask Combine for geometric constraints to control that different types of welding defects in the image generated by training only appear in the weld area, that is, obtain a synthetic image annotating the type and location of welding defects , ; Among them, Is the target noise, which is a training process variable; Is the noise estimate, Is the time step, Is the time step The latent variable at, Is the type label of the welding defect; Is the regularization parameter, Is the mask total variation regularization term; Is the image generated by training, that is, a synthetic image annotating the type and location of welding defects; Combine the real steel structure welding image And the synthetic image annotating the type and location of welding defects , and merge them into a steel structure welding image set .

[0007] Furthermore, in the above method, perform semantic segmentation on the obtained unannotated real steel structure welding area image , including: Adopt a superpixel segmentation algorithm (such as improved SLIC), and perform semantic segmentation on the unannotated real steel structure welding area image in the HSV color space. The superpixel block merging method is: ; ; Among them, Is the measurement value of the merging decision, Is the color distance between superpixel blocks, Is the Euclidean distance between superpixel blocks, And Are the corresponding standard deviations respectively; Is the preset segmentation threshold, Is the basic threshold, which can be taken as 0.8~1.2 for the steel structure of building engineering, Is the scaling coefficient, is the image complexity factor, calculated from the Gray Level Co - occurrence Matrix (GLCM) considering contrast, energy, and entropy.

[0008] Furthermore, in the above - mentioned method, the training objective function , includes: Introduce an anisotropy term in the objective function to constrain the smoothness within the masked area of the training - generated image : ; Among them, is the horizontal - direction gradient operator, is the vertical - direction gradient operator; the anisotropy term suppresses the jagged artifacts of the welding defects in the training - generated image , while retaining the tortuous morphology of the real cracks. In addition, the stress concentration factor of the generated cracks conforms to the laws of elastic fracture mechanics.

[0009] Furthermore, in the above - mentioned method, use the conditional control loss function in combination with the weld - part mask for geometric constraints, including: Adopt the ControlNet loss function, input the weld - part mask into the encoder of ControlNet to generate a conditional feature map , and perform channel - weighted fusion with the main - branch feature map in the generative model: ; Among them, and are weight matrices, is the Sigmoid activation function.

[0010] Furthermore, in the above - mentioned method, while using the conditional control loss function in combination with the weld - part mask ( for geometric constraints, it also includes: Based on the micro - surface theory, randomly sample the surface roughness and metallicity of the unlabeled real - welding - area image of the steel structure during the image - generation stage ; Based on the surface roughness and metallicity , calculate the light - reflection characteristics of each pixel of the labeled real - welding - area image of the steel structure , and based on the light - reflection characteristics of each pixel, obtain the generated image , to control the generated images during training to be more physically reasonable: ; Among them, is the normal distribution function, is the Fresnel term, is a model considering the reduction of effective light caused by mutual occlusion between micro-surfaces, is the normal of the object surface, is the incident light direction, is the viewing direction, is the half-angle vector; represents the incident light direction, that is, the direction in which the light ray shoots towards the object surface; represents the outgoing light direction, that is, the direction in which the light ray shoots out after being reflected from the object surface.

[0011] Furthermore, in the above method, the steel structure welding image set is processed data-wise to generate a deformable mesh matching the type of welding defect , and a preprocessed image set , including: Decouple the images in the steel structure welding image set in the CIE-Lab space, separating the luminance information and chromaticity information; obtain the L channel based on the luminance information; obtain the a and b channels based on the chromaticity information; Introduce the parameter of the non-linear gamma function to correct the L channel, reducing the influence of ambient light: ; Perform chromaticity equalization on the a and b channels to eliminate the interference of ambient color temperature on the weld image; Based on the corrected L channel and the chromaticity equalized a and b channels, obtain the corrected image; establish an initial uniform grid in the virtual coordinate system, where the grid cell size is 8×8 pixels; Calculate the gray gradient covariance matrix within each grid unit of the initial uniform grid; based on the gray gradient covariance matrix within each grid unit, optimize the vertex coordinates of the grid with the goal of minimizing the grid deformation energy function , so that the deformable mesh matches the type of welding defect; Deformable grid Variation of relevant parameters Merge with the corresponding steel structure welding image set to generate a preprocessed image set .

[0012] Furthermore, in the above method, introduce the parameters of the nonlinear gamma function to L correct the channels and reduce the influence of ambient light, including: Adopt double-factor dynamic gamma adjustment, through the defect density factor D and the light influence factor T to jointly regulate the parameters of the gamma function : ; ; ; wherein, is a mask with the type label of welding defects, is L the median brightness of the channel, is the Gaussian weight function, is the image L-channel coordinate where the brightness value is located; e is the natural constant; is the hyperbolic tangent function.

[0013] Furthermore, in the above method, for a and b channels, perform chromaticity equalization to eliminate the interference of ambient color temperature on the weld image, and ensure that the defect detection step only depends on the physical features in the welding image, including: Perform statistical normalization on the chromaticity channel values a and b to eliminate color temperature shift: ; ; wherein, and are the mean and standard deviation of the input image channels respectively, and are the corresponding chromaticity statistical parameters under standard illumination, and a and b are the chromaticity channel values in the image color space.

[0014] Furthermore, in the above method, the grid deformation energy function adopts: ; wherein, is the total number of grids, , and are weight coefficients used to constrain the smoothness of the chrominance channel and avoid noise interference; is used to prevent excessive mesh adjustment, and the final mesh vertex displacement value is: ; Among them, and are the displacement amounts of the mesh vertices respectively.

[0015] Furthermore, in the above method, the preprocessed image set is input into a pre-trained building engineering steel structure welding defect detection model to output welding defect features in the weld area, including: In the building engineering steel structure welding defect detection model, a multi-task neural network IDN is included. This multi-task neural network IDN includes three sub-tasks: defect detection, defect quantification, and feature alignment; the building engineering steel structure welding defect detection model uses a weighted multi-task loss function : ; Among them, , and are weight coefficients respectively, is the defect detection loss, is the defect quantification loss, is the feature alignment loss; the weight coefficients , and are initially set to 1.0, 0.5, and 0.2 according to the sub-task priorities, and the weight values are dynamically adjusted during the training process of the building engineering steel structure welding defect detection model by monitoring the loss degrees of each sub-task to prevent a single sub-task from dominating the training process: ; Among them, is the cumulative loss mean of a single sub-task; The deformable convolution is applied to the feature extraction layer of the backbone network of the multi-task neural network IDN, and the relevant parameter variation of the deformable mesh is used as the offset of the deformable convolution; The feature maps extracted from the preprocessed image set by the backbone network of the multi-task neural network IDN are shared by the task heads of the three sub-tasks of defect detection, defect quantification, and feature alignment of the multi-task neural network IDN; The losses of the task headers for the three subtasks of defect detection, defect quantification, and feature alignment are simultaneously backpropagated, continuously adjusting the parameters of the multi-task neural network IDN, jointly updating the backbone network weights, and forcing the model to focus on the consistency of defect local details, global morphology, and physical features; the regional welding defect features include: the detected and recognized welding defect types, location information, and defect physical parameters.

[0016] Further, in the above method, the relevant parameter variations of the deformable grid are used as the offsets of the deformable convolution, including: Taking the relevant parameter variations of the deformable grid as prior knowledge to guide the offset direction of the convolution kernel. For a single sampling point of the convolution kernel, the offset calculation method is: ; where is the offset in the x direction, is the parameter variation of the deformable grid in the x direction, is the size of the feature map extracted by the backbone network in the x direction, is the relevant parameter variation in the x direction; is the offset in the y direction, is the parameter variation in the y direction, and are the sizes of the feature map and the relevant parameter variation in the y direction respectively.

[0017] Further, in the above method, the defect detection loss function is used to ensure that the model accurately classifies the defect types and locates their positions: ; ; ; where is the classification loss function, is the class weight, is the predicted probability of the model for the positive class, which is used to suppress the gradients of easily classified samples; is the localization loss function, which is used to measure the degree of boundary overlap, is the Euclidean distance, is the length of the diagonal of the smallest closed region, is a preset parameter, is the aspect ratio penalty term; and are the weighting coefficients.

[0018] Furthermore, in the above method, the defect quantization loss function uses the quantization parameter as the threshold boundary to maintain a smooth transition between the qualified and unqualified sample regions: ; wherein, is the physical feature value of the defect predicted by the model, is the control parameter, .

[0019] Furthermore, in the above method, the feature alignment loss function adopts the cosine similarity loss based on HOG and LBP: ; wherein, is the feature extraction function for capturing the texture and geometric characteristics of the welding defect; is the grid feature corresponding to the welding defect area predicted by the model, is the grid feature of the preset standard welding defect area.

[0020] According to another aspect of the present invention, there is also provided a computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: execute the method as described in any one of the above.

[0021] Compared with the prior art, the present invention obtains images of a number of unlabeled real steel structure welding areas from an image acquisition device ; based on the unlabeled real welding area images of the steel structure , generate synthetic images annotating the types and positions of welding defects ; combine the real steel structure welding images and the synthetic images annotating the types and positions of welding defects into a steel structure welding image set ; execute a physically enhanced image preprocessing process, and through multispectral optimization and spatial transformation, perform data processing on the steel structure welding image set to generate a deformable grid matching the type of welding defect , as well as a preprocessed image set ; use the preprocessed image set Input the pre-trained building engineering steel structure welding defect detection model to output the welding defect features in the weld area, including: the detected and identified welding defect types, location information, and defect physical parameters; by comparing the welding defect features in the weld area with the preset standard defect database, annotate the information of the type, location, and physical size of the welding defects output by the detection model on the real steel structure welding image This invention is a steel structure welding quality detection method with low data dependence, strong robustness in welding defect recognition, and high intelligence, which can improve the intelligent level of building engineering steel structure welding quality detection. Description of the Drawings

[0022] Figure 1 is the flowchart of the steel structure welding quality detection method based on generative artificial intelligence according to an embodiment of the present invention; Figure 2 is the generation example diagram of the steel structure welding image set according to an embodiment of the present invention ; Figure 3 is the example diagram of the preprocessing of the steel structure welding image according to an embodiment of the present invention; Figure 4 is the example diagram of the detection and identification of steel structure welding defects according to an embodiment of the present invention; Figure 5 is the architecture diagram of the steel structure welding quality detection system based on generative artificial intelligence according to an embodiment of the present invention. Detailed Embodiment

[0023] The present invention will be further described in detail below with reference to the accompanying drawings.

[0024] In a typical configuration of the present application, the terminal, the devices of the service network, and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces, and memories.

[0025] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer-readable medium.

[0026] A computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media such as modulated data signals and carrier waves.

[0027] As Figure 1 shown, the present invention provides a method for detecting the welding quality of a steel structure, including: Step S1, obtaining image information of the steel structure welding area: Obtaining images of a number of unlabeled real steel structure welding areas from an image acquisition device ; Here, the image acquisition device is such as a camera, a camera, etc.; Step S2, generating a steel structure welding image set: Based on the unlabeled real steel structure welding area images , generating synthetic images with labeled welding defect types and positions ; Combining the real steel structure welding images and the synthetic images with labeled welding defect types and positions into a steel structure welding image set ; Here, based on the unlabeled real steel structure welding area images , generating synthetic images with labeled welding defect positions , is based on the unlabeled real steel structure welding area images , generating synthetic images that conform to physical laws and have accurate labels of welding defect types and positions ; Preferably, as Figure 2 shown, step S2, based on the unlabeled real steel structure welding area images , generating synthetic images with labeled welding defect types and positions , including: Step S21: Perform semantic segmentation on the obtained unannotated real steel structure welding area image to generate a weld part mask to accurately distinguish the weld area and the base metal part in the steel structure welding image. Based on the weld area , calculate the mask coverage rate ; based on the mask coverage rate , evaluate the recognition degree of the weld area. When the mask coverage rate is less than the preset weld recognition threshold , eliminate this image and label the image as "low quality".

[0028] ; where W is the pixel width of the unannotated real steel structure welding area image , and H is the pixel height of the unannotated real steel structure welding area image .

[0029] Step S22: For the unannotated real steel structure welding area image greater than or equal to the preset weld recognition threshold , use a multi-modal pre-trained model to perform prompt backpropagation on the weld area to obtain the result of prompt backpropagation; here, the multi-modal pre-trained model is such as the CLIP model, etc.; Step S23: Based on the result of prompt backpropagation, associate the type label of the welding defect with the corresponding area of the weld part mask ; here, the defect category label can be preset according to the technical requirements of the actual project; Step S24: Based on the corresponding area of the weld part mask associated with the defect category label, and insert the LoRA layer of rank in the generative model to train the objective function ; use the objective function to control and adjust the image generated by training the generative model; at the same time, use the conditional control loss function in combination with the weld part mask for geometric constraint to control that different types of welding defects in the image generated by training only appear in the weld area, that is, obtain a synthetic image labeling the type and position of the welding defect .

[0030] ; Among them, is the target noise, which is a training process variable; is the noise estimation, is the time step, is the time step latent variable at, is the type label of the welding defect; is the regularization parameter, is the masked total variation regularization term; is the image generated during training, that is, the synthetic image annotating the type and location of the welding defect.

[0031] Here, the generative model is such as Stable Diffusion, etc.; the conditional control loss function is such as ControlNet, etc.; the types of welding defects include: respective irregular welding defects such as cracks, non-uniform pores, etc.

[0032] Step S25, combine the real steel structure welding image and the synthetic image annotating the type and location of the welding defect to form a steel structure welding image set .

[0033] Here, the types of welding defects include: respective irregular welding defects such as cracks, non-uniform pores, etc.

[0034] Step S3, preprocess the steel structure welding image: Execute the physical enhancement-based image preprocessing (PAP) process, and through multispectral optimization and spatial transformation, perform data processing on the steel structure welding image set to finally generate a deformable grid that highly matches the type of welding defect , and the preprocessed image set .

[0035] Preferably, as Figure 3 shown, step S3 includes: Step S31, decouple the images in the steel structure welding image set in the CIE-Lad space, separate the luminance information and chromaticity information; obtain the L channel based on the luminance information; obtain the a and b channels based on the chromaticity information to avoid information cross-interference.

[0036] Step S32, introduce the parameter of the nonlinear gamma function to correct the L channel and reduce the influence of ambient light: ; Here, represents the brightness value of the image after gamma correction at the coordinate position. By performing gamma transformation on the L channel (representing brightness) of the original image, the influence of ambient light on the image brightness is reduced, making the image brightness distribution more reasonable, which helps to present image details more clearly. For example, in the steel structure welding image, key information such as welding defects can be highlighted.

[0037] is the brightness value of the original image at the coordinate position, is the parameter of the gamma function, represents performing a power operation transformation on the original brightness value according to the gamma function rule, so as to obtain the corrected brightness value, that is , thereby realizing the adjustment and optimization of the image brightness.

[0038] Step S33, perform chromaticity equalization on a and b channels to eliminate the interference of ambient color temperature on the weld image, ensuring that the defect detection step only depends on the physical characteristics in the welding image.

[0039] Step S34, based on the corrected L channel and the chromaticity equalized a and b channels, obtain the corrected image; establish an initial uniform grid in the virtual coordinate system based on the corrected image, where the grid cell size is 8×8 pixels; Step S35, calculate the gray gradient covariance matrix within each grid unit in the initial uniform grid; based on the gray gradient covariance matrix within each grid unit, optimize the grid vertex coordinates with the goal of minimizing the grid deformation energy function , so that the final deformable grid highly matches the type of welding defect.

[0040] Here, the types of welding defects include: various irregular welding defects such as cracks and non-uniform pores. The final deformable grid highly adapts to the types of irregular welding defects.

[0041] Step S36, merge the change amount of the relevant parameters of the deformable grid with the corresponding steel structure welding image set to generate the preprocessed image set .

[0042] Here, the deformable mesh records the variation of relevant parameters during the process from the initial uniform mesh to the final deformable mesh, including the variation of parameters such as the vertex coordinates of the mesh. These variations are to make the mesh highly adaptable to the irregular welding defects (such as cracks, non-uniform pores, etc.) in the welding image.

[0043] During specific merging, mapping can be performed based on coordinates; the deformable mesh records the variation of parameters such as the vertex coordinates of the mesh , and these variations can be mapped to the coordinate system of the corresponding steel structure welding image set .

[0044] Step S4, Steel structure welding defect detection and recognition: Input the preprocessed image set into the pre-trained steel structure welding defect detection model for building engineering to output the welding defect features in the weld area, including: the detected and recognized welding defect types, location information, defect physical parameters, etc.

[0045] Preferably, before the steel structure welding defect detection and recognition, it also includes training the steel structure welding defect detection model for building engineering: Step S41, Set up the steel structure welding defect detection model for building engineering, which includes a multi-task neural network IDN (Interpretability Defect Recognition Network), including three sub-tasks: defect detection (locating and classifying steel structure welding defects), defect quantification (calculating physical parameters such as the diameter and length of welding defects), and feature alignment (ensuring that the detected welding defects match the grid features of the preset standard defects).

[0046] The steel structure welding defect detection model for building engineering adopts a weighted multi-task loss function : ; Among them, , and are the weight coefficients respectively, is the defect detection loss, is the defect quantification loss, is the feature alignment loss; the weight coefficients , and are initially set to 1.0, 0.5, and 0.2 respectively according to the sub-task priorities, and the weight values are dynamically adjusted during the training process of the steel structure welding defect detection model for building engineering by monitoring the loss degrees of each sub-task to prevent a single sub-task from dominating the training process: ; Among them, is the average cumulative loss of a single subtask.

[0047] Step S42, as Figure 4 shown, apply Deformable Conv to the feature extraction layer of the backbone network of the multi-task neural network IDN, and accept the deformable grid associated parameter variation as the offset of the deformable convolution, which can effectively improve the defect detection speed.

[0048] Step S43, the feature maps extracted by the backbone network of the multi-task neural network IDN from the preprocessed image set are shared by the task heads of the three subtasks of defect detection, defect quantification and feature alignment of the multi-task neural network IDN to improve data utilization.

[0049] Step S44, the losses of the task heads of the three subtasks of defect detection, defect quantification and feature alignment are backpropagated simultaneously, continuously adjusting the parameters of the multi-task neural network IDN, jointly updating the backbone network weights, forcing the model to focus on the consistency of defect local details, global morphology and physical features, improving the interpretability of the detection results, so as to obtain a pre-trained building engineering steel structure welding defect detection model.

[0050] Here, during the training process, by backpropagating the losses of the three task heads of defect detection, defect quantification and feature alignment simultaneously, the parameters of the multi-task neural network IDN, including the backbone network weights, can be continuously adjusted. This can enable the model to learn how to better focus on the consistency of defect local details, global morphology and physical features, thereby improving the detection performance of the model and the interpretability of the detection results.

[0051] Step S5, steel structure welding quality assessment: By comparing the welding defect features in the weld area with the preset standard defect database, annotate the information of the type, location and physical size of the welding defects output by the detection model on the real steel structure welding image ; Based on the annotation of the type, location and physical size of the welding defects on the real steel structure welding image generate a steel structure welding quality inspection report, and count the quantity and size range of a certain type of welding defect.

[0052] In particular, in step S2, the following methods can be used for image semantic segmentation, objective function training and geometric constraints: Step S21, for the obtained unannotated real steel structure welding area image , perform semantic segmentation, including: Adopt a superpixel segmentation algorithm such as improved SLIC (Simple Linear Iterative Clustering) to perform semantic segmentation on the unannotated real steel structure welding area image in the HSV color space Performing semantic segmentation can avoid the welding defects sensitive to light in the real welding image from being wrongly ignored. The superpixel block merging method is: ; ; Among them, is the measurement value for the merging decision, is the color distance between superpixel blocks, is the Euclidean distance between superpixel blocks, and are the corresponding standard deviations respectively; is the preset segmentation threshold, is the basic threshold, which can be taken as 0.8 - 1.2 for steel structures in building engineering, is the scaling factor, is the image complexity factor, which is calculated by the gray - level co - occurrence matrix (GLCM) considering contrast, energy, and entropy. This method can generate a high - precision binary mask of the weld area through color - texture double constraints .

[0053] Step S24, train the objective function , including: Introduce the anisotropic term into the objective function to constrain the smoothness within the mask area of the training - generated image : ; Among them, is the horizontal direction gradient operator, is the vertical direction gradient operator; the anisotropic term suppresses the jagged artifacts of welding defects in the training - generated image while retaining the zigzag shape of real cracks. In addition, the stress concentration factor of the generated crack conforms to the laws of elastic fracture mechanics.

[0054] Step S24, use the conditional control loss function to combine with the weld part mask ( for geometric constraints, including: Adopt the ControlNet loss function and use the weld part mask Input the encoder of ControlNet to generate conditional feature maps and perform channel-wise weighted fusion with the feature maps of the main branch in the generative model : ; Among them, and are weight matrices, is the Sigmoid activation function

[0055] Preferably, in step S24, while using the conditional control loss function in combination with the weld part mask for geometric constraints, it further includes: Step S25, based on the microfacet theory, randomly sample the surface roughness and metallicity of the unannotated real welded area images of steel structures during the image generation stage ; Based on the surface roughness and metallicity , calculate the light reflection characteristics for each pixel of the annotated real welded area images of steel structures , and based on the light reflection characteristics of each pixel, obtain the generated image to control the generated images during training to be more physically reasonable: ; Among them, is the normal distribution function, is the Fresnel term (the change in reflectivity of light at different incident angles), is the model considering the reduction of effective light caused by mutual occlusion between microfacets, is the surface normal of the object, is the incident light direction, is the viewing direction, is the half-angle vector

[0056] Here, the ControlNet function can execute multiple constraint modules simultaneously. Here, additional constraint modules are activated based on the microfacet-related formulas in the ControlNet function

[0057] Surface roughness and metallicity are important material property parameters of the object surface. Surface roughness affects the scattering of light on the micro-surface. A rough surface causes more severe light scattering, while a smooth surface has more regular reflection. Metallicity determines the reflection and absorption characteristics of the material to light. Metal materials have a high reflectivity, and non-metal materials have a greater proportion of light absorption or refraction. Randomly sampling these parameters provides the material basic information of the micro-surface at the position of each pixel for subsequent calculation of the light reflection characteristics of each pixel.

[0058] is the Bidirectional Reflectance Distribution Function (BRDF), which is used to describe the reflection characteristics of light on the object surface. Among them, represents the incident light direction, that is, the direction in which the light shoots towards the object surface; represents the outgoing light direction, that is, the direction in which the light shoots out after being reflected from the object surface.

[0059] In the formula for calculating the light reflection characteristics, such as the bidirectional reflectance distribution function in the related calculations, surface roughness and metallicity will affect the values of some of the functions. For example, surface roughness will affect the normal distribution function , and thus affect the calculation result of the overall light reflection; metallicity will act on the Fresnel term and change the calculation of the reflectivity of light at different incident angles.

[0060] Specifically, in step S3, the following methods can be adopted in gamma adjustment, chromaticity equalization, and deformable mesh generation: In step S32, introduce the parameter of the non-linear gamma function to correct the L channel and reduce the influence of ambient light, including: Adopt double-factor dynamic gamma adjustment, and jointly regulate the parameter D of the gamma function through the defect density factor T and the light influence factor to achieve the effect of enhancing the defect-sensitive area and suppressing the highlight area: ; ; ; Among them, is the mask with the type label of the welding defect, is LChannel median brightness, is the Gaussian weight function.

[0061] Here, is the mask with the type label of welding defects. By dividing the sum of it by the sum of the total mask the relative density of the defect in the image is obtained, reflecting the distribution degree of the defect.

[0062] is the coordinate of the L channel (brightness channel) of the image at the brightness value, is the median brightness of the L channel, is the Gaussian weight function.

[0063] ;

[0064] First, calculate the weighted difference between the brightness of each pixel and the median brightness, and then take the median to measure the influence degree of uneven illumination on the image.

[0065] e is the natural constant; is the hyperbolic tangent function.

[0066] Step S33, for a and b channels for chromaticity equalization to eliminate the interference of ambient color temperature on the weld image and ensure that the defect detection step only depends on the physical features in the welding image, including: Perform statistical normalization on the chromaticity channel values a and b to eliminate the color temperature shift: ; ; Among them, and are the mean and standard deviation of the input image channels respectively, and are the corresponding chromaticity statistical parameters under standard illumination.

[0067] Here, a and b are the chromaticity channel values in the image color space. Generally, in the Lab color space, L represents the brightness channel, the a channel represents the color component from green to red, and the b channel represents the color component from blue to yellow.

[0068] In step S35, the grid deformation energy function adopts: ; Among them, is the total number of grids, , and are weight coefficients, used to constrain the smoothness of the chrominance channel and avoid noise interference; used to prevent excessive mesh adjustment. The final mesh vertex displacement value is: ; wherein, and are respectively the displacement amounts of the mesh vertices.

[0069] Specifically, in step S4, the following method is used to determine the deformable convolution offset, defect detection loss, defect quantization loss, and feature alignment loss: Step S42, accept the relevant parameter variation of the deformable mesh as the offset of the deformable convolution, including: Take the relevant parameter variation of the deformable mesh as prior knowledge to guide the offset direction of the convolution kernel. For a single sampling point of the convolution kernel, the offset calculation method is: ; wherein, and are the sizes of the feature maps extracted by the backbone network, and are the relevant parameter variations sizes.

[0070] Here, is the x - direction offset, is the parameter variation of the deformable mesh in the x - direction, is the size of the feature map extracted by the backbone network in the x - direction, is the size of the relevant parameter variation in the x - direction. By scaling the mesh parameter variation according to the ratio of the feature map and the mesh size, the offset of each sampling point of the convolution kernel in the x - direction is obtained. is the y - direction offset, is the parameter variation in the y - direction, , are respectively the sizes of the feature map and the relevant parameter variation (mesh parameter) in the y - direction. The principle is the same as that in the x - direction and is used to determine the offset of the sampling points of the convolution kernel in the y - direction. In this way, the convolution kernel can sample more accurately on the image according to the calculated offset, improving the ability to extract target features such as welding defects.

[0071] In step S41, the defect detection loss function is used to ensure that the model accurately classifies the defect types and locates their positions: ; ; ; wherein, is the classification loss function (Focal Loss), is the class weight, is the prediction probability of the model for the positive class, which is used to suppress the gradient of easily classified samples; is the localization loss function (CIoU Loss), which is used to measure the degree of boundary overlap, is the Euclidean distance, is the diagonal length of the smallest closed region, is a preset parameter, is the aspect ratio penalty term; and are the weighting coefficients.

[0072] In step S41, the defect quantization function constrains the model through the quantization parameter so that the physical characteristics of the defects predicted by the model are more in line with the actual laws, The quantization parameter is determined according to the current national standard GB50661-2011 "Code for Welding of Steel Structures", such as the pore diameter, crack length, etc.

[0073] The defect quantization loss function uses the quantization parameter as the threshold boundary to maintain a smooth transition between the qualified and unqualified sample regions: ; wherein, is the physical characteristic value of the defect predicted by the model, is the control parameter, .

[0074] Here, as the predicted value exceeds the threshold , the loss increases exponentially, and continues to control the growth rate of the loss, indicating that the higher the degree of non-conformance, the greater the loss.

[0075] In step S41, feature alignment can solve the problem of "black box" decision-making of traditional deep learning models and improve the credibility and generalization ability of the detection results. The feature alignment loss function , adopt the cosine similarity loss based on HOG (Histogram of Oriented Gradients) and LBP (Local Binary Pattern): ; Among them, is a feature extraction function, used to capture the texture and geometric characteristics of welding defects; is the grid feature corresponding to the area of the welding defect predicted by the model, is the grid feature of the preset standard welding defect area.

[0076] The present invention is a steel structure welding quality detection method with low data dependence, strong robustness in welding defect recognition, and high intelligence, which can improve the intelligent level of steel structure welding quality detection in construction projects.

[0077] As Figure 5 shown, for the steel structure welding quality detection system based on generative artificial intelligence, its system composition architecture is as Figure 5 shown, and it is characterized in that the system includes the following core modules: An image acquisition module, used to acquire the image of the steel structure welding area obtained by the image acquisition device; A welding defect data generation module, that is, a generative steel structure welding defect data engine (GDE), which can generate physically reasonable defect images of the steel structure welding area based on an improved generative model (such as Stable Diffusion), and merge real steel structure welding images and the images generated during training , to form a steel structure welding image set .

[0078] An image preprocessing module, which can perform physically enhanced image preprocessing (PAP), and perform data processing on the steel structure welding image set , including L channel gamma correction and a / b channel chromaticity equalization; generate a deformable grid covering the weld area in the virtual coordinate system , to obtain a preprocessed image set .

[0079] A defect detection module, based on an interpretable defect recognition network (IDN), performs welding defect detection and recognition on the preprocessed image data.

[0080] A quality assessment module, performs quantitative assessment on the results of defect detection, outputs a detection report that meets the construction engineering process standards, and gives a welding quality assessment of the steel structure weld area in the report.

[0081] For the detailed content of each device embodiment of the present invention, reference may be specifically made to the corresponding part of each method embodiment, which will not be elaborated herein.

[0082] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

[0083] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and similar devices. Additionally, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.

[0084] In addition, a part of the present invention can be applied as a computer program product. For example, computer program instructions, when executed by a computer, can call or provide the methods and / or technical solutions according to the present invention through the operation of the computer. The program instructions for calling the methods of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in the working memory of a computer device that runs according to the program instructions. Herein, an embodiment according to the present invention includes a device that includes a memory for storing computer program instructions and a processor for executing the program instructions. When the computer program instructions are executed by the processor, the device is triggered to run based on the methods and / or technical solutions according to the foregoing multiple embodiments of the present invention.

[0085] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the apparatus claims can also be implemented by one unit or device through software or hardware. The words such as "first" and "second" are used to denote names and do not represent any particular order.

Claims

1. A method for detecting the welding quality of a steel structure, characterized in that, Including: Obtain images of several unannotated real steel structure welding areas from an image acquisition device ; Based on the unannotated images of the real welding areas of steel structures , generate synthetic images with annotated welding defect types and locations ; Combine the real steel structure welding images and the synthetic images with annotated welding defect types and locations into a steel structure welding image set ; Perform data processing on the steel structure welding image set to generate a deformable mesh matching the type of welding defect , as well as a preprocessed image set ; Input the preprocessed image set into a pre-trained welding defect detection model for building engineering steel structures to output the welding defect features in the weld area; By comparing the welding defect characteristics in the weld area with the preset standard defect database, the information on the type, location, and physical size of the welding defects output by the detection model is annotated on the real steel structure welding image above.

2. The steel structure welding quality inspection method according to claim 1, characterized in that, Based on the unannotated images of the actual welded areas of steel structures , generate synthetic images with the types and locations of the annotated welding defects , including: For the obtained unlabeled real steel structure welding area image , perform semantic segmentation to generate a weld part mask to accurately distinguish the weld area and the base metal part in the steel structure welding image. Based on the weld area , calculate the mask coverage rate ; Based on the mask coverage rate , evaluate the recognition degree of the weld area. When the mask coverage rate is less than the preset weld recognition threshold , reject this image Among them, W is an image of the real steel structure welding area without annotation is the pixel width of H is an image of the real steel structure welding area without annotation is the pixel height; For an unannotated real steel structure welding area image greater than or equal to a preset weld recognition threshold , a multi-modal pre-trained model is used to perform prompt reverse inference on the weld area to obtain the result of prompt reverse inference; ​ Based on the results deduced from the prompts, associate the type labels of welding defects with the corresponding regions of the weld part mask ; Based on the weld part mask associated with the defect category label for the corresponding region, and insert a LoRA layer with rank into the generative model, and the training objective function ; use the objective function to control and adjust the image generated by the training of the generative model; at the same time, use the conditional control loss function in combination with the weld part mask to perform geometric constraints, so as to control that different types of welding defects in the image generated by training only appear in the weld area, that is, a synthetic image annotating the type and position of the welding defects is obtained , wherein, is the target noise, which is a training process variable; is the noise estimation, is the time step, is the time step of the latent variable, is the type label of the welding defect; is the regularization parameter, is the masked total variation regularization term; is the training-generated image, that is, the synthetic image annotating the type and location of the welding defect; The real steel structure welding images and the synthetic images annotating the types and positions of welding defects are combined into a steel structure welding image set .

3. The steel structure welding quality detection method according to claim 2, characterized in that, For the obtained unannotated real steel structure welding area image , perform semantic segmentation, including: Using the superpixel segmentation algorithm, perform semantic segmentation on the unannotated real steel structure welding area image in the HSV color space The superpixel block merging method is as follows: Among them, is the measurement value of the merging decision, is the color distance between superpixel blocks, is the Euclidean distance between superpixel blocks, and are the corresponding standard deviations respectively; is the preset segmentation threshold, is the basic threshold, which can be taken as 0.8 - 1.2 for the steel structure of building engineering, is the scaling factor, is the image complexity factor, which is calculated by the gray - level co - occurrence matrix considering contrast, energy and entropy.

4. The method for detecting the welding quality of a steel structure according to claim 2, characterized in that, Training objective function , including: Introduce anisotropy into the objective function to constrain the smoothness within the masked region of the training-generated image : Among them, is the horizontal direction gradient operator, is the vertical direction gradient operator; the anisotropy term suppresses the serrated artifacts of welding defects in the training-generated image while retaining the tortuous morphology of real cracks. In addition, the stress concentration factor of the generated cracks conforms to the laws of elastic fracture mechanics.

5. The steel structure welding quality detection method according to claim 2, characterized in that Use a conditional control loss function and the weld part mask Combine to perform geometric constraints, including: Adopt the ControlNet loss function and mask the weld part Input it into the encoder of ControlNet to generate a conditional feature map and perform channel weighted fusion with the feature map of the main branch in the generative model as follows: Among them, and is the weight matrix, is the Sigmoid activation function.

6. The steel structure welding quality detection method according to claim 2, characterized in that Using a conditional control loss function and the weld part mask While combining geometric constraints, it further includes: Based on the micro-surface theory, randomly sample unlabeled images of the real welded areas of steel structures during the image generation stage of the surface roughness and metallicity ; Based on the surface roughness and metallicity , calculate the light reflection characteristics for each pixel of the labeled images of the real welded areas of steel structures , and obtain the generated images based on the light reflection characteristics of each pixel to control the generated images during training to be more physically reasonable: ​ Among them, is the normal distribution function, is the Fresnel term, is a model that takes into account the reduction of effective light caused by the mutual occlusion between micro-surfaces, is the normal of the object surface, is the incident light direction, is the viewing direction, is the half-angle vector; represents the incident light direction, that is, the direction in which the light ray shoots towards the object surface; represents the outgoing light direction, that is, the direction in which the light ray shoots out after being reflected from the object surface.

7. The steel structure welding quality inspection method according to claim 2, characterized in that, For a steel structure welding image set Perform data processing to generate a deformable grid that matches the type of welding defect , and a preprocessed image set , including: For the steel structure welding image set The images in are decoupled in the CIE-Lab space to separate the luminance information and chromaticity information; based on the luminance information, L channel is obtained; based on the chromaticity information, a and b channels are obtained; Introduce the parameters of the non-linear gamma function For L Channel correction to reduce the influence of ambient light: ； Pair a and b Perform chromaticity equalization on the channels to eliminate the interference of the ambient color temperature on the weld image; Based on the corrected L channel and the chromaticity equalized a and b channel, a corrected image is obtained; an initial uniform grid is established in a virtual coordinate system based on the corrected image, where the grid cell size is 8×8 pixels; Calculate the gray gradient covariance matrix within each grid unit of the initial uniform grid ; Based on the gray gradient covariance matrix within each grid unit , optimize the grid vertex coordinates with the goal of minimizing the grid deformation energy function so that the deformable grid matches the type of welding defect; Deformable meshes Variation amounts of relevant parameters And the corresponding steel structure welding image sets Are merged to generate a preprocessed image set .

8. The steel structure welding quality inspection method according to claim 7, characterized in that, Introduce the parameters of the non-linear gamma function For L Channel correction is performed to reduce the influence of ambient light, including: Adopt dual-factor dynamic gamma adjustment, and through the defect density factor D and the light influence factor T jointly regulate the parameters of the gamma function : Among them, is a mask with a type label of welding defects, is L the median brightness of the channel, is the Gaussian weight function, is the coordinate of the L channel of the image at the brightness value; e is the natural constant; is the hyperbolic tangent function.

9. The steel structure welding quality inspection method according to claim 7, characterized in that Pair a and b Perform chromaticity equalization on the channels to eliminate the interference of the ambient color temperature on the weld image and ensure that the defect detection step only relies on the physical features in the welding image, including: Statistically normalize the chrominance channel values a and b to eliminate color temperature shift: Among them, and are the mean and standard deviation of the input image channels respectively, and are the corresponding chromaticity statistical parameters under standard illumination, and a and b are the chromaticity channel values in the image color space.

10. The steel structure welding quality inspection method according to claim 7, characterized in that, The mesh deformation energy function Adopts: Among them, is the total number of grids, , and are weight coefficients, used to constrain the smoothness of the chrominance channel and avoid noise interference; used to prevent excessive grid adjustment, and the final grid vertex displacement value is: Among them, and are the displacement amounts of the grid vertices respectively.

11. The steel structure welding quality inspection method according to claim 7, characterized in that, The preprocessed image set is input into a pre-trained welding defect detection model for building engineering steel structures to output the welding defect features in the weld area, including: In setting up a welding defect detection model for steel structures in construction projects, it includes a multi-task neural network IDN. The multi-task neural network IDN includes three sub-tasks: defect detection, defect quantification, and feature alignment. The welding defect detection model for steel structures in construction projects uses a weighted multi-task loss function : Among them, , and are weight coefficients respectively, is the defect detection loss function, is the defect quantization loss function, is the feature alignment loss; the weight coefficients , and are initially set to 1.0, 0.5 and 0.2 respectively according to the subtask priorities. During the training process of the building engineering steel structure welding defect detection model, the weight values are dynamically adjusted by monitoring the loss degrees of each subtask to prevent a single subtask from dominating the training process: Among them, is the cumulative loss mean for a single subtask; Apply deformable convolution to the feature extraction layer of the IDN backbone network of the multi-task neural network, and accept the deformable grid of the relevant parameter variation as the offset of the deformable convolution; The backbone network of the multi-task neural network IDN extracts feature maps from the preprocessed image set The feature maps extracted from are shared by the task heads of the three subtasks of the multi-task neural network IDN, namely defect detection, defect quantification, and feature alignment; The losses of the task heads of the three subtasks of defect detection, defect quantification, and feature alignment are simultaneously backpropagated, continuously adjusting the parameters of the multi-task neural network IDN, jointly updating the backbone network weights, and forcing the model to focus on the consistency of defect local details, global morphology, and physical features; The regional welding defect features include: the detected and identified welding defect types, location information, and defect physical parameters.

12. The steel structure welding quality detection method according to claim 11, characterized in that, Accept a deformable grid The change amount of the relevant parameters As the offset of the deformable convolution, including: Take the deformable grid The change amount of relevant parameters As prior knowledge to guide the offset direction of the convolutional kernel. For a single sampling point of the convolutional kernel, the offset calculation method is as follows: Among them, is the x-direction offset, is the parameter variation of the deformable grid in the x direction, is the size of the feature map extracted by the backbone network in the x direction, is the relevant parameter variation in the size in the x direction; is the y-direction offset, is the parameter variation in the y direction, , are respectively the sizes of the feature map and the relevant parameter variation in the y direction.

13. The steel structure welding quality detection method according to claim 11, characterized in that, Defect Detection Loss Function Used to ensure that the model accurately classifies the defect types and locates their positions: ； ； ； Among them, is the classification loss function, is the class weight, is the predicted probability of the model for the positive class, which is used to suppress the gradient of easily classified samples; is the localization loss function, which is used to measure the degree of boundary overlap, is the Euclidean distance, is the diagonal length of the smallest closed region, is a preset parameter, is the aspect ratio penalty term; and are the weighting coefficients.

14. The steel structure welding quality inspection method according to claim 11, characterized in that, Defect Quantification Loss Function Take the quantization parameter as the threshold boundary to maintain a smooth transition between the qualified and unqualified sample regions: Among them, is the defect physical feature value predicted by the model, is the control parameter, .

15. The steel structure welding quality detection method according to claim 10, characterized in that, Feature alignment loss function , using the cosine similarity loss based on HOG and LBP: Among them, is a feature extraction function used to capture the texture and geometric characteristics of welding defects; is the grid feature corresponding to the area of the welding defect predicted by the model, is the grid feature of the preset standard welding defect area.

16. A computer-readable storage medium having computer-executable instructions stored thereon, wherein, When the computer-executable instructions are executed by the processor, the processor is caused to: execute the method according to any one of claims 1 to 15.

Images