A steel structure weld defect detection method and system

Abstract

This invention belongs to the field of machine vision and non-destructive testing technology for steel structures, specifically relating to a method and system for detecting weld defects in steel structures. The method includes: acquiring an image of the weld in the steel structure to be inspected; extracting the initial centerline of the weld; determining a set of candidate points for the weld edge and solving for a non-uniform boundary line; defining the region enclosed by the boundary line as the target detection region; extracting the image within the target detection region; calculating local weld width information and local centerline curvature information; generating a local geometric topological feature map set with the same size as the original image; and feeding the weld image and the local geometric topological feature map set as multimodal inputs into a feature fusion convolutional neural network to output the category and location information of the weld defect. This invention enables the feature fusion convolutional neural network to comprehensively analyze visual and structural information, thereby effectively improving the accuracy of steel structure weld defect detection.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and non-destructive testing technology for steel structures. More specifically, this invention relates to a method and system for detecting weld defects in steel structures. Background Technology

[0002] Welding is the core process for connecting steel structure components. The quality of the weld directly determines the overall load-bearing safety and operational reliability of the steel structure. Therefore, accurate and efficient non-destructive testing of the weld is a key link in the entire process of steel structure production and operation.

[0003] In actual engineering conditions, the acquired weld images are easily affected by factors such as steel structure surface corrosion, uneven lighting, and the variable shape of the weld itself, resulting in low contrast and blurred boundaries between the weld area and the background area. Traditional image processing methods based on threshold segmentation and edge detection operators struggle to accurately locate the true boundary of the weld under complex interference, and are prone to misidentifying background noise and imaging artifacts as weld areas. This leads to an increased false alarm rate and missed detection rate in defect detection, failing to meet the accuracy requirements of engineering inspection.

[0004] With the development of deep learning technology, weld defect detection methods based on convolutional neural networks can learn defect features from original weld images in an end-to-end manner, which improves the accuracy of defect detection to some extent. However, existing methods of this kind usually treat the weld image as a regular two-dimensional image input to the network, relying on the network to autonomously learn pixel-level texture and color features.

[0005] The geometric and topological features of the weld, such as the centerline trend, local width variations, and boundary curvature, are crucial for distinguishing between real defects and normal weld fluctuations. For example, the distribution of crack defects is directly related to the direction of the weld centerline, and the severity of defects such as porosity and incomplete penetration is directly related to their relative positions within the weld cross-section. Existing methods lack the ability to explicitly model and utilize this prior structural information, resulting in limited ability to identify geometrically sensitive defects. When faced with complex morphologies and small defects similar to the background texture, the detection stability is insufficient, easily leading to misjudgments. Summary of the Invention

[0006] To address the technical problem that existing technologies rely solely on pixel texture features and lack explicit structural information modeling, resulting in limited detection accuracy and susceptibility to misjudgments in complex backgrounds and scenarios with minute defects, this invention provides a method and system for detecting defects in steel structure welds.

[0007] In a first aspect, the present invention provides a method for detecting weld defects in steel structures, comprising: acquiring a weld image of the steel structure to be detected, and extracting an initial centerline of the weld; extracting a grayscale profile along the normal direction of each point on the initial centerline, determining a set of candidate points for weld edges, and solving for smooth, non-uniform boundary lines on the left and right sides of the weld; based on the boundary lines on the left and right sides, using the geometric midpoint trajectory of the boundary lines as the corrected centerline of the weld, and defining the area enclosed by the boundary lines as the target detection area; extracting an image within the target detection area, and calculating local weld width information and local centerline curvature information for each pixel point within the area based on the corrected centerline and the boundary lines, generating a local geometric topology feature map set with the same size as the original image; and feeding the weld image and the local geometric topology feature map set as multimodal inputs into a feature fusion convolutional neural network to output the category and location information of the weld defect.

[0008] This invention first extracts the initial centerline of the weld and locates smooth, continuous left and right boundary lines of the weld along the normal direction, thereby determining the corrected centerline and the target detection area, effectively eliminating background noise interference. Then, it calculates the local weld width and centerline curvature for each pixel within the area, generating a geometric topological feature map. Finally, the original weld image and this feature map are combined into a multimodal input and fed into a feature fusion convolutional neural network to accurately output the defect category and location. This invention, by explicitly modeling prior information about the weld geometry, successfully guides the network to deeply fuse apparent visual texture and intrinsic structural features, significantly improving the accuracy of defect detection.

[0009] Preferably, the extraction of the initial centerline of the weld includes: constructing the initial centerline of the weld by calculating the eigenvalues ​​of the Hessian matrix of pixels in the weld image and applying a linear structure response function; wherein, for each pixel in the weld image, two eigenvalues ​​of the Hessian matrix of the pixel are calculated; a linear structure response function is set such that, for a weld to exhibit a bright linear structure, the response value is the absolute value of the eigenvalue when the largest absolute value of the two eigenvalues ​​is negative, otherwise the response value is zero; a response threshold is set, and pixels with response values ​​greater than the threshold are marked as high response points; morphological thinning is performed on the binary image composed of all high response points to obtain a connected skeleton with a single pixel width, which serves as the initial centerline of the weld.

[0010] Utilizing the geometric properties of welds as bright linear structures: when the eigenvalue with the largest absolute value is negative, it indicates that the normal direction of that point has a large negative second derivative, conforming to a bright ridge structure, and the response value is taken as the absolute value of this eigenvalue; otherwise, the response value is zero, thus effectively enhancing the linear weld region and suppressing non-linear background. High-response points are then filtered by setting a threshold, and the binary image is morphologically thinned to obtain a single-pixel-width connected skeleton as the initial centerline of the weld. This allows for the stable extraction of the weld's center direction from complex backgrounds, providing an accurate foundation for subsequent precise edge localization and geometric feature calculation.

[0011] Preferably, determining the set of candidate weld edge points includes: locating local extreme points of the response by performing Gaussian first derivative filtering on the grayscale profile; specifically, it includes the following steps: for each point on the initial centerline, extending a preset distance to both sides along the point normal direction to extract a grayscale profile with a preset length; performing first derivative filtering on the grayscale profile using a Gaussian kernel with a preset standard deviation; extracting local response maxima and minima on the filtered response curve, and determining the maxima and minima relative to the initial centerline as candidate points for the left and right weld edges, respectively.

[0012] This invention extracts grayscale profiles along the normal direction of the initial centerline, uses Gaussian first derivative filtering to detect locations of drastic grayscale changes, and identifies local maxima and minima on the response curve as candidate points for the left and right edges, respectively. This can accurately capture the transition boundary of the weld from dark to light and then from light to dark, providing a reliable set of candidate points for the subsequent construction of a smooth and continuous weld boundary.

[0013] Preferably, the step of obtaining smooth non-uniform boundary lines on both sides of the weld includes: establishing a Markov random field energy function containing univariate and bivariate potential functions, and obtaining smooth non-uniform boundary lines on both sides of the weld by minimizing the energy function; wherein, the Gaussian first derivative filter response value at the candidate edge point and the square of the difference between the vertical distances from two candidate edge points in adjacent normal directions to the initial center line are respectively dimensionless; the univariate potential function value is the negative of the absolute value of the dimensionless Gaussian first derivative filter response value; the bivariate potential function value is the square of the difference between the vertical distances after dimensionless processing multiplied by a weighting coefficient; the total energy function is defined as the sum of all univariate and bivariate potential functions, and a graph model is constructed for the edge candidate point sets on the left and right sides of the weld, respectively, and the minimum value of the total energy function is solved by the graph cutting algorithm.

[0014] This invention models weld boundary search as an energy minimization problem: a univariate potential function takes the negative of the absolute value of the filtered response of candidate points, making strong response points have lower energy; a bivariate potential function penalizes drastic changes in the distance from adjacent candidate points to the initial centerline, ensuring a smooth boundary. The minimum total energy on both sides is solved using a graph-cutting algorithm, thus obtaining a smooth, continuous boundary line that closely matches the actual weld.

[0015] Preferably, based on the corrected center line and boundary lines, the local weld width information is calculated for each pixel within the region, including: for each pixel within the target detection region, determining the center line point closest to the pixel in Euclidean distance on the corrected center line; determining the intersection point of the normal direction with the left and right boundary lines along the normal direction of the nearest center line point; calculating the Euclidean distance between the two intersection points, and defining the Euclidean distance as the local weld width at the nearest center line point; creating a single-channel width feature map with the same size as the original image, and assigning the pixel value of the pixel within the target detection region to the local weld width corresponding to the nearest center line point.

[0016] Preferably, based on the corrected centerline and boundary line, the local centerline curvature information is calculated for each pixel in the region, including: for each point on the corrected centerline, taking a preset number of consecutive points in the neighborhood of the point to form a local centerline segment; using the least squares method to fit the points on the local centerline segment into a circle and calculating the radius of the circle; defining the local curvature of the point on the corrected centerline as the reciprocal of the circle radius; creating a single-channel curvature feature map with the same size as the original image; for any pixel in the target detection region, finding the nearest corrected centerline point of the pixel, and assigning the pixel value at the corresponding position on the curvature feature map to the local curvature corresponding to the nearest corrected centerline point.

[0017] Preferably, the step of feeding the weld seam image and the local geometric topology feature map set as multimodal inputs into the feature fusion convolutional neural network includes: using the weld seam image as a base image channel; using the generated local weld seam width feature map as an independent geometric feature channel; using the generated local centerline curvature feature map as another independent geometric feature channel; concatenating the base image channel and the geometric feature channel along the channel dimension to obtain a multi-channel input tensor, and inputting the tensor into the feature fusion convolutional neural network.

[0018] This invention achieves spatial alignment and fusion of multi-source information through channel cascading, enabling the network to simultaneously learn three complementary features: grayscale texture, cross-sectional width, and centerline curvature of the weld. This guides the network to integrate visual and geometric information, more accurately distinguishing the difference between defects and normal fluctuations, thereby improving the accuracy of weld defect detection.

[0019] Preferably, before stitching the basic image channel and the geometric feature channel along the channel dimension, a normalization preprocessing step for the geometric feature channel is further included. Specifically, the local weld width feature map is subjected to min-max normalization processing to map the feature values ​​to the range of 0 to 1; the local centerline curvature feature map is first subjected to logarithmic transformation to compress the dynamic range of the values, and then subjected to min-max normalization processing to map the feature values ​​to the range of 0 to 1.

[0020] Before feature fusion, this invention performs min-max normalization on the local weld width feature map, mapping its values ​​to the 0-1 range to ensure the width information is within a uniform scale. For the local centerline curvature feature map, a logarithmic transformation is first performed to compress its inherently large dynamic range, followed by min-max normalization, thus preventing extreme curvature values ​​from dominating the feature space. The effect is to eliminate differences in dimensions and numerical amplitudes between different geometric features, enabling width and curvature information to be effectively aligned with the original image channels at the same scale. This facilitates the neural network's equal learning and fusion of multimodal features, thereby improving the accuracy and stability of weld defect detection.

[0021] Preferably, the feature fusion convolutional neural network adopts the U-Net architecture. The input of the feature fusion convolutional neural network is a concatenated multi-channel input tensor. The network structure includes an encoder path, a decoder path, and skip connections connecting the two. The encoder path extracts deep semantic features and reduces the feature map size step by step through consecutive convolutional layers and max pooling layers. The decoder path upsamples step by step through consecutive transposed convolutional layers to restore the spatial resolution of the feature map. The skip connections concatenate the feature maps of the corresponding layers in the encoder path with the corresponding upsampled feature maps in the decoder path, fusing shallow detail features and deep semantic features. The final output layer of the network is a 1×1 convolutional layer, which, in conjunction with the Softmax activation function, outputs a pixel-level defect classification prediction map with the same size as the original weld image. The defect classification prediction map corresponds to the category and pixel-level location information of the weld defect.

[0022] This invention adopts the U-Net architecture. The encoder extracts deep semantic features step by step through convolution and pooling, and the decoder restores spatial resolution step by step through transposed convolution. Skip connections fuse the shallow details of the encoder and the deep semantics of the decoder. Finally, after 1×1 convolution and Softmax, a pixel-level defect classification map with the same size as the original image is output, which can accurately locate the category and pixel-level position of weld defects.

[0023] Secondly, the present invention provides a steel structure weld defect detection system, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned steel structure weld defect detection method is implemented.

[0024] By adopting the above technical solution, a computer program for detecting defects in steel structure welds is generated and stored in a memory so that it can be loaded and executed by a processor. A terminal device can then be made based on the memory and processor for convenient use.

[0025] The technical solution of the present invention has the following beneficial technical effects: By combining linear structural responses with Markov random field energy function minimization, smooth and non-uniform weld boundaries can be accurately and completely extracted, thereby defining the target detection area and avoiding interference from background areas. Simultaneously, by constructing local weld width and centerline curvature geometric topological features for pixels within the detection area, the representational power and discriminative power of the features are enhanced. Multimodal fusion input of the original image and the geometric topological feature set enables the feature fusion convolutional neural network to integrate visual and structural information for judgment, thus effectively improving the accuracy of steel structure weld defect detection. Attached Figure Description

[0026] Figure 1 This is a flowchart of a method for detecting defects in steel structure welds; Figure 2 This is a schematic diagram of a steel structure weld defect detection system; Figure 3 This is a schematic diagram of the multimodal input and processing of a feature fusion convolutional neural network. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0028] This invention discloses a method for detecting defects in steel structure welds, referring to... Figure 1 This includes the following steps: S1. Obtain the weld image and extract the initial center line of the weld.

[0029] Specifically, the weld images of the steel structure to be inspected are acquired, and the initial centerline of the weld is extracted. First, using OpenCV 4.0 or later, the image processing library is used to load the weld image of the steel structure to be inspected in grayscale mode. To suppress image noise, Gaussian smoothing is applied to the grayscale image, with the Gaussian kernel size set to 5×5 and the standard deviation... Set it to 1.5.

[0030] For each pixel in the smoothed image The Hessian matrix is ​​calculated by convolving it with a Gaussian second derivative kernel. The three components , , .in, Let be the second partial derivative of the image in the x-direction. Let be the second partial derivative of the image in the y-direction. The mixed second-order partial derivatives of the image in the x and y directions are represented by three components, which are used to characterize the second-order gray-level variation characteristics of the neighborhood of a pixel, thereby distinguishing linear structures from other morphological regions.

[0031] Using the `eigvals` function from the NumPy numerical computation library (version 1.19 and above), calculate the two eigenvalues ​​of the Hessian matrix corresponding to each pixel. and Among them, two eigenvalues ​​satisfy Eigenvalues ​​are used to characterize the directionality of grayscale changes in the neighborhood of a pixel. The weld appears as a bright line structure in the image, and its normal direction has a large second derivative. Eigenvalues ​​with larger absolute values ​​are negative, while the second derivative in the tangent direction is close to 0. Eigenvalues ​​with smaller absolute values ​​are close to 0.

[0032] A linear structure response function is defined to enhance the bright linear structure corresponding to the weld and suppress other non-linear regions. This response function is as follows: for a weld exhibiting a bright linear structure, the response value is the absolute value of the eigenvalue when the largest of the two eigenvalues ​​is negative; otherwise, the response value is 0. When <0, Otherwise, R(x,y)=0. Wherein, For pixels The corresponding linear structure response value is used to characterize the degree of fit between the pixel and the bright linear structure of the weld. The higher the response value, the higher the probability that the pixel belongs to the weld linear structure.

[0033] Optionally, to improve adaptability to welds of different widths, multiple scales can be selected according to the weld width increasing from small to large. ,For example The values ​​were 0.5, 1, 1.5, and 2, respectively, to calculate the linear structure response at each scale. And at the same pixel location, the maximum response value across all scales is taken to construct a multi-scale fused response map. .

[0034] Set response threshold For example, for an 8-bit grayscale image, It can be set to 15, or taken as 30% of the maximum response value; all elements in the image that meet the requirements... The pixel is set to 1, and the other pixels are set to 0 to generate a binary response image.

[0035] Morphological thinning algorithms, such as the Zhang-Suen thinning algorithm, are applied to the binary response image to iteratively peel off edge pixels until all high-response regions are thinned into a connected skeleton of single-pixel width, which is the initial centerline of the weld.

[0036] In an optional embodiment, the Frangi blood vessel enhancement filtering algorithm can be used to construct the linear structure response function based on the obtained Hessian matrix eigenvalues. and Calculate the similarity measure of blood vessels ,Will As a pixel-level response value; this process is repeated at multiple scales, i.e., using different... The values ​​are smoothed using a Gaussian kernel, and the maximum value of the response across all scales is taken as the final response map. The generated multi-scale response map is binarized using the threshold function to obtain the initial weld region. Then, the skeletonize function in the scikit-image image morphology library is used to skeletonize the binarized image and extract the initial center line of the weld with a single pixel width.

[0037] S2. Take a grayscale profile along the normal and solve for the smooth boundary line of the weld.

[0038] Specifically, grayscale profiles are extracted along the normal direction of each point on the initial centerline to determine the candidate point set of the weld edge, and the smooth non-uniform boundary lines on the left and right sides of the weld are obtained by solving.

[0039] Iterate through each pixel on the initial center line obtained from the above steps and perform the following operations: For each point on the initial center line By connecting the two neighboring points before and after this point and The local tangent vector at that point is calculated, and the tangent vector is rotated orthogonally by 90 degrees to obtain the normal vector at that point; the normal vector is then calculated from the point along this normal direction. Extend the line by a distance of L / 2 to the left and right to obtain a line segment with a total length of L. The preset length L must be greater than the expected maximum weld width. For example, if the expected maximum weld width is 40 pixels, L can be set to 60 pixels.

[0040] The pixel grayscale values ​​on the line segment of length L are extracted by bilinear interpolation to form a one-dimensional grayscale profile. This grayscale profile can characterize the grayscale variation pattern in the normal direction of the weld, and the weld edge corresponds to the position where the grayscale changes drastically.

[0041] Using Gaussian first derivative kernel Convolution filtering is performed on this one-dimensional grayscale profile, where the standard deviation is... Used to control the scale of edge detection The preferred range is 1 to 2.5, in this embodiment Set to 2; this filtering operation can amplify the gray-scale abrupt change regions in the gray-scale profile and enhance the response signal corresponding to the edge.

[0042] In an optional embodiment, the Gaussian first-order derivative filter can be constructed as follows: first, a one-dimensional Gaussian kernel is generated using the Gaussian kernel acquisition function, and then the Gaussian kernel is convolved with the first-order difference operator [-1,0,1]; the filter is then used to perform a one-dimensional convolution operation on each extracted grayscale profile data using the convolve function.

[0043] After filtering, a one-dimensional response curve is obtained. This curve is then iterated through to extract the positions of all local maxima and local minima. Since the weld seam is a bright area, the left edge represents a transition from dark to bright, corresponding to a local maxima on the response curve, while the right edge represents a transition from bright to dark, corresponding to a local minima. Therefore, the locations of the local maxima and local minima are... All local maxima points on the left normal are considered as candidate points for the left edge, located in All local minima on the right normal are used as candidate points for the right edge. All candidate points together constitute the set of candidate points for the weld edge. At the same time, the filter response intensity value corresponding to each candidate point is recorded.

[0044] The search problem for the weld boundary is transformed into a graph search problem. Independent graph models are constructed for the extraction of edge points on the left and right sides of the weld. Taking the left edge as an example, the following operations are performed: A Markov random field energy function is constructed to measure the total cost of selecting the current edge point. A smaller energy value indicates a weld boundary that closely resembles the true edge and is smoother and more reasonable. Solving for the optimal boundary is equivalent to minimizing this energy function. The expression for the Markov random field energy function is: ; In the formula, This is the total energy function value, used to comprehensively characterize the confidence level of edge point selection and boundary smoothness; For the first sampled along the initial centerline The first on the legal line The candidate edge points are determined by the set of candidate weld edge points; This is a univariate potential function used to characterize the confidence that a candidate edge point is a true weld edge. Its value is the negative of the absolute value of the dimensionless processed Gaussian first derivative filter response at the candidate edge point. ,in for After Gaussian first-derivative filtering and elimination of physical dimensions, this function makes candidate points with larger absolute values ​​of the filtered response and higher edge confidence have lower corresponding energy values, and are preferentially selected in the solution process. This is a binary potential function used to constrain the positional continuity of adjacent edge points, ensuring the smoothness of the boundary line. Its expression is: , This represents the dimensionless value of the vertical distance from the candidate point to the initial centerline. These are dimensionless weighting coefficients used to balance the influence of univariate and bivariate potential functions. The preferred range is 0.5 to 5, in this embodiment Set to 2; this function penalizes drastic changes in the vertical distance between adjacent edge points and the initial centerline, encouraging the generation of smooth boundary lines.

[0045] Among them, the univariate potential function is determined by the position of the candidate point at the weld edge and the filter response value, while the bivariate potential function is used to represent the positional continuity between adjacent edge points.

[0046] Graph models are constructed for the candidate edge points on the left and right sides of the weld, and the minimum value of the total energy function is solved by the graph cutting algorithm. Specifically, by constructing a graph containing source node s and sink node t, the candidate points are used as intermediate nodes, and the weights of the edges are set according to the above energy function. Solving the minimum cut st can obtain the optimal edge point sequence that minimizes energy, that is, the left and right boundary lines that are smooth and fit the actual weld edge.

[0047] Optionally, the minimum value of the energy function can be found in the following way: To solve for the left boundary line, construct a directed acyclic graph, where the nodes are composed of all candidate points on the left edge; define the Markov random field energy function as follows: ,in It is the set of boundary point lines to be solved, and the univariate potential function. Set to the negative value of the candidate point filter response intensity, i.e. binary potential function Defined as candidate points on two adjacent grayscale profiles and The square of the Euclidean distance between them is calculated; the minimum value of the energy function is found using Dijkstra's shortest path algorithm, and the path with the lowest energy obtained is the optimal left boundary line; the right boundary line is calculated independently using the same process to obtain the right boundary line of the weld.

[0048] S3. Determine the target detection region and generate a geometric topological feature map.

[0049] Specifically, based on the left and right boundary lines, the geometric midpoint trajectory of the boundary lines is used as the correction center line of the weld, and the area enclosed by the boundary lines is defined as the target detection area. The image within the target detection area is extracted, and based on the correction center line and the boundary lines, the local weld width information and local center line curvature information are calculated for each pixel in the area to generate a local geometric topology feature map with the same size as the original image.

[0050] For each row y in the image, find the x-coordinate of the upper left boundary line of that row. and the x-coordinate of the right boundary line Calculate the x-coordinate of the corrected center line on this row. ; connect all rows ( By finding the point (y), a continuous corrected centerline is obtained. This corrected centerline can accurately represent the actual direction of the weld and eliminate the deviation of the initial centerline.

[0051] Simultaneously, create a zero-mask image with the same dimensions as the original image. For each row y, apply zeros to the image from the top of that row. arrive The pixel values ​​between the two parts are set to 1, and the rest are kept to 0; the area corresponding to this binary mask image is the target detection area. This step can limit the effective range of defect detection and eliminate the interference of the background area.

[0052] The local geometric topology feature map set generated in this step includes a local weld width feature map and a local centerline curvature feature map. Both feature maps have the same size as the original weld image. The specific generation process is as follows: Generate a local weld width feature map, and pre-calculate the local weld width corresponding to all points on the correction centerline: for each point on the correction centerline... Calculate the normal vector at that point, and find the intersection of the normal vector and the left boundary line. Right boundary line intersection and Calculate the Euclidean distance between two intersection points. This distance is the centerline point. Local weld width .

[0053] Create a single-channel floating-point feature map with the same dimensions H×W as the original image and all initial values ​​set to 0. This feature map is the local weld width feature map. It iterates through every pixel within the target detection area. Using a kd-tree or brute-force search method, find the centerline point on the corrected centerline that has the closest Euclidean distance to the pixel. ;Will Corresponding pre-calculated local weld width The value is assigned to the pixel at the corresponding position in the feature map Mw, i.e. Pixels outside the target detection area retain a pixel value of 0.

[0054] Through the above steps, a local weld width feature map aligned with the original image space is generated. The value of each pixel in the map represents the width information of the weld cross-section where that pixel is located.

[0055] In an optional embodiment, a local weld width feature map can be generated by creating an all-zero floating-point matrix of the same size as the original image, for each pixel within the target detection area. Find the row containing the pixel. The corresponding x-coordinates of the left and right boundary points and Calculate the local weld width of this row. , this width value Assign to the width map The pixel value of the location.

[0056] Generate a local centerline curvature feature map, and pre-calculate the local curvature corresponding to all points on the corrected centerline: for each point on the corrected centerline... Define a neighborhood window containing the point itself, select N consecutive points before and after the point to form a local centerline segment containing 2N+1 points; where the value of N determines the scale of curvature calculation, used to characterize the macroscopic bending of the weld, and the preferred range of N is 10 to 25, and in this embodiment, N is taken as 15.

[0057] The coordinates of 2N+1 points on the local centerline segment By applying a least-squares-based circle fitting algorithm, the radius of the best-fit circle can be obtained. ; this point Local curvature Defined as the reciprocal of the radius of the fitted circle, i.e. To avoid division by zero, when the radius of the fitted circle is extremely large and the corresponding local center line segment is close to a straight line, the curvature of that point is set to 0.

[0058] Create a single-channel floating-point feature map with the same dimensions H×W as the original image and all initial values ​​set to 0. This feature map is the local centerline curvature feature map.

[0059] Traverse every pixel within the target detection region. A kd-tree (with 10 leaf nodes) is constructed based on 2D pixel coordinates. The nearest neighbor search algorithm is used to find the centerline point with the closest Euclidean distance to the pixel on the corrected centerline. Alternatively, a brute-force search method can be used, traversing the corrected centerline pixels one by one with a step size of 1 pixel to find the centerline point with the closest Euclidean distance to that pixel. .Will Corresponding pre-calculated local curvature Assign the value to the corresponding pixel in the feature map, i.e. Pixels outside the target detection area retain a pixel value of 0.

[0060] Through the above steps, a local centerline curvature feature map aligned with the original image space is generated. The value of each pixel in the map represents the degree of curvature of the weld at that pixel's location.

[0061] In an optional embodiment, the local centerline curvature feature map can be generated by creating an all-zero floating-point matrix and applying it to each point on the corrected centerline. Use adjacent points before and after and ,in The integer is 5 in this embodiment. Approximate values ​​of the first and second derivatives are calculated, and the relationship is calculated using discrete curvature. Calculate the point curvature value For each pixel within the target detection area Find the point on the correction center line that is closest to it. and the curvature value at that point Assigned to the curvature diagram The pixel position. In the above formula, To correct the first-order difference approximation of the x-coordinate of the centerline point, To correct the first-order difference approximation of the ordinate of the centerline point, The second-order difference approximation of the x-axis is given by [value]. The value is the second-order difference approximation of the ordinate.

[0062] Optionally, the generated local weld width feature map and local centerline curvature feature map are normalized by using a minimum-maximum normalization method to scale the numerical range to between 0 and 1, forming the final local geometric topology feature map set.

[0063] S4, a multimodal input network, outputs defect category and location.

[0064] Specifically, the weld seam image and the local geometric topology feature set are used as multimodal inputs and fed into a feature fusion convolutional neural network to output the category and location information of weld seam defects.

[0065] like Figure 3As shown, the multimodal input and processing process of the feature fusion convolutional neural network is as follows: the acquired basic image features, local width features, and local curvature features are concatenated into multi-channel features, and then input into the network to perform coded feature extraction and decoded feature reconstruction in sequence, and finally output the corresponding prediction results.

[0066] First, the input data is preprocessed to construct a multi-channel input tensor: the original H×W single-channel grayscale weld image I is normalized, and the pixel values ​​are scaled to the [0,1] interval through linear transformation, serving as the first channel (basic image channel) of the input tensor; the generated H×W local weld width feature map is then processed. Perform min-max normalization; the normalization formula is: ,in For taking values A small constant is used to prevent the denominator from being zero; the normalized value is... As the second channel of the input tensor; for the generated H×W local centerline curvature feature map Preprocessing is performed first, using logarithmic transformation. Compress the numerical range and then perform minimum-maximum normalization to obtain the normalized value. , which serves as the third channel of the input tensor.

[0067] The three single-channel images of H×W×1 are stitched together along the channel dimension to form a multi-channel input tensor of H×W×3. This tensor spatially aligns pixel-level grayscale information, weld width information, and centerline curvature information, realizing the fusion of visual information and geometric topological structure information.

[0068] The multi-channel input tensor is fed into the feature fusion convolutional neural network. The feature fusion convolutional neural network adopts an encoder-decoder structure. In this embodiment, the U-Net architecture is used. The input of the network is a four-dimensional tensor of dimension B×C×H×W, where B is the batch size, C is the number of channels (C is 3 in this embodiment), which correspond to the normalized grayscale image, the local weld width feature map, and the local centerline curvature feature map, respectively. H and W are the height and width of the input image, respectively.

[0069] The structure of a feature fusion convolutional neural network includes an encoder path, a decoder path, and skip connections connecting the two: Encoder path: Through a series of convolutional layers and max pooling layers, higher-level semantic features are extracted step by step, while reducing the spatial size of the feature map; Decoder path: The spatial resolution of the feature map is gradually restored through a series of upsampling layers or transposed convolutional layers; Skip connection: The feature maps of the corresponding layers in the encoder path are concatenated with the feature maps of the decoder path, fusing the high-resolution detail information in the shallow layer and the semantic information in the deep layer; The final layer of the network is a 1×1 convolutional layer, followed by a Sigmoid or Softmax activation function. The network output is a prediction map of dimension H×W×N, where N is the number of defect categories. For binary defect detection tasks, N is 1, and the value of each pixel in the prediction map represents the probability that the point belongs to a defect. For multi-class defect detection tasks, N is the total number of defect categories (such as background, cracks, pores, etc.). Each pixel is classified using the Softmax activation function, generating a pixel-level segmentation map of defect categories and locations with the same size as the original image. The multimodal input and processing of the feature fusion convolutional neural network are as follows... Figure 3 As shown.

[0070] Optionally, the encoder part of the feature fusion convolutional neural network adopts a dual-branch structure: the input of the first branch is the original grayscale weld image of a single channel, and a pre-trained ResNet-34 is used as the backbone network to extract depth image features; the input of the second branch is a dual-channel feature map composed of a normalized local weld width map and a local centerline curvature map, and a shallow convolutional network is used to extract geometric features; at the end of the encoder path, the feature maps output by the two branches are concatenated in the channel dimension, and channel dimensionality reduction and information fusion are performed through a 1×1 convolutional layer; the decoder part adopts the standard U-Net upsampling structure, which performs upsampling step by step through transposed convolution, and after each level of upsampling, the feature maps of the corresponding level of the encoder of the first image branch are concatenated through skip connections to retain high-resolution detail information; the output layer of the network is a 1×1 convolutional layer, the number of convolutional kernels is equal to the total number of defect categories, and each pixel is classified through the Softmax activation function.

[0071] To verify the technical effectiveness of the method provided in this embodiment, the following comparative experiment was conducted: Experimental conditions: The dataset consisted of a publicly available weld defect image library containing 2000 images of 512×512 pixels, divided into training, validation, and test sets in an 8:1:1 ratio; the hardware platform was an NVIDIA RTX3090, and the software environment was the PyTorch 1.10 deep learning framework; all models adopted the U-Net architecture, using the Adam optimizer, with an initial learning rate of 0.0001, a batch size of 8, and a total training duration of 100 epochs; the evaluation metrics used were mean intersection-over-union ratio (mIoU), pixel accuracy (PA), and Dice coefficient.

[0072] Scheme 1 is the baseline model, using only a single-channel grayscale weld image as input to the U-Net network. Scheme 2 is the ablation experiment, with input being a two-channel tensor concatenated from the grayscale image and a local weld width feature map. Scheme 3 is another set of ablation experiments, with input being a two-channel tensor concatenated from the grayscale image and a local centerline curvature feature map. Scheme 4 is the complete model proposed in this paper, with input being a three-channel tensor concatenated from the grayscale image, width feature map, and curvature feature map.

[0073] The specific experimental data are recorded as follows: Scheme 1: mIoU 85.3%, PA 98.2%, Dice 92.1%. Scheme 2: mIoU 87.5%, PA 98.7%, Dice 93.3%. Scheme 3: mIoU 86.8%, PA 98.5%, Dice 92.9%. Scheme 4: mIoU reached 89.1%, PA 99.0%, Dice 94.2%.

[0074] Experimental results show that, compared with the baseline model that only uses the original grayscale image, inputting any geometric topology feature map can improve detection performance: inputting the width feature map alone improves the mIoU by 2.2 percentage points, proving that local weld width information plays a key role in identifying defects related to weld size, such as undercut and incomplete penetration; inputting the curvature feature map alone improves the mIoU by 1.5 percentage points, indicating that the representation of the curvature of the centerline helps the network understand the overall trend of the weld, thereby more accurately locating the defect area.

[0075] When the width and curvature feature maps are used together with the original image as multimodal input, the model performance reaches its optimum, with mIoU improving by 3.8 percentage points compared to the baseline model. This proves that the two geometric features provide complementary structural prior knowledge, which can guide the neural network to focus on the deep geometric properties of the image, rather than relying solely on surface texture information, thereby improving the accuracy of steel structure weld defect detection.

[0076] like Figure 2 As shown in the figure, this invention also discloses a steel structure weld defect detection system. This system is applied to the detection of workpieces and areas to be tested, and mainly includes a cooperating lighting device, an image acquisition device, and a processing terminal. The lighting device provides illumination conditions, working in conjunction with the image acquisition device to acquire images of the steel structure welds to be tested, and then transmits these images to the processing terminal. The processing terminal internally includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the aforementioned steel structure weld defect detection method.

[0077] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0078] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting defects in welds of steel structures, characterized in that, include: Acquire images of the welds in the steel structure to be inspected and extract the initial centerline of the welds; Extract grayscale profiles along the normal direction of each point on the initial centerline, determine the candidate point set of the weld edge, and solve for the smooth non-uniform boundary lines on the left and right sides of the weld. Based on the left and right boundary lines, the geometric midpoint trajectory of the boundary lines is used as the correction center line of the weld, and the area enclosed by the boundary lines is defined as the target detection area. The image within the target detection area is extracted, and based on the correction center line and the boundary lines, the local weld width information and local center line curvature information are calculated for each pixel in the area to generate a local geometric topology feature map with the same size as the original image. The weld seam image and the local geometric topology feature set are used as multimodal inputs and fed into a feature fusion convolutional neural network to output the category and location information of weld seam defects.

2. The method for detecting defects in steel structure welds according to claim 1, characterized in that, The extraction of the initial centerline of the weld seam includes: constructing the initial centerline of the weld seam by calculating the eigenvalues ​​of the Hessian matrix of the pixels in the weld seam image and applying a linear structure response function; wherein, for each pixel in the weld seam image, two eigenvalues ​​of the Hessian matrix of the pixel are calculated. A linear structure response function is defined such that the weld exhibits a bright linear structure. When the largest absolute value of the two eigenvalues ​​is negative, the response value is the absolute value of the eigenvalue; otherwise, the response value is zero. Set a response threshold and mark pixels with response values ​​greater than the threshold as high response points; Morphological thinning is performed on the binary image composed of all high response points to obtain a connected skeleton with a single pixel width, which serves as the initial centerline of the weld.

3. The method for detecting defects in steel structure welds according to claim 1, characterized in that, The determination of the candidate point set for the weld edge includes: locating the local extreme points of the response by performing Gaussian first derivative filtering on the grayscale profile; specifically, it includes the following steps: for each point on the initial center line, extend a preset distance to both sides along the point normal direction to extract a grayscale profile with a preset length. The grayscale profile is filtered by first-order derivative using a Gaussian kernel with a preset standard deviation. On the filtered response curve, local response maxima and minima are extracted. Based on the positions of the maxima and minima relative to the initial centerline, they are determined as candidate points for the left and right weld edges, respectively.

4. The method for detecting defects in steel structure welds according to claim 3, characterized in that, The process of obtaining smooth, non-uniform boundary lines on both sides of the weld includes: establishing a Markov random field energy function containing univariate and bivariate potential functions, and obtaining smooth, non-uniform boundary lines on both sides of the weld by minimizing the energy function; wherein, the Gaussian first derivative filter response value at the candidate edge point and the square of the difference between the vertical distances from two candidate edge points in adjacent normal directions to the initial center line are respectively dimensionless; the univariate potential function value is the negative of the absolute value of the dimensionless Gaussian first derivative filter response value; the bivariate potential function value is the square of the difference between the vertical distances after dimensionless processing multiplied by a weighting coefficient; the total energy function is defined as the sum of all univariate and bivariate potential functions, and a graph model is constructed for the edge candidate point sets on the left and right sides of the weld, respectively, and the minimum value of the total energy function is solved by a graph cutting algorithm.

5. The method for detecting defects in steel structure welds according to claim 1, characterized in that, Based on the corrected center line and boundary line, calculate the local weld width information for each pixel in the region, including: for each pixel in the target detection region, determine the center line point with the closest Euclidean distance to the pixel on the corrected center line; Determine the intersection points of the normal direction and the left and right boundary lines along the normal direction of the nearest center line point; Calculate the Euclidean distance between the two intersection points, and define the Euclidean distance as the local weld width at the nearest centerline point; Create a single-channel width feature map with the same size as the original image, and assign the pixel value of each pixel in the target detection area to the local weld width corresponding to the nearest centerline point.

6. The method for detecting defects in steel structure welds according to claim 1, characterized in that, Based on the corrected centerline and boundary line, calculate the local centerline curvature information for each pixel in the region, including: for each point on the corrected centerline, take a preset number of consecutive points in the neighborhood of the point to form a local centerline segment. The points on the local centerline segment are fitted into a circle using the least squares method, and the radius of the circle is calculated; the local curvature of the points on the modified centerline is defined as the reciprocal of the circle radius; Create a single-channel curvature feature map with the same size as the original image. For any pixel within the target detection area, find the nearest corrected centerline point of the pixel and assign the pixel value at the corresponding position on the curvature feature map to the local curvature corresponding to the nearest corrected centerline point.

7. The method for detecting defects in steel structure welds according to claim 1, characterized in that, The step of feeding the weld seam image and the local geometric topology feature map set as multimodal inputs into the feature fusion convolutional neural network includes: using the weld seam image as the base image channel; using the generated local weld seam width feature map as an independent geometric feature channel; and using the generated local centerline curvature feature map as another independent geometric feature channel. The basic image channels and geometric feature channels are concatenated along the channel dimension to obtain a multi-channel input tensor, which is then input into the feature fusion convolutional neural network.

8. The method for detecting defects in steel structure welds according to claim 7, characterized in that, Before concatenating the basic image channels and geometric feature channels along the channel dimension, a normalization preprocessing step for the geometric feature channels is included, specifically: The local weld width feature map is subjected to min-max normalization to map the feature values ​​to the interval between 0 and 1; The local centerline curvature feature map is first subjected to a logarithmic transformation to compress the dynamic range of the values, and then subjected to min-max normalization to map the feature values ​​to the interval between 0 and 1.

9. The method for detecting defects in steel structure welds according to claim 1, characterized in that, The feature fusion convolutional neural network adopts the U-Net architecture. The input of the feature fusion convolutional neural network is the concatenated multi-channel input tensor. The network structure includes encoder path, decoder path and skip connection connecting the two. The encoder path extracts deep semantic features and reduces the feature map size step by step through consecutive convolutional layers and max pooling layers; The decoder path upsamples the feature map step by step through successive transposed convolutional layers to restore the spatial resolution of the feature map; The skip connection concatenates the feature map of the corresponding level in the encoder path with the upsampled feature map of the corresponding level in the decoder path, thus fusing shallow detail features with deep semantic features. The final output layer of the network is a 1×1 convolutional layer, which, in conjunction with the Softmax activation function, outputs a pixel-level defect classification prediction map with the same size as the original weld image. The defect classification prediction map corresponds to the category and pixel-level location information of the weld defect.

10. A steel structure weld defect detection system, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the steel structure weld defect detection method according to any one of claims 1-9.

Images