Sheet metal part pre-coating surface defect detection method and system

Abstract

The present application provides a kind of sheet metal parts coating front surface defect detection method and system, obtains the original surface reflection image set of sheet metal parts collected by optical detection equipment, and carries out multiscale surface feature decoupling processing, obtains substrate reflection component image set and geometric structure component image set.Based on substrate reflection component image set and geometric structure component image set, construct defect sensitive feature fusion map, each pixel point feature vector is formed by the substrate reflection component value and the geometric structure component value of corresponding position.The pre-trained defect region positioning network is called to process the fusion map, and a pixel-level defect mask image is generated, which includes the confidence score value of each pixel point belonging to the defect region.According to the connected domain set formed by the pixel points satisfying the preset condition of confidence score value, the defect contour boundary coordinate sequence and defect type discrimination result of each connected domain are generated.The present application can accurately detect sheet metal surface defects and improve detection quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and system for detecting surface defects on sheet metal parts before painting. Background Technology

[0002] In the pre-painting production process of sheet metal parts, surface defect detection is a crucial step in ensuring product quality. Sheet metal surface defects are diverse, including scratches, dents, bumps, and stains. These defects not only affect the product's appearance but can also negatively impact the painting process, leading to reduced coating adhesion, uneven coating thickness, and consequently, the product's lifespan and performance. Traditional sheet metal surface defect detection methods primarily rely on manual visual inspection, which has several limitations. Manual inspection is inefficient, unsuitable for large-scale production, and easily affected by subjective factors such as fatigue and experience differences among inspectors, making it difficult to guarantee the accuracy and consistency of results. With the development of industrial automation and intelligence, machine vision-based defect detection methods are gradually emerging. However, existing machine vision-based methods often fail to accurately distinguish between substrate material reflection and surface geometric reflection when dealing with the complex optical reflection characteristics of sheet metal surfaces, resulting in insufficient sensitivity to defects and a high likelihood of missed or false detections. Furthermore, there is a lack of effective feature extraction and classification methods for defects of different types and sizes, making it difficult to achieve high-precision defect detection and identification. Summary of the Invention

[0003] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for detecting surface defects of sheet metal parts before painting, the method comprising: Acquire a set of raw surface reflection images of sheet metal parts collected by optical inspection equipment; Multi-scale surface feature decoupling processing is performed on the original surface reflection image set to decompose the original surface reflection image set into a set of substrate reflection component images characterizing the substrate material of the sheet metal part and a set of geometric structure component images characterizing the surface contour of the sheet metal part. A defect-sensitive feature fusion map is constructed based on the set of base reflection component images and the set of geometric structure component images. The feature vector of each pixel in the defect-sensitive feature fusion map is composed of the base reflection component value and the geometric structure component value at the corresponding position. The pre-trained defect region localization network is invoked to process the defect-sensitive feature fusion map and generate a pixel-level defect mask image for identifying defect regions on the surface of sheet metal parts. The pixel-level defect mask image contains a confidence score value for each pixel to belong to the defect region. Based on the set of connected components formed by pixels in the pixel-level defect mask image whose confidence scores meet preset conditions, a defect contour boundary coordinate sequence and defect type discrimination result are generated for each connected component.

[0004] In another aspect, embodiments of the present invention also provide a surface defect detection system for sheet metal parts before painting, including a processor and a machine-readable storage medium. The machine-readable storage medium is connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code. The processor is used to execute the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0005] Based on the above, this embodiment of the invention decomposes the original surface reflection image into a set of substrate reflection component images and a set of geometric structure component images through multi-scale surface feature decoupling processing. This effectively separates the influence of the sheet metal substrate material and surface contour on the reflection image. When constructing the defect-sensitive feature fusion map, it comprehensively considers the differences in substrate reflection and changes in geometric curvature, enabling the feature vector of each pixel in the fusion map to more comprehensively reflect surface defect information and improve the sensitivity to defects. A pre-trained defect region localization network is called to process the defect-sensitive feature fusion map. Utilizing the powerful feature extraction and classification capabilities of deep learning models, pixel-level defect mask images can be accurately generated to identify defect regions on the sheet metal surface. Finally, based on the pixel-level defect mask images, a defect contour boundary coordinate sequence and defect type discrimination result are generated for each connected component, achieving precise defect localization and classification. This effectively improves the quality and efficiency of surface defect detection before sheet metal painting, reducing the missed detection and false detection rates. Attached Figure Description

[0006] Figure 1 This is a schematic diagram of the execution flow of the sheet metal part surface defect detection method before painting provided in the embodiment of the present invention.

[0007] Figure 2 This is a schematic diagram of exemplary hardware and software components of the sheet metal part surface defect detection system before painting provided in an embodiment of the present invention. Detailed Implementation

[0008] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a method for detecting surface defects of sheet metal parts before painting, provided in one embodiment of the present invention. The method for detecting surface defects of sheet metal parts before painting will be described in detail below.

[0009] Step S110: Obtain a set of original surface reflection images collected by an optical inspection device on the surface of the sheet metal part.

[0010] In this embodiment, an industrial-grade high-resolution optical inspection device is used to acquire surface images of automotive body sheet metal parts. This optical inspection device includes multiple line-scan cameras at different angles and a ring LED light source system. The camera resolution and acquisition speed are preset according to the surface detail requirements of the sheet metal parts and the production line speed. During the acquisition process, the sheet metal parts move at a constant speed via a conveyor belt, ensuring that all areas of the surface are completely captured. The acquired raw surface reflection image set contains multiple images, the size of each image determined by the camera resolution and field of view. The image format is 24-bit true color, and the images are stored in a lossless compressed image file format. To ensure data quality, white balance calibration and lens distortion correction are performed on the optical inspection device before acquisition. During the acquisition process, the ambient light intensity is monitored in real time, and the light source power is automatically adjusted for compensation when the light intensity deviates from the preset range. The acquired raw surface reflection image set covers the reflection characteristics of different areas of the sheet metal parts under different lighting angles.

[0011] Step S120: Perform multi-scale surface feature decoupling processing on the original surface reflection image set, decomposing the original surface reflection image set into a set of substrate reflection component images representing the substrate material of the sheet metal part and a set of geometric structure component images representing the surface contour of the sheet metal part.

[0012] In this embodiment, the original surface reflection image set obtained in step S110 undergoes multi-scale surface feature decoupling processing. First, for each original surface reflection image, a multi-scale decomposition algorithm is used to decompose it in different spatial frequency domains, thereby separating the low-frequency components reflecting the substrate material properties and the high-frequency components reflecting the surface geometry. This processing effectively eliminates the interference of uneven substrate reflection and surface geometry on defect detection in the original images. Through this step, the same set of substrate reflection component images and geometric structure component images as the original images are finally obtained. The substrate reflection component images mainly present the uniform reflection characteristics of the sheet metal surface material, while the geometric structure component images highlight the contour and shape changes of the surface.

[0013] Step S121: Perform spatial frequency domain transformation processing on each original surface reflection image in the original surface reflection image set to obtain a multi-scale frequency coefficient distribution map of each original surface reflection image in the first frequency sub-band, the second frequency sub-band, and the third frequency sub-band.

[0014] In this embodiment, each original surface reflection image undergoes spatial frequency domain transformation processing. Specifically, a two-dimensional discrete wavelet transform algorithm is used, selecting orthogonal wavelet basis functions with preset vanishing moments to perform multi-level wavelet decomposition on the original surface reflection image. During the decomposition process, the image is subjected to high-pass and low-pass filtering in the horizontal and vertical directions, respectively, to obtain the coefficient distribution of different frequency sub-bands. The first frequency sub-band corresponds to the low-frequency component, mainly containing the overall brightness and substrate reflection information of the image; the second frequency sub-band corresponds to the mid-frequency component, containing certain texture and detail information; and the third frequency sub-band corresponds to the high-frequency component, mainly reflecting the edge and abrupt change information of the image. After spatial frequency domain transformation processing, each original surface reflection image is decomposed into a multi-scale frequency coefficient distribution map of the first, second, and third frequency sub-bands. The size of the coefficient distribution map of each frequency sub-band is one-quarter of the original image, while maintaining the spatial correspondence with the original image.

[0015] Step S122: Perform nonlinear gain compression processing on the first frequency sub-band coefficients in the multi-scale frequency coefficient distribution map to suppress the intensity fluctuation of the substrate reflection component in the first frequency sub-band, and obtain the compressed first frequency sub-band coefficient distribution map.

[0016] In this embodiment, nonlinear gain compression processing is applied to the coefficients of the first frequency sub-band. Since the first frequency sub-band mainly contains information about the substrate reflection component, and the intensity fluctuations of this component may mask the defect signal, nonlinear compression is needed to suppress these fluctuations. First, statistical analysis is performed on the amplitude of the first frequency sub-band coefficients to determine their dynamic range. Then, a piecewise nonlinear compression function is designed based on this dynamic range. This function uses linear mapping in the low-amplitude region to preserve detail information and logarithmic compression in the high-amplitude region to reduce the contrast in the strong reflection region. This processing effectively balances the substrate reflection intensity in different regions. After processing, a compressed distribution map of the first frequency sub-band coefficients is obtained, with its coefficient amplitude range reasonably adjusted.

[0017] Step S1221: Extract the amplitude statistical histogram of the first frequency sub-band coefficients from the multi-scale frequency coefficient distribution map, and determine the dynamic compression threshold based on the coefficient distribution density of each amplitude interval in the amplitude statistical histogram.

[0018] In this embodiment, all coefficient values ​​in the first frequency sub-band coefficient distribution map are first traversed, and the number of coefficients in each amplitude interval is counted to construct an amplitude statistical histogram. The amplitude intervals are divided at equal intervals, and the number of intervals is determined according to the dynamic range of the coefficients, covering the entire dynamic range of the first frequency sub-band coefficients. Then, the coefficient distribution density of each amplitude interval is calculated, which is the ratio of the number of coefficients in each interval to the total number of coefficients. Next, the distribution density is accumulated starting from the low amplitude intervals. When the accumulated distribution density reaches a preset ratio, the corresponding amplitude is the dynamic compression threshold. The preset ratio is adjusted according to the uniformity of the sheet metal substrate material. For sheet metal parts with uniform material, the preset ratio can be set higher to retain more details; for sheet metal parts with non-uniform material, the preset ratio can be appropriately reduced to better suppress intensity fluctuations. The dynamic compression threshold determined in this way can adapt to sheet metal parts with different substrate reflection characteristics, improving the adaptability and effectiveness of compression processing.

[0019] Step S1222: Construct a piecewise nonlinear compression function based on the dynamic compression threshold. The piecewise nonlinear compression function adopts a linear mapping relationship in the interval where the amplitude is less than the dynamic compression threshold, and adopts a logarithmic compression mapping relationship in the interval where the amplitude is greater than or equal to the dynamic compression threshold.

[0020] In this embodiment, a piecewise nonlinear compression function is constructed based on the dynamic compression threshold determined in step S1221. For coefficient values ​​with amplitudes less than the dynamic compression threshold, a linear mapping relationship is used. The mapping method is that the compressed amplitude equals the original amplitude multiplied by a linear gain factor, where the linear gain factor is determined based on the dynamic compression threshold and the target compression range, such that when the original amplitude equals the dynamic compression threshold, the compressed amplitude is a specific proportion of the target compression range. For coefficient values ​​with amplitudes greater than or equal to the dynamic compression threshold, a logarithmic compression mapping relationship is used. The mapping method is that the compressed amplitude equals the result of the logarithmic function acting on (original amplitude minus dynamic compression threshold plus a constant), multiplied by the logarithmic gain factor, and finally added to the compressed amplitude at the dynamic compression threshold by the linear mapping part. The selection of the logarithmic base and the logarithmic gain factor must ensure that the maximum value of the compressed amplitude does not exceed the upper limit of the target compression range, while ensuring that the compression effect in the high amplitude region meets expectations. Through this piecewise nonlinear compression function, useful defect information can be preserved to the greatest extent while suppressing the fluctuation of the substrate reflection intensity.

[0021] Step S1223: Input the first frequency sub-band coefficient into the piecewise nonlinear compression function, perform amplitude compression calculation on each first frequency sub-band coefficient, and generate compressed first frequency sub-band coefficient values.

[0022] In this embodiment, each coefficient value in the first frequency sub-band coefficient distribution map is input one by one into the piecewise nonlinear compression function constructed in step S1222. For each coefficient value, it is determined whether it is less than the dynamic compression threshold. If the coefficient value is less than the dynamic compression threshold, the compressed coefficient value is calculated according to the linear mapping relationship. If the coefficient value is greater than or equal to the dynamic compression threshold, the compressed coefficient value is calculated according to the logarithmic compression mapping relationship. During the calculation process, it is necessary to pay attention to limiting the range of coefficient values ​​to ensure that the compressed coefficient values ​​are within the preset effective range and to avoid overflow or invalid values. By performing the above compression calculation on all first frequency sub-band coefficients, the compressed first frequency sub-band coefficient values ​​at each position are obtained, and these coefficient values ​​constitute the preliminary compressed first frequency sub-band coefficient distribution map.

[0023] Step S1224: Calculate the compression ratio parameter of each first frequency sub-band coefficient based on the ratio of the compressed first frequency sub-band coefficient value to the original first frequency sub-band coefficient value.

[0024] In this embodiment, for each first frequency sub-band coefficient, the ratio of the compressed coefficient value to the original coefficient value is calculated to obtain the compression ratio parameter. When the original coefficient value is zero, the compression ratio parameter is set to a constant to avoid the error of dividing by zero. The compression ratio parameter reflects the degree of compression for each coefficient value; the smaller the value, the greater the compression. By calculating the compression ratio parameter, the compression effect can be quantitatively evaluated. For example, when the compression ratio parameters of adjacent coefficients differ significantly, it may indicate that local discontinuities have been introduced into the compression process, requiring smoothing processing in subsequent steps.

[0025] Step S1225: Perform spatial consistency smoothing on the compressed first frequency sub-band coefficient values. Utilize the spatial correlation between adjacent first frequency sub-band coefficients to perform local neighborhood mean filtering on the compressed first frequency sub-band coefficient values ​​to eliminate local discontinuities introduced by nonlinear compression, thereby obtaining a spatially smoothed compressed first frequency sub-band coefficient distribution map.

[0026] In this embodiment, local neighborhood mean filtering is used to perform spatially consistent smoothing on the compressed first frequency sub-band coefficient values. First, the size of the filtering neighborhood window is determined, ensuring the window size effectively eliminates local discontinuities without excessively blurring useful details. For each coefficient value to be filtered, the average of all coefficient values ​​within its neighborhood window is calculated as the smoothed coefficient value. When calculating the average, considering the compression ratio parameter obtained in step S1224, a weighted average is applied to the coefficient values ​​within the neighborhood. Coefficients with smaller compression ratio parameters (i.e., greater compression) are assigned lower weights, and coefficients with larger compression ratio parameters are assigned higher weights. The weights are calculated by dividing the compression ratio parameter value of each coefficient within the neighborhood by the sum of the compression ratio parameter values ​​of all coefficients within the neighborhood. This weighted mean filtering smooths local discontinuities while preserving the original coefficient distribution characteristics as much as possible. After spatially consistent smoothing, a spatially smoothed compressed first frequency sub-band coefficient distribution map is obtained, in which the coefficient values ​​are smoother, and local discontinuities are effectively eliminated.

[0027] Step S123: Perform edge-preserving filtering on the third frequency sub-band coefficients in the multi-scale frequency coefficient distribution map to retain the edge abrupt change information of the geometric structure components in the third frequency sub-band, and obtain the filtered third frequency sub-band coefficient distribution map.

[0028] In this embodiment, edge-preserving filtering is applied to the third frequency sub-band coefficients. The third frequency sub-band coefficients primarily reflect the geometric structure and edge information of the sheet metal surface, which is crucial for defect detection. Therefore, an edge-preserving filtering algorithm is needed to remove noise while retaining edge abrupt changes. Specifically, an adaptive bilateral filtering algorithm is used. This algorithm determines the filtering weights by considering spatial distance and pixel value similarity, enabling it to smooth noise in certain areas while maintaining edge clarity. During the filtering process, the filtering parameters are dynamically adjusted based on the gradient information of the third frequency sub-band coefficients, resulting in weaker filtering intensity in edge regions and stronger filtering intensity in flat regions. After edge-preserving filtering, noise in the third frequency sub-band coefficient distribution map is effectively suppressed, while edge abrupt changes in the geometric structure are preserved.

[0029] Step S1231: Calculate the gradient magnitude of the third frequency sub-band coefficients to generate a gradient magnitude response map of the third frequency sub-band coefficients. The gradient magnitude of each pixel in the gradient magnitude response map is used to characterize the edge strength of the geometric component at the corresponding position.

[0030] In this embodiment, a gradient operator is used to calculate the gradient magnitude of the third frequency sub-band coefficients. The gradient operator includes horizontal and vertical convolution kernels, which are used to perform convolution operations on the distribution map of the third frequency sub-band coefficients to obtain horizontal and vertical gradient components. For each pixel, its gradient magnitude is obtained by calculating the square root of the sum of the squares of the horizontal and vertical gradient components. The larger the gradient magnitude, the stronger the edge intensity at that location, i.e., the more drastic the change in the geometric structure components at that location. The gradient magnitude response map generated in this way clearly reflects the distribution of geometric edges in the third frequency sub-band coefficients.

[0031] Step S1232: Construct an adaptive filtering window size adjustment function based on the gradient magnitude response map. The adaptive filtering window size adjustment function generates a first window size in the region where the gradient magnitude reaches a first preset threshold and generates a second window size in the region where the gradient magnitude does not reach a second preset threshold.

[0032] In this embodiment, an adaptive filtering window size adjustment function is constructed based on the gradient magnitude response map generated in step S1231. First, a first preset threshold and a second preset threshold are set, where the first preset threshold is greater than the second preset threshold. For each pixel in the gradient magnitude response map, when the gradient magnitude is greater than or equal to the first preset threshold, the region is determined to be a strong edge region. At this time, a first window size is generated, set to a smaller size to avoid over-filtering and edge blurring. When the gradient magnitude is less than the second preset threshold, the region is determined to be a flat region. At this time, a second window size is generated, set to a larger size to enhance the filtering effect on noise. When the gradient magnitude is between the second and first preset thresholds, a linear interpolation method is used to determine the window size, which gradually decreases from a larger size to a smaller size as the gradient magnitude increases. Through this adaptive window size adjustment function, the filtering window size can be dynamically adjusted according to the edge strength of different regions, effectively removing noise from flat regions while ensuring the preservation of edge information.

[0033] Step S1233: Perform an adaptive bilateral filtering operation on each third frequency sub-band coefficient, using the window size determined by the adaptive filtering window size adjustment function as the filtering neighborhood range, and using spatial distance weight and pixel value similarity weight as filtering coefficients to perform a weighted average calculation on the third frequency sub-band coefficients to generate the edge-preserving filtering third frequency sub-band coefficient value.

[0034] In this embodiment, an adaptive bilateral filtering operation is performed on each third frequency sub-band coefficient. For each coefficient value to be filtered, the size of the filtering window at the location of the coefficient is first obtained according to the adaptive filtering window size adjustment function determined in step S1232. Then, within this window range, the spatial distance weight and pixel value similarity weight between each neighboring coefficient value and the coefficient value to be filtered are calculated. The spatial distance weight is calculated using a Gaussian function, with the center of the window as the origin, and the weight decreases as the distance increases. The pixel value similarity weight is also calculated using a Gaussian function, with the weight increase as the pixel value difference decreases. Finally, the spatial distance weight and pixel value similarity weight are multiplied to obtain the comprehensive weight of each neighboring coefficient. A weighted average is then calculated for the coefficient values ​​within the neighborhood to obtain the edge-preserving filtered third frequency sub-band coefficient value. Through this adaptive bilateral filtering operation, different filtering parameters can be used in different regions to achieve the dual goals of edge preservation and noise removal.

[0035] Step S1234: Extract the residual information between the filtered third frequency sub-band coefficient value and the original third frequency sub-band coefficient, perform detail supplementation processing on the residual information, and superimpose the residual information back onto the filtered third frequency sub-band coefficient value according to a preset ratio coefficient to compensate for the texture details lost during the edge preservation filtering process, and obtain the third frequency sub-band coefficient distribution map after detail supplementation.

[0036] In this embodiment, the residual information between the filtered third frequency sub-band coefficient value and the original third frequency sub-band coefficient value is first calculated. The residual information reflects the details lost during the filtering process, especially some high-frequency texture information. Then, the residual information is processed to supplement details using a threshold truncation method. The portion of the residual whose absolute value is greater than a preset threshold is retained, while the portion less than or equal to the preset threshold is set to zero to remove noise components from the residual. The preset threshold is determined based on the noise level of the third frequency sub-band coefficient and is set by statistically analyzing the standard deviation of the residual. Next, the processed residual information is superimposed back onto the filtered third frequency sub-band coefficient value according to a preset scaling factor. The preset scaling factor ranges from zero to one and is adjusted according to the required level of detail preservation. A larger value supplements more details but may introduce noise; a smaller value supplements less details but results in a smoother filtering effect. Through this detail supplementation process, the texture details lost during edge-preserving filtering can be effectively compensated, so that the third frequency sub-band coefficient distribution map has both good noise suppression effect and retains rich geometric structural details.

[0037] Step S124: Perform inverse spatial frequency domain transformation on the compressed first frequency subband coefficient distribution map and the filtered third frequency subband coefficient distribution map to generate a substrate reflection component image and a geometric structure component image with the same spatial resolution as each original surface reflection image.

[0038] In this embodiment, the compressed first frequency sub-band coefficient distribution map and the filtered third frequency sub-band coefficient distribution map are subjected to inverse spatial frequency domain transformation. Since two-dimensional discrete wavelet transform was used for decomposition in step S121, the corresponding inverse transform here is two-dimensional discrete wavelet inverse transform. During the inverse transform process, the compressed first frequency sub-band coefficients are taken as low-frequency components, the filtered third frequency sub-band coefficients are taken as high-frequency components, and the second frequency sub-band coefficients are not used in this embodiment and are set as zero matrices. The inverse transform process is the reverse of the decomposition process, and the frequency domain coefficients are converted back to the spatial domain image through the wavelet reconstruction algorithm. The specific steps are as follows: first, upsampling and filtering operations are performed on each frequency sub-band coefficient, and then the reconstruction results of different frequency sub-bands are superimposed to obtain the spatial domain image. After the inverse transform processing, the generated base reflection component image and geometric structure component image have the same spatial resolution as the original surface reflection image. The base reflection component image mainly reflects the uniform reflection characteristics of the sheet metal surface material, while the geometric structure component image highlights the contour and shape changes of the surface. Together, they constitute the multi-scale feature decoupling result of the original image.

[0039] Step S125: Perform pixel value normalization mapping processing on the substrate reflection component image and the geometric structure component image respectively, map the pixel value distribution range of the substrate reflection component image to a first preset value interval to obtain a set of substrate reflection component images, and map the pixel value distribution range of the geometric structure component image to a second preset value interval to obtain a set of geometric structure component images.

[0040] In this embodiment, pixel value normalization mapping is performed on the substrate reflection component image and the geometric structure component image respectively. For the substrate reflection component image, the minimum and maximum values ​​of its pixel values ​​are first calculated, and then each pixel value is mapped to a first preset numerical interval using a linear mapping formula. The upper and lower limits of the first preset numerical interval are set according to the image display and subsequent processing requirements. For the geometric structure component image, the minimum and maximum values ​​of its pixel values ​​are also calculated, and then each pixel value is mapped to a second preset numerical interval using a linear mapping formula. The setting of the second preset numerical interval facilitates subsequent feature fusion processing. During the normalization process, if the difference between the maximum and minimum values ​​of a pixel value is zero, that is, all pixel values ​​in the image are the same, then all pixel values ​​are mapped to the middle value of the corresponding interval. Through the normalization mapping process, the pixel values ​​of the substrate reflection component image and the geometric structure component image are distributed within the preset numerical intervals. After the above processing, a set of substrate reflection component images and a set of geometric structure component images are obtained, and the number of images in each set is the same as the original surface reflection image set.

[0041] Step S130: Construct a defect-sensitive feature fusion map based on the set of substrate reflection component images and the set of geometric structure component images. The feature vector of each pixel in the defect-sensitive feature fusion map is composed of the substrate reflection component value and the geometric structure component value at the corresponding position.

[0042] In this embodiment, a defect-sensitive feature fusion map is constructed based on the set of substrate reflection component images and the set of geometric structure component images obtained in step S120. First, local texture difference analysis is performed on the substrate reflection component images to extract features reflecting changes in surface material uniformity; local curvature change analysis is performed on the geometric structure component images to extract features reflecting abrupt changes in surface shape. Then, these two types of features are fused at the pixel level to form a feature vector for each pixel. The feature fusion method uses feature dimension concatenation, combining the substrate reflection difference features and geometric curvature change features into a higher-dimensional feature vector according to their feature dimensions. To reduce the feature dimension and remove redundant information, adaptive compression processing is performed on the fused feature vector, and principal component analysis is used to extract the main feature components. Finally, the processed feature vectors are arranged according to the spatial position of the pixels to construct the defect-sensitive feature fusion map. This fusion map can comprehensively reflect the material and geometric structure features of the sheet metal surface.

[0043] Step S131: Perform local texture difference analysis on each of the substrate reflection component images in the substrate reflection component image set, calculate the degree of difference between each pixel and its neighboring pixels in the substrate reflection component value, and generate a substrate reflection difference feature map.

[0044] In this embodiment, local texture difference analysis is performed on each substrate reflection component image. For each pixel in the image, a local neighborhood window is determined. The window size is adaptively adjusted based on the geometric curvature change value corresponding to that pixel in the geometric structure component image; the larger the curvature change value, the smaller the window size, to improve sensitivity to local details. Then, the mean and median of the substrate reflection component values ​​of all pixels within the neighborhood window are calculated. By comparing the difference between the pixel's own substrate reflection component value and the mean and median, the degree of difference between that pixel and its neighboring pixels is measured. Simultaneously, considering the skewness of the distribution of substrate reflection component values ​​within the neighborhood, the difference values ​​are weighted and adjusted to make the calculation of the degree of difference more accurate under skewed distribution conditions. Finally, the difference degree values ​​of each pixel are arranged according to spatial location to generate a substrate reflection difference feature map. This map can highlight areas of material inhomogeneity in the substrate reflection component image, which are often locations where defects may exist.

[0045] Step S1311: Determine the local neighborhood window for each pixel, wherein the size of the local neighborhood window is adaptively adjusted according to the geometric curvature change value corresponding to the pixel in the geometric structure component image.

[0046] In this embodiment, for each pixel in the substrate reflection component image, pixels with the same coordinates are found in the corresponding geometric structure component image, and the geometric curvature change value of that pixel is obtained. The geometric curvature change value reflects the degree of curvature of the sheet metal surface at that location. The larger the curvature change value, the more drastic the surface shape change, requiring a smaller neighborhood window to capture local texture details; the smaller the curvature change value, the relatively flat the surface, allowing for a larger neighborhood window to analyze texture differences. A minimum and maximum window size are set, both of which are odd numbers to ensure that the window center is aligned with the pixel. The neighborhood window size is determined based on the relationship between the geometric curvature change value and a preset curvature threshold: when the geometric curvature change value is greater than or equal to the higher curvature threshold, the window size is the minimum window size; when the geometric curvature change value is less than or equal to the lower curvature threshold, the window size is the maximum window size; when the geometric curvature change value is between the lower and higher curvature thresholds, the window size is calculated through linear interpolation. This adaptive adjustment method allows the size of the local neighborhood window to adapt to regions with different surface shapes, improving the accuracy of local texture difference analysis.

[0047] Step S1312: Calculate the mean and median of the basal reflectance components of all pixels within the local neighborhood window to obtain the mean local reflectance parameter and the median local reflectance parameter.

[0048] In this embodiment, for the local neighborhood window determined in step S1311, all pixels within the window are traversed, and the basal reflectance component values ​​of these pixels are collected. The local reflectance mean parameter is calculated, which is the sum of the basal reflectance component values ​​of all pixels within the window divided by the number of pixels. Then, these basal reflectance component values ​​are sorted in ascending order. If the number of pixels is odd, the local reflectance median parameter is the value at the middle position of the sorted sequence; if the number of pixels is even, the local reflectance median parameter is the average of the two middle values ​​of the sorted sequence. The local reflectance mean parameter reflects the average level of the basal reflectance component values ​​within the neighborhood, while the local reflectance median parameter reflects the median level of the basal reflectance component values ​​within the neighborhood. Combining the two can more comprehensively describe the basal reflectance distribution characteristics within the neighborhood.

[0049] Step S1313: Construct a local reflection distribution skewness coefficient based on the local reflection mean parameter and the local reflection median parameter. The local reflection distribution skewness coefficient is used to characterize the distribution symmetry of the base reflection component values ​​within the local neighborhood window.

[0050] In this embodiment, a local reflectance distribution skewness coefficient is constructed based on the local reflectance mean parameter and local reflectance median parameter obtained in step S1312. The skewness coefficient is calculated by dividing the difference between the mean and the median by the standard deviation. The standard deviation is calculated by first squaring the difference between each pixel value and the mean, then averaging these squared values, and finally taking the square root. When the mean is greater than the median, the skewness coefficient is positive, indicating a right-skewed distribution; when the mean is less than the median, the skewness coefficient is negative, indicating a left-skewed distribution; when the mean equals the median, the skewness coefficient is zero, indicating a symmetrical distribution. The local reflectance distribution skewness coefficient reflects the distribution pattern of the base reflectance component values ​​within the neighborhood.

[0051] Step S1314: Perform a difference operation between the base reflection component value of the pixel and the local reflection mean parameter to obtain a first difference value, and perform a difference operation between the base reflection component value of the pixel and the local reflection median parameter to obtain a second difference value.

[0052] In this embodiment, for the current pixel, its base reflection component value is the value to be processed. A first difference value is calculated, which is the difference between the value to be processed and the local mean reflection parameter. A second difference value is calculated, which is the difference between the value to be processed and the local median reflection parameter. The first difference value reflects the degree of deviation of the pixel value from the neighborhood average, and the second difference value reflects the degree of deviation of the pixel value from the neighborhood median. Since the mean is more sensitive to extreme values, while the median is not, the first and second difference values ​​reflect the differences between the pixel and its neighborhood from different perspectives.

[0053] Step S1315: Perform weighted fusion on the first difference value and the second difference value, and use the local reflectance distribution skewness coefficient as the weight adjustment factor for weighted fusion to calculate the degree of difference between the pixel and its neighboring pixels in the substrate reflectance component value.

[0054] In this embodiment, the first and second difference values ​​are weighted and fused. The sum of the weighting coefficients is set to one. The weighting coefficients are determined using the local reflectance distribution skewness coefficient as an adjustment factor. When the absolute value of the skewness coefficient is small, it indicates that the distribution is nearly symmetrical; in this case, the mean and median are close, and the weighting coefficients can be set to be equal. When the skewness coefficient is positive and large, it indicates that the distribution is right-skewed with large extreme values; in this case, the median better represents the central tendency of the neighborhood. Therefore, the weight of the second difference value is increased, and the weight of the first difference value is decreased. When the skewness coefficient is negative and large in absolute value, it indicates that the distribution is left-skewed; similarly, the weight of the second difference value is increased, and the weight of the first difference value is decreased. Then, the degree of difference is the absolute value of the sum of the weighted first and second difference values. Taking the absolute value is to unify the direction of the degree of difference measurement. Through this weighted fusion method, the weights of the mean and median can be dynamically adjusted according to the skewness of the neighborhood distribution, making the calculation of the degree of difference more accurate and robust.

[0055] Step S1316: Arrange the difference values ​​of all pixels according to their spatial location to generate a substrate reflection difference feature map.

[0056] In this embodiment, the difference value of each pixel calculated in step S1315 is arranged according to its spatial position in the substrate reflection component image to form a substrate reflection difference feature map with the same size as the substrate reflection component image. The value of each pixel in the substrate reflection difference feature map is the corresponding difference value. The larger the value, the greater the difference between the pixel and its neighboring pixels in the substrate reflection component value, suggesting that there may be a material defect at that location. The substrate reflection difference feature map is normalized to map the difference value to a specific numerical range. The normalization formula is (difference value minus minimum value) divided by (maximum value minus minimum value), where the minimum and maximum values ​​are the minimum and maximum values ​​of the difference value in the substrate reflection difference feature map, respectively. The generated substrate reflection difference feature map clearly reflects the local differences in substrate reflection on the sheet metal surface.

[0057] Step S132: Perform local curvature change analysis on each geometric component image in the geometric component image set, calculate the average curvature change rate of the local surface where each pixel is located, and generate a geometric curvature change feature map.

[0058] In this embodiment, local curvature change analysis is performed on each geometric component image. The geometric component images reflect the contour shape of the sheet metal surface. By calculating the average rate of curvature change of the local surface where each pixel is located, the degree of abrupt change in surface shape can be quantified. First, Gaussian filtering is applied to the geometric component images to remove noise interference. Then, the second-order partial derivatives of the images are calculated to obtain the basic elements required for curvature calculation. Next, the average curvature of each pixel is calculated based on the second-order partial derivatives. Finally, the rate of change of the average curvature within the local neighborhood is calculated to obtain the average rate of curvature change. The average rate of curvature change of each pixel is arranged according to its spatial location to generate a geometric curvature change feature map. This feature map can highlight areas where the surface shape undergoes drastic changes; these areas are often the locations of defects (such as depressions, protrusions, etc.).

[0059] Step S133: The difference value of each pixel in the substrate reflection difference feature map is concatenated with the curvature change rate value of the corresponding pixel in the geometric curvature change feature map by feature dimension to form an initial fusion feature vector for each pixel.

[0060] In this embodiment, the substrate reflection difference feature map and the geometric curvature change feature map are concatenated along their feature dimensions. For each pixel, the degree of difference is obtained from the substrate reflection difference feature map, and the rate of curvature change is obtained from the geometric curvature change feature map. These two values ​​are combined into a two-dimensional initial fused feature vector. The feature dimension concatenation method arranges the values ​​of the two different features in the order of their feature dimensions to form a higher-dimensional feature vector. In this way, features reflecting material differences and geometric shape changes are combined, and the initial fused feature vector of each pixel contains both local difference information of substrate reflection and curvature change information of the geometric structure.

[0061] Step S134: Perform adaptive compression of the feature dimension on the initial fusion feature vector of each pixel, and extract the feature components in the initial fusion feature vector whose cumulative variance contribution rate exceeds the preset ratio by principal component analysis to obtain the dimensionality-reduced fusion feature vector.

[0062] In this embodiment, adaptive compression of the feature dimension is performed on the initial fused feature vector. First, the initial fused feature vectors of all pixels are collected to form a feature matrix. Then, principal component analysis is performed on the feature matrix to calculate the covariance matrix of the feature matrix. Eigenvalue decomposition is then performed on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The magnitude of the eigenvalue represents the variance contribution of the corresponding principal component. The eigenvalues ​​are arranged in descending order, and the cumulative variance contribution rate is calculated. When the cumulative variance contribution rate exceeds a preset ratio, the top few principal components are selected as the main feature components, where the number of principal components is the minimum number required for the cumulative variance contribution rate to exceed the preset ratio. For each initial fused feature vector, the dimensionality-reduced fused feature vector is obtained by multiplying it with the eigenvector matrix of the top few principal components. Feature dimensionality reduction through principal component analysis can reduce the dimension of the feature vector while retaining the main feature information.

[0063] Step S135: Arrange the dimensionality-reduced fused feature vectors according to the spatial position of the pixels to construct a defect-sensitive feature fusion map with the same spatial resolution as the original surface reflection image. The feature vector dimension of each pixel in the defect-sensitive feature fusion map is equal to the dimension of the dimensionality-reduced fused feature vector.

[0064] In this embodiment, the dimensionality-reduced fused feature vectors of each pixel obtained in step S134 are arranged according to their spatial positions in the original surface reflection image. The defect-sensitive feature fusion map is a three-dimensional data structure with the same width and height as the original surface reflection image and a depth equal to the dimension of the dimensionality-reduced fused feature vectors. The corresponding dimensionality-reduced fused feature vector is stored at each location. The defect-sensitive feature fusion map constructed in this way integrates key feature information of substrate reflection differences and geometric curvature changes, and has the same spatial resolution as the original image. The feature vector of each pixel in the defect-sensitive feature fusion map contains rich clues about whether a defect exists at that location, and is the core feature representation for defect detection.

[0065] Step S140: Call the pre-trained defect region localization network to process the defect-sensitive feature fusion map and generate a pixel-level defect mask image for identifying the defect region on the surface of the sheet metal part. The pixel-level defect mask image contains a confidence score value for each pixel to belong to the defect region.

[0066] In this embodiment, a pre-trained defect region localization network is invoked to process the defect-sensitive feature fusion map constructed in step S130. This defect region localization network is a deep learning-based semantic segmentation network capable of classifying each pixel in an image and determining whether it belongs to a defect region. The network input is the defect-sensitive feature fusion map, and the output is a pixel-level defect mask image. Each pixel's value is a confidence score indicating that it belongs to a defect region, with the value ranging from a specific interval. The closer the value is to the upper limit of the interval, the greater the likelihood that the pixel belongs to a defect region. The training process of the defect region localization network utilizes a large amount of sheet metal surface image data labeled with defect regions. Backpropagation is used to optimize the network parameters, enabling it to accurately identify various types of surface defects. When invoking the network, the defect-sensitive feature fusion map is first pre-processed, including normalization and dimensionality adjustment, to meet the network's input requirements. Then, the pre-processed feature map is input into the network, and through feature extraction, feature fusion, and pixel classification processes, a pixel-level defect mask image is obtained.

[0067] Step S141: Input the defect-sensitive feature fusion map into the feature encoder of the defect region localization network, and perform step-by-step downsampling processing on the defect-sensitive feature fusion map through multiple cascaded dilated convolution residual modules to extract a set of defect semantic feature maps at different spatial scales.

[0068] In this embodiment, the defect-sensitive feature fusion map is input into the feature encoder of the defect region localization network. The feature encoder consists of multiple cascaded dilated convolutional residual modules. Each module contains two convolutional layers, a batch normalization layer, an activation function, and a shortcut connection. The dilation rate of the dilated convolution gradually increases from the first module to the last module, enabling each module to capture contextual information of different ranges. After each dilated convolutional residual module, downsampling is performed through a convolution operation with a specific stride, reducing the spatial resolution of the feature map by a certain proportion while increasing the number of channels by a certain factor. The feature encoder contains multiple cascaded dilated convolutional residual modules. After step-by-step downsampling, multiple defect semantic feature maps at different spatial scales are obtained, with the spatial resolution decreasing sequentially and the number of channels increasing sequentially. These defect semantic feature maps at different spatial scales contain rich defect information ranging from local details to global context.

[0069] Step S142: Input the set of defect semantic feature maps at different spatial scales into the feature pyramid fusion module of the defect region localization network, and perform multi-scale feature fusion on the defect semantic feature maps at different spatial scales through top-down path and lateral connection path to generate a multi-scale fused feature map with spatial details and structural correlation information.

[0070] In this embodiment, a set of defect semantic feature maps at different spatial scales is input into the feature pyramid fusion module. The feature pyramid fusion module uses a combination of top-down and lateral connection paths for multi-scale feature fusion. The top-down path starts with the defect semantic feature map with the lowest resolution, upsamples it to the same resolution as the previous layer's feature map, and then fuses it with the corresponding scale feature map from the lateral connection path. The lateral connection path directly transmits defect semantic feature maps of different scales from the feature encoder to the fusion module, adjusts the number of channels through convolution, and then fuses them with the upsampled feature map from the top-down path. The fusion method uses element-wise addition, followed by feature refinement through convolution to eliminate the aliasing effect caused by upsampling. This process continues from the highest layer to the lowest layer, ultimately generating a multi-scale fused feature map with the same resolution as the original defect-sensitive feature fusion map. This multi-scale fused feature map contains both the semantic information of high-level features and retains the spatial detail information of low-level features.

[0071] Step S1421: Sort the defect semantic feature map sets at different spatial scales according to spatial resolution to obtain a first spatial resolution feature map, a second spatial resolution feature map set, and a third spatial resolution feature map.

[0072] In this embodiment, the defect semantic feature map sets obtained in step S141 at different spatial scales are sorted by spatial resolution. Each feature map set contains multiple feature maps arranged from highest to lowest spatial resolution. The feature map with the highest resolution is designated as the first spatial resolution feature map, the feature map with the lowest resolution is designated as the third spatial resolution feature map, and the feature maps in between are designated as the second spatial resolution feature map set. This sorting method clarifies the spatial scale relationship between different feature maps.

[0073] Step S1422: Perform an upsampling operation on the third spatial resolution feature map to expand the spatial size of the third spatial resolution feature map to be consistent with the spatial size of the adjacent second spatial resolution feature map, thereby obtaining a first upsampled feature map.

[0074] In this embodiment, an upsampling operation is performed on the third spatial resolution feature map. The upsampling uses bilinear interpolation to enlarge the width and height of the third spatial resolution feature map to the same size as the adjacent second spatial resolution feature map. Bilinear interpolation determines the value of a pixel by calculating the weighted average of the four nearest pixels surrounding the target pixel, resulting in a smoother upsampling result. After upsampling, a first upsampled feature map is obtained, whose spatial dimensions are the same as the adjacent second spatial resolution feature map. Figure 1 To.

[0075] Step S1423: Add the first upsampled feature map to the adjacent second spatial resolution feature map element by element, and refine the addition result through a convolutional layer to obtain the first refined fused feature map.

[0076] In this embodiment, the first upsampled feature map is added element-wise with the adjacent second spatial resolution feature map. Element-wise addition requires that the spatial dimensions and number of channels of the two feature maps be exactly the same. Therefore, before addition, it is necessary to ensure that the number of channels in the second spatial resolution feature map is the same as that in the first upsampled feature map. If the number of channels is different, the number of channels in the second spatial resolution feature map is adjusted through a convolutional layer. The added feature map contains a preliminary fusion of high-level semantic information and mid-level detail information. Then, the addition result is input into a convolutional layer for feature fusion refinement. The convolutional layer uses a specific-sized convolutional kernel to further extract and integrate the fused features, eliminate redundant information, and enhance the expressive power of useful features. After processing by the convolutional layer, a first refined fused feature map is obtained, which integrates information from the third spatial resolution feature map and the adjacent second spatial resolution feature map.

[0077] Step S1424: Continue to perform upsampling operation on the first refined fused feature map to expand the spatial size of the first refined fused feature map to be consistent with the spatial size of the first spatial resolution feature map, and obtain the second upsampled feature map.

[0078] In this embodiment, an upsampling operation is performed on the first refined fused feature map. Using the same bilinear interpolation method as in step S1422, the spatial size of the first refined fused feature map is expanded to match the spatial size of the first spatial resolution feature map. During the upsampling process, the number of channels in the feature map remains unchanged. After the upsampling process, a second upsampled feature map is obtained, whose spatial size is the same as the first spatial resolution feature map, thus preparing for fusion with the first spatial resolution feature map.

[0079] Step S1425: Perform element-wise multiplication of the second upsampled feature map with the first spatial resolution feature map, and refine the multiplication result through a convolutional layer to generate a multi-scale fusion feature map with spatial details and structural correlation information.

[0080] In this embodiment, the second upsampled feature map is multiplied element-wise with the first spatial resolution feature map. Element-wise multiplication highlights regions where corresponding feature values ​​in both feature maps are large, enhancing the saliency of the features. The multiplied feature map combines high-level semantic information and low-level spatial detail information. Then, the multiplication result is input into a convolutional layer for feature refinement. The convolutional layer uses a specific-sized convolutional kernel and activation function to perform further nonlinear transformations and feature extraction on the fused features, ultimately generating a multi-scale fused feature map. This multi-scale fused feature map has the same spatial resolution as the original defect-sensitive feature fused map and contains defect semantic features extracted from different spatial scales.

[0081] Step S143: Input the multi-scale fused feature map into the pixel classifier of the defect region localization network, and calculate the original confidence score of each pixel belonging to the defect region by applying convolution operation pixel by pixel.

[0082] In this embodiment, the multi-scale fused feature map is input into the pixel classifier of the defect region localization network. The pixel classifier consists of multiple convolutional layers and activation functions. First, the number of channels in the multi-scale fused feature map is adjusted to the number of channels required for classification (one in this embodiment, corresponding to binary classification of defect regions and non-defect regions) through the convolutional layers. Then, an activation function (such as the sigmoid function) is applied to map the convolution result to the [0,1] interval, obtaining the original confidence score for each pixel belonging to the defect region. The pixel-by-pixel convolution operation ensures that the classification result of each pixel is based on the multi-scale fused features of its local neighborhood, thereby accurately determining whether the pixel belongs to the defect region. The higher the original confidence score, the greater the probability that the pixel belongs to the defect region.

[0083] Step S144: Perform spatial context constraint optimization on the original confidence score. Utilize the continuity characteristics of the pixels in the defect area in spatial distribution, and iteratively optimize the original confidence score through a conditional random field model to correct isolated false detection pixels and generate an optimized confidence score value for each pixel.

[0084] In this embodiment, the original confidence score is optimized using spatial context constraints. Since defect regions are typically spatially continuous, and the original confidence score may contain isolated high-scoring pixels (false positives) or low-scoring pixels (false negatives), spatial context information is needed for optimization. A conditional random field (CRF) model is employed, which considers the relationship between adjacent pixels and defines an energy function to describe the probability that a pixel belongs to a defect region. The energy function includes a univariate potential function and a binary potential function. The univariate potential function is based on the original confidence score, while the binary potential function considers the similarity (such as color, texture, etc.) between adjacent pixels; the higher the similarity, the greater the probability that adjacent pixels belong to the same category. The energy function is minimized using an iterative optimization algorithm (such as a confidence propagation algorithm) to obtain the optimized confidence score value for each pixel. After spatial context constraint optimization, the confidence scores of isolated false positive pixels are reduced, while the confidence scores of pixels in the defect region that are missed are increased, making the optimized confidence scores more consistent with the actual spatial distribution of the defect region.

[0085] Step S145: Generate a pixel-level defect mask image based on the optimized confidence score value of each pixel. The value of each pixel in the pixel-level defect mask image is equal to the optimized confidence score value at the corresponding position.

[0086] In this embodiment, the optimized confidence score values ​​of each pixel obtained in step S144 are arranged according to their spatial position to generate a pixel-level defect mask image. The size of the pixel-level defect mask image is the same as that of the original surface reflection image, and the value of each pixel is the optimized confidence score value, ranging from [0,1]. Through this mask image, the probability that each position on the surface of the sheet metal part belongs to a defect area can be seen intuitively.

[0087] Step S150: Based on the set of connected components formed by pixels whose confidence scores meet preset conditions in the pixel-level defect mask image, generate the defect contour boundary coordinate sequence and defect type discrimination result for each connected component.

[0088] In this embodiment, defect contour extraction and type discrimination are performed based on pixel-level defect mask images. First, the pixel-level defect mask image is binarized and segmented using a preset threshold. Pixels with confidence scores higher than the threshold are classified as defect foreground regions, while pixels with scores lower than or equal to the threshold are classified as normal background regions. Then, connected component analysis is performed on the binarized image to identify all interconnected defect foreground pixels forming connected regions. For each connected region, morphological processing is performed to fill internal holes and smooth edge contours, and then its outer boundary coordinate sequence is extracted. Simultaneously, shape features of each connected region, such as area, perimeter, roundness, and aspect ratio, are extracted. These shape features are input into a pre-trained defect type classifier to obtain the defect type discrimination result. Through this step, the location and contour of the defect region can be accurately located, and the type of defect can be determined.

[0089] Step S151: Perform a binarization segmentation operation on the pixel-level defect mask image, classify the pixels with confidence scores greater than a preset threshold as defect foreground regions, and classify the pixels with confidence scores less than or equal to the preset threshold as normal background regions, to obtain a defect binary mask image.

[0090] In this embodiment, a binarization segmentation operation is performed on the pixel-level defect mask image. A preset threshold is set based on the accuracy requirements of defect detection and a balance between false positive and false negative rates. Each pixel in the pixel-level defect mask image is traversed. If the confidence score of a pixel is greater than the preset threshold, the pixel is marked as a defect foreground region (value is one); if the confidence score is less than or equal to the preset threshold, the pixel is marked as a normal background region (value is zero). This binarization segmentation operation yields a defect binary mask image, which clearly shows the location and extent of the defect region.

[0091] Step S152: Perform a connected component labeling operation on the defect binary mask image to identify the connected components formed by all interconnected defect foreground pixels in the defect binary mask image and generate a connected component set.

[0092] In this embodiment, connected component labeling is performed on the defect binary mask image. The defect foreground pixels in the defect binary mask image are labeled using either region growing or scanline methods. Region growing starts with an unlabeled foreground pixel and merges its neighboring foreground pixels into the same region until no more neighboring foreground pixels remain. This process is repeated for the next unlabeled foreground pixel, ultimately resulting in all independent connected components. Each connected component is assigned a unique label value to distinguish different defect regions. The generated set of connected components contains the location and extent information of all detected defect regions, with each connected component corresponding to a potential defect.

[0093] Step S153: Perform a morphological closing operation on each connected component in the set of connected components to fill the hole regions inside the connected component and smooth the edge contour of the connected component to obtain a connected component with smoothed contour.

[0094] In this embodiment, a morphological closing operation is performed on each connected component in the set of connected components. Morphological closing is an operation that first dilates and then erodes, capable of filling small holes within the connected components and smoothing their edge contours. First, a suitable structuring element template is selected based on the size and shape of the connected component; the structuring element template can be circular, square, or other shapes. Then, a dilation operation is performed on the connected component, expanding its boundary outwards by a certain distance, connecting the boundary of the internal holes to the boundary of the connected component. Next, an erosion operation is performed on the dilated connected component, shrinking its boundary inwards by the same distance, restoring the original size of the connected component and filling the internal holes. Through the morphological closing operation, connected components with smooth contours and no internal holes are obtained.

[0095] For example, step S1531: Obtain the minimum bounding rectangle region of each connected component, and use the minimum bounding rectangle region as the operation range of the morphological closing operation.

[0096] In this embodiment, for each connected component, its minimum bounding rectangle region is calculated. The minimum bounding rectangle is the rectangle with the smallest area that completely contains the connected component, and its sides can be at an angle to the image coordinate axes. By calculating the extreme values ​​of the coordinates of all pixels in the connected component, the position and size of the minimum bounding rectangle can be determined. Using the minimum bounding rectangle region as the operating range for morphological closing operations can reduce unnecessary processing of the background region and improve computational efficiency. At the same time, limiting the operating range can prevent morphological operations from affecting adjacent connected components.

[0097] Step S1532: Determine the location of the internal hole region of the connected domain, identify the boundary pixels of the internal hole region by scanning the pixel matrix of the connected domain, and generate a set of internal hole regions.

[0098] In this embodiment, internal hole regions are identified by scanning the pixel matrix of connected components. An internal hole region refers to a background pixel region completely surrounded by connected component pixels. During the scanning process, starting from the boundary pixels of the connected components, a flooding fill algorithm is used to mark the background region; the unmarked background region is the internal hole region. The boundary pixels of the internal hole regions are identified, and these boundary pixels constitute the outline of the hole. A set of internal hole regions is generated, recording the position and extent of each hole region.

[0099] Step S1533: Construct a structuring element template for morphological closing operations, wherein the size of the structuring element template is adaptively adjusted according to the ratio of the area parameter and the perimeter parameter of the connected region.

[0100] In this embodiment, a structuring element template for morphological closing operations is constructed. The size of the structuring element template is adaptively adjusted based on the ratio of the area parameter to the perimeter parameter of the connected region. The area parameter is the number of pixels contained in the connected region, and the perimeter parameter is the number of pixels at the boundary of the connected region. When the ratio of the area parameter to the perimeter parameter is large, it indicates that the shape of the connected region is relatively compact and the holes are small, in which case the size of the structuring element template can be small; when the ratio is small, it indicates that the shape of the connected region is relatively long and narrow or irregular, and the holes may be large, in which case the size of the structuring element template needs to be large. The size adjustment range of the structuring element template is between a preset minimum size and a preset maximum size, ensuring that it can effectively fill the holes without excessively blurring the outline of the connected region.

[0101] Step S1534: Perform an expansion operation on the connected domain using the structuring element template, extending the boundary of the connected domain outward by a preset pixel distance, so that the boundary of the internal hole region is connected to the boundary of the connected domain, thus obtaining the expanded connected domain.

[0102] In this embodiment, the structuring element template constructed in step S1533 is used to perform an expansion operation on the connected components. The expansion operation involves sliding the structuring element template across the connected components. When the center of the structuring element template is located at a foreground pixel within the connected components, all pixels within the coverage area of ​​the structuring element template are marked as foreground pixels. Through the expansion operation, the boundaries of the connected components expand outwards, connecting the boundaries of the internal hole regions with the boundaries of the connected components. The expansion distance is determined by the size of the structuring element template; the larger the structuring element template, the greater the expansion distance.

[0103] Step S1535: Use the same structuring element template as the dilation operation to perform an erosion operation on the dilated connected domain, shrinking the outward-expanding boundary in the dilation operation inward to restore it, while keeping the internal hole area filled, thus generating a hole-filled connected domain.

[0104] In this embodiment, the same structuring element template as used in the dilation operation is employed to perform an erosion operation on the dilated connected region. The erosion operation involves sliding the structuring element template across the dilated connected region. Only when all pixels within the template's coverage area are foreground pixels is the center pixel marked as a foreground pixel. Through the erosion operation, the outward-expanding boundary during dilation is shrunk inward, restoring the connected region to its original size and shape. Simultaneously, since the internal hole regions were already connected to the connected region boundary during dilation, these hole regions are filled with foreground pixels after the erosion operation. The resulting connected region, after hole filling, has no holes and a more complete outline.

[0105] Step S1536: Perform contour smoothing on the connected domain after hole filling to filter out jagged noise points on the edge contour of the connected domain and obtain the contour smoothed connected domain.

[0106] In this embodiment, contour smoothing is performed on the connected components after hole filling. Polynomial curve fitting or moving average filtering is used to smooth the edge contour points of the connected components. For each pixel on the edge contour, considering its multiple neighboring contour points, the position of the pixel is adjusted by fitting a curve or calculating a moving average to eliminate jagged noise points. Contour smoothing makes the edges of the connected components smoother, reducing errors in subsequent contour extraction and shape feature calculation. After contour smoothing, the smoothed connected components are obtained.

[0107] Step S154: Extract the coordinates of the outer boundary pixels of the connected domains after contour smoothing, and arrange them according to the spatial adjacency of the boundary pixels to generate a sequence of defect contour boundary coordinates for each connected domain.

[0108] In this embodiment, the coordinates of the outer boundary pixels of the connected components after contour smoothing are extracted. A boundary tracking algorithm is employed, starting from a boundary pixel of the connected component, and sequentially tracking adjacent boundary pixels in a clockwise or counter-clockwise direction until returning to the starting point, forming a closed boundary contour. During the tracking process, the coordinates (x, y) of each boundary pixel are recorded. These coordinates are arranged according to the tracking order to generate a sequence of defect contour boundary coordinates for each connected component. This sequence of defect contour boundary coordinates accurately describes the shape and location of the defect region and can be used for subsequent defect visualization and dimensional measurement.

[0109] Step S155: Perform shape feature extraction on the smoothed connected components, calculate the area parameter, perimeter parameter, roundness parameter and aspect ratio parameter of the connected components, and construct a shape feature vector based on the combination of the area parameter, perimeter parameter, roundness parameter and aspect ratio parameter. Input the shape feature vector into a pre-trained defect type classifier and output the defect type discrimination result corresponding to each connected component.

[0110] In this embodiment, shape feature extraction is performed on the smoothed connected components. The area parameter (number of pixels within the connected component) is calculated; the perimeter parameter (number of pixels at the boundary of the connected component) is calculated; the roundness parameter (formulated as (perimeter squared) divided by (area multiplied by four times pi)) is calculated, with a roundness parameter closer to one indicating a closer approximation of a circle; and the aspect ratio parameter (ratio of the length to the width of the smallest bounding rectangle of the connected component) is calculated. The area, perimeter, roundness, and aspect ratio parameters are combined to form a shape feature vector. This shape feature vector is then input into a pre-trained defect type classifier. This classifier uses machine learning algorithms such as Support Vector Machines or Random Forests to learn shape feature patterns of different defect types through training samples, thereby classifying the input shape feature vector and outputting the defect type classification result for each connected component, such as scratches, dents, bumps, and impurities.

[0111] Figure 2 The illustration shows exemplary hardware and software components of a sheet metal pre-painting surface defect detection system 100 that can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used on the sheet metal pre-painting surface defect detection system 100 and to perform the functions in this application.

[0112] The sheet metal part surface defect detection system 100 before painting can be a general-purpose server or a special-purpose server, both of which can be used to implement the sheet metal part surface defect detection method before painting of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0113] For example, the sheet metal surface defect detection system 100 before painting may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the sheet metal surface defect detection system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The sheet metal surface defect detection system 100 also includes an I / O interface 150 between a computer and other input / output devices.

[0114] For ease of explanation, only one processor is described in the sheet metal part pre-painting surface defect detection system 100. However, it should be noted that the sheet metal part pre-painting surface defect detection system 100 of this application may also include multiple processors, and therefore the steps performed by one processor described in this application may also be performed jointly by multiple processors or individually. For example, if the processor of the sheet metal part pre-painting surface defect detection system 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0115] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned method for detecting surface defects of sheet metal parts before painting is implemented.

[0116] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for detecting surface defects of sheet metal parts before painting, characterized in that, The method includes: Acquire a set of raw surface reflection images of sheet metal parts collected by optical inspection equipment; Multi-scale surface feature decoupling processing is performed on the original surface reflection image set to decompose the original surface reflection image set into a set of substrate reflection component images characterizing the substrate material of the sheet metal part and a set of geometric structure component images characterizing the surface contour of the sheet metal part. A defect-sensitive feature fusion map is constructed based on the set of base reflection component images and the set of geometric structure component images. The feature vector of each pixel in the defect-sensitive feature fusion map is composed of the base reflection component value and the geometric structure component value at the corresponding position. The pre-trained defect region localization network is invoked to process the defect-sensitive feature fusion map and generate a pixel-level defect mask image for identifying defect regions on the surface of sheet metal parts. The pixel-level defect mask image contains a confidence score value for each pixel to belong to the defect region. Based on the set of connected components formed by pixels in the pixel-level defect mask image whose confidence scores meet preset conditions, a defect contour boundary coordinate sequence and defect type discrimination result are generated for each connected component.

2. The method for detecting surface defects of sheet metal parts before painting according to claim 1, characterized in that, The step of performing multi-scale surface feature decoupling processing on the original surface reflection image set decomposes the original surface reflection image set into a set of substrate reflection component images representing the substrate material of the sheet metal part and a set of geometric structure component images representing the surface contour of the sheet metal part, including: Each original surface reflection image in the original surface reflection image set is subjected to spatial frequency domain transformation processing to obtain a multi-scale frequency coefficient distribution map of each original surface reflection image in the first frequency sub-band, the second frequency sub-band, and the third frequency sub-band. Nonlinear gain compression processing is performed on the first frequency sub-band coefficients in the multi-scale frequency coefficient distribution map to suppress the intensity fluctuation of the base reflection component in the first frequency sub-band, resulting in a compressed first frequency sub-band coefficient distribution map. An edge-preserving filter is applied to the third frequency sub-band coefficients in the multi-scale frequency coefficient distribution map to retain the edge abrupt information of the geometric components in the third frequency sub-band, thus obtaining the filtered third frequency sub-band coefficient distribution map. The compressed first frequency subband coefficient distribution map and the filtered third frequency subband coefficient distribution map are subjected to inverse spatial frequency domain transformation to generate a substrate reflection component image and a geometric structure component image with the same spatial resolution as each original surface reflection image. Pixel value normalization mapping is performed on the substrate reflection component image and the geometric structure component image respectively. The pixel value distribution range of the substrate reflection component image is mapped to a first preset value interval to obtain a set of substrate reflection component images, and the pixel value distribution range of the geometric structure component image is mapped to a second preset value interval to obtain a set of geometric structure component images.

3. The method for detecting surface defects of sheet metal parts before painting according to claim 2, characterized in that, The process of performing nonlinear gain compression processing on the first frequency sub-band coefficients in the multi-scale frequency coefficient distribution map to suppress the intensity fluctuation of the substrate reflection component in the first frequency sub-band, and obtaining the compressed first frequency sub-band coefficient distribution map, includes: Extract the amplitude statistical histogram of the first frequency sub-band coefficients from the multi-scale frequency coefficient distribution map, and determine the dynamic compression threshold based on the coefficient distribution density of each amplitude interval in the amplitude statistical histogram; A piecewise nonlinear compression function is constructed based on the dynamic compression threshold. The piecewise nonlinear compression function adopts a linear mapping relationship in the interval where the amplitude is less than the dynamic compression threshold, and adopts a logarithmic compression mapping relationship in the interval where the amplitude is greater than or equal to the dynamic compression threshold. The first frequency sub-band coefficient is input into the piecewise nonlinear compression function, and the amplitude compression calculation is performed on each first frequency sub-band coefficient to generate the compressed first frequency sub-band coefficient value. The compression ratio parameter of each first frequency sub-band coefficient is calculated based on the ratio of the compressed first frequency sub-band coefficient value to the original first frequency sub-band coefficient value. Spatially consistent smoothing is performed on the compressed first frequency sub-band coefficient values. Local neighborhood mean filtering is applied to the compressed first frequency sub-band coefficient values ​​using the spatial correlation between adjacent first frequency sub-band coefficients to eliminate local discontinuities introduced by nonlinear compression, resulting in a spatially smoothed compressed first frequency sub-band coefficient distribution map.

4. The method for detecting surface defects of sheet metal parts before painting according to claim 2, characterized in that, The process of performing edge-preserving filtering on the third frequency sub-band coefficients in the multi-scale frequency coefficient distribution map preserves the edge abrupt change information of the geometric structure components in the third frequency sub-band, resulting in a filtered third frequency sub-band coefficient distribution map, includes: The gradient magnitude of the third frequency sub-band coefficients is calculated to generate a gradient magnitude response map of the third frequency sub-band coefficients. The gradient magnitude of each pixel in the gradient magnitude response map is used to characterize the edge strength of the geometric component at the corresponding position. An adaptive filtering window size adjustment function is constructed based on the gradient magnitude response map. The adaptive filtering window size adjustment function generates a first window size in the region where the gradient magnitude reaches a first preset threshold and generates a second window size in the region where the gradient magnitude does not reach a second preset threshold. An adaptive bilateral filtering operation is performed on each third frequency sub-band coefficient. The window size determined by the adaptive filtering window size adjustment function is used as the filtering neighborhood range. The spatial distance weight and pixel value similarity weight are used as the filtering coefficients. The weighted average of the third frequency sub-band coefficients is calculated to generate the edge-preserving filtering third frequency sub-band coefficient value. The residual information between the filtered third frequency subband coefficient value and the original third frequency subband coefficient is extracted. The residual information is then processed to add details. The residual information is then superimposed back onto the filtered third frequency subband coefficient value according to a preset ratio coefficient to compensate for the texture details lost during the edge-preserving filtering process, resulting in a detailed distribution map of the third frequency subband coefficient.

5. The method for detecting surface defects of sheet metal parts before painting according to claim 1, characterized in that, The defect-sensitive feature fusion map is constructed based on the set of base reflection component images and the set of geometric structure component images. The feature vector of each pixel in the defect-sensitive feature fusion map is composed of the base reflection component value and the geometric structure component value at the corresponding location, including: For each base reflection component image in the base reflection component image set, perform local texture difference analysis, calculate the degree of difference between each pixel and its neighboring pixels in the base reflection component value, and generate a base reflection difference feature map; Perform local curvature change analysis on each geometric structure component image in the geometric structure component image set, calculate the average curvature change rate of the local surface where each pixel is located, and generate a geometric curvature change feature map. The difference value of each pixel in the substrate reflection difference feature map is concatenated with the curvature change rate value of the corresponding pixel in the geometric curvature change feature map by feature dimension to form the initial fusion feature vector of each pixel; For each pixel, the initial fused feature vector is subjected to adaptive compression of the feature dimension. The feature components with a cumulative variance contribution rate exceeding a preset ratio in the initial fused feature vector are extracted by principal component analysis to obtain the dimensionality-reduced fused feature vector. The dimensionality-reduced fused feature vectors are arranged according to the spatial position of pixels to construct a defect-sensitive feature fusion map with the same spatial resolution as the original surface reflection image. The feature vector dimension of each pixel in the defect-sensitive feature fusion map is equal to the dimension of the dimensionality-reduced fused feature vector.

6. The method for detecting surface defects of sheet metal parts before painting according to claim 5, characterized in that, The step involves performing local texture difference analysis on each of the substrate reflection component images in the substrate reflection component image set, calculating the degree of difference between each pixel and its neighboring pixels in the substrate reflection component value, and generating a substrate reflection difference feature map, including: A local neighborhood window is determined for each pixel, and the size of the local neighborhood window is adaptively adjusted according to the geometric curvature change value corresponding to the pixel in the geometric structure component image; Calculate the mean and median of the basal reflectance component values ​​of all pixels within the local neighborhood window to obtain the mean local reflectance parameter and the median local reflectance parameter. The local reflection distribution skewness coefficient is constructed based on the local reflection mean parameter and the local reflection median parameter. The local reflection distribution skewness coefficient is used to characterize the distribution symmetry of the base reflection component values ​​within the local neighborhood window. The first difference value is obtained by performing a difference operation between the base reflection component value of the pixel and the local reflection mean parameter; the second difference value is obtained by performing a difference operation between the base reflection component value of the pixel and the local reflection median parameter. The first difference value and the second difference value are weighted and fused, and the local reflectance distribution skewness coefficient is used as the weight adjustment factor for the weighted fusion to calculate the degree of difference between the pixel and its neighboring pixels in the base reflectance component value. Arrange the difference values ​​of all pixels according to their spatial location to generate a substrate reflection difference feature map.

7. The method for detecting surface defects of sheet metal parts before painting according to claim 1, characterized in that, The process of calling a pre-trained defect region localization network to process the defect-sensitive feature fusion map and generate a pixel-level defect mask image for identifying defect regions on the surface of the sheet metal part includes: The defect-sensitive feature fusion map is input into the feature encoder of the defect region localization network. The defect-sensitive feature fusion map is downsampled step by step through multiple cascaded dilated convolution residual modules to extract a set of defect semantic feature maps at different spatial scales. The set of defect semantic feature maps at different spatial scales is input into the feature pyramid fusion module of the defect region localization network. Multi-scale feature fusion is performed on the defect semantic feature maps at different spatial scales through top-down path and lateral connection path to generate multi-scale fused feature maps with spatial details and structural correlation information. The multi-scale fused feature map is input into the pixel classifier of the defect region localization network, and the original confidence score of each pixel belonging to the defect region is calculated by applying convolution operation pixel by pixel. Spatial context constraint optimization is performed on the original confidence score. By utilizing the continuous characteristics of the spatial distribution of pixels in the defect area, the original confidence score is iteratively optimized through a conditional random field model to correct isolated false detection pixels and generate an optimized confidence score value for each pixel. A pixel-level defect mask image is generated based on the optimized confidence score value of each pixel. The value of each pixel in the pixel-level defect mask image is equal to the optimized confidence score value at the corresponding position.

8. The method for detecting surface defects of sheet metal parts before painting according to claim 7, characterized in that, The feature pyramid fusion module, which inputs the set of defect semantic feature maps at different spatial scales into the defect region localization network, performs multi-scale feature fusion on the defect semantic feature maps at different spatial scales through top-down and lateral connection paths, generating a multi-scale fused feature map with spatial details and structural correlation information, including: The defect semantic feature map sets at different spatial scales are sorted according to spatial resolution to obtain a first spatial resolution feature map, a second spatial resolution feature map set, and a third spatial resolution feature map. An upsampling operation is performed on the third spatial resolution feature map to expand the spatial size of the third spatial resolution feature map to be consistent with the spatial size of the adjacent second spatial resolution feature map, thereby obtaining a first upsampled feature map. The first upsampled feature map is added element-wise to the adjacent second spatial resolution feature map, and the addition result is refined by feature fusion through a convolutional layer to obtain the first refined fused feature map. The first refined fused feature map is further upsampled to expand its spatial size to match that of the first spatial resolution feature map, thus obtaining the second upsampled feature map. The second upsampled feature map is multiplied element-wise with the first spatial resolution feature map, and the result of the multiplication is refined by a convolutional layer to generate a multi-scale fused feature map with spatial details and structural correlation information.

9. The method for detecting surface defects of sheet metal parts before painting according to claim 1, characterized in that, The step of generating a defect contour boundary coordinate sequence and defect type discrimination result for each connected component based on the set of connected components formed by pixels whose confidence scores in the pixel-level defect mask image meet preset conditions includes: A binarization segmentation operation is performed on the pixel-level defect mask image. Pixels with confidence scores greater than a preset threshold are classified as defect foreground regions, and pixels with confidence scores less than or equal to the preset threshold are classified as normal background regions, thus obtaining a defect binary mask image. Perform a connected component labeling operation on the defect binary mask image to identify the connected components formed by all interconnected defect foreground pixels in the defect binary mask image, and generate a connected component set; A morphological closing operation is performed on each connected component in the set of connected components to fill the void regions inside the connected components and smooth the edge contours of the connected components, resulting in a connected component with smoothed contours. Extract the coordinates of the outer boundary pixels of the connected components after contour smoothing, and arrange them according to the spatial adjacency of the boundary pixels to generate a sequence of defect contour boundary coordinates for each connected component; Shape feature extraction is performed on the connected components after contour smoothing. The area, perimeter, roundness, and aspect ratio parameters of the connected components are calculated. A shape feature vector is constructed based on the combination of the area, perimeter, roundness, and aspect ratio parameters. The shape feature vector is input into a pre-trained defect type classifier, and the defect type discrimination result corresponding to each connected component is output.

10. A surface defect detection system for sheet metal parts before painting, characterized in that, The sheet metal part surface defect detection system before painting includes a processor and a memory, the memory and the processor are connected, the memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the sheet metal part surface defect detection method before painting as described in any one of claims 1-9.

Images