Real-time detection and classification method for surface defects of copper foil based on machine vision

Abstract

This invention belongs to the field of image data processing, specifically a machine vision-based method for real-time detection and classification of surface defects on copper foil. It involves extracting consecutive frame images, calculating inter-frame optical flow fields to generate a motion-aware mask, and then multiplying the motion-aware mask element-wise with the multi-scale spatial feature map of the current frame to suppress background response in motion-blurred regions. The high-level semantic features of previous frames, aligned by optical flow spatial alignment, are then concatenated with the low-level detail features of the current frame at the channel level and subjected to dimensionality reduction mapping to obtain a reconstructed feature map of the defect edge gradient. Defects are located and classified based on this reconstructed feature map. This invention suppresses motion-blurred background response and utilizes temporal features to reconstruct defect edge gradients, solving the problem of loss and missed detection of fine defect features caused by motion blur in single-frame images during high-speed copper foil movement.

Description

Technical Field

[0001] This invention belongs to the field of image data processing, specifically a method for real-time detection and classification of surface defects on copper foil based on machine vision. Background Technology

[0002] Existing methods for detecting surface defects on copper foil mostly employ static spatial domain feature extraction schemes based on single-frame images. The specific implementation process involves acquiring a single-frame image of the copper foil surface using an image acquisition device, inputting this image into a deep convolutional neural network, extracting texture features layer by layer in the spatial dimension, fusing spatial feature maps of different scales through a feature pyramid network, and finally outputting the location and category of the defect through a classification and regression network. This type of scheme only performs feature extraction and fusion in the pure spatial dimension of a single-frame image.

[0003] The copper foil production line operates at extremely high speeds, and the image acquisition equipment moves relative to the copper foil during the exposure time, causing motion blur in the acquired single-frame images. When processing motion-blurred regions, the aforementioned static spatial domain feature extraction schemes fail to distinguish the boundaries between fine defects and background textures due to isolated spatial convolution operations. Blurred pixels are treated as normal textures during convolution, resulting in the loss of fine defect features and missed detections. Therefore, the core problem of existing technologies lies in the loss of fine defect features and missed detections caused by motion blur in single-frame images during high-speed copper foil movement. Summary of the Invention

[0004] The purpose of this invention is to provide a method for real-time detection and classification of surface defects on copper foil based on machine vision, which can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A machine vision-based method for real-time detection and classification of defects on copper foil surfaces includes: extracting consecutive frame images of the acquired copper foil surface and calculating the inter-frame optical flow field between the consecutive frame images; Based on the amplitude and direction distribution of the inter-frame optical flow field, a motion-sensing mask characterizing the degree of motion blur is generated; Extract the multi-scale spatial feature map of the current frame image, and perform element-wise multiplication and weighting of the motion-aware mask and the multi-scale spatial feature map of the current frame image to suppress the background feature response of the motion-blurred region in the current frame image; In the feature pyramid network, a recursive aggregation path for temporal features is constructed. The high-level semantic features of the previous frame image after being aligned with the inter-frame optical flow field space are combined with the low-level detail features of the current frame image at the channel level. The combined features are then subjected to dimensionality reduction mapping to obtain a reconstructed feature map of the defect edge gradient. Based on the reconstructed feature map, the surface defects of the copper foil are located and classified.

[0006] Preferably, the step of calculating the inter-frame optical flow field between the consecutive frame images includes: constructing an optical flow energy function that includes brightness constancy and gradient constancy, introducing a structural consistency constraint term into the optical flow energy function, wherein the structural consistency constraint term is calculated based on the image local structure tensor; The initial optical flow field of the consecutive frame images is obtained by iteratively solving the optical flow energy function. The initial optical flow field is subjected to median filtering and bilateral filtering to remove isolated noise points in the initial optical flow field and keep the optical flow edges smooth, thereby obtaining the target optical flow field. Based on the vector magnitude of each pixel in the target optical flow field, the motion blur kernel width of each pixel is calculated, and the motion blur kernel width is normalized to a preset value range to generate the motion sensing mask.

[0007] Preferably, the step of performing element-wise multiplication and weighting of the motion-aware mask with the multi-scale spatial feature map of the current frame image includes: downsampling the motion-aware mask to the same resolution size as each scale feature map in the multi-scale spatial feature map to obtain the scale mask corresponding to each scale. Local extreme points are extracted from the scale mask corresponding to each scale. An adaptive Gaussian weight kernel is set with the local extreme points as the center, and the scale mask is spatially Gaussian weighted diffusion is performed to generate an adaptive spatial mask. The adaptive spatial mask is multiplied channel-by-channel with the feature map of the corresponding scale in the multi-scale spatial feature map to obtain a weighted multi-scale feature map that suppresses motion blur background and enhances local abrupt features.

[0008] Preferably, the step of aligning the high-level semantic features of the preceding frame image with the inter-frame optical flow field includes: performing bilinear interpolation coarse alignment on the high-level semantic features of the preceding frame image based on the inter-frame optical flow field to obtain a coarse alignment feature map. The coarse alignment feature map and the high-level semantic features of the current frame image are input into a concatenated deformable convolutional network. The deformable convolutional network learns the spatial offset based on the difference between the coarse alignment feature map and the high-level semantic features of the current frame image. Based on the spatial offset, the convolution sampling position of the coarsely aligned feature map is corrected to obtain finely aligned high-level semantic features that eliminate sub-pixel alignment errors, which are then used as high-level semantic features of the preceding frame to participate in the channel-level stitching operation.

[0009] Preferably, the step of performing dimensionality reduction mapping on the spliced ​​features includes: dividing the spliced ​​feature map obtained after channel-level splicing operation into multiple parallel feature subgroups along the channel dimension; A cross-group cross-fusion operation is performed on the multiple parallel feature subgroups. The cross-group cross-fusion operation generates the channel attention weights of the current feature subgroup by concatenating and fully connecting the channel global pooling vectors of the current feature subgroup with the channel global pooling vectors of other feature subgroups. After weighting the current feature subgroup using the channel attention weights, all weighted feature subgroups are merged by channel convolution, and the number of channels in the merged feature map is compressed to be consistent with the low-level detail features of the current frame image through pointwise convolution to obtain the reconstructed feature map.

[0010] Preferably, the step of locating and classifying copper foil surface defects based on the reconstructed feature map includes: extracting the gradient magnitude and gradient direction of each pixel on the reconstructed feature map, and filtering candidate defect anchor points according to the gradient magnitude; Centered on the candidate defect anchor point, a direction-sensitive rotating anchor frame is generated based on the statistical distribution of the gradient direction. The reconstructed feature map and the rotating anchor frame are subjected to deformable region of interest pooling to extract local defect features that are aligned with the geometry of the rotating anchor frame; The local defect features are input into the classification branch and the regression branch. The classification branch outputs the category probability of the defect in the rotating anchor frame, and the regression branch outputs the position offset and size scaling factor of the rotating anchor frame.

[0011] Preferably, the step of introducing a structural consistency constraint term into the optical flow energy function includes: calculating the local structure tensor of the corresponding pixel neighborhood in the consecutive frame images, and obtaining the eigenvalues ​​and eigenvectors of the local structure tensor; The local anisotropy metric is calculated based on the eigenvalues, and the local anisotropy metric represents the directional intensity of the local texture. The weight coefficient of the structural consistency constraint term is dynamically adjusted based on the local anisotropy metric value. When the local anisotropy metric value increases, the weight coefficient is increased to enhance the structural constraint. When the local anisotropy metric decreases, the weighting coefficient is reduced to avoid oversmoothing, and the structural consistency constraint term with the weighting coefficient is added to the optical flow energy function.

[0012] Preferably, the step of setting an adaptive Gaussian weight kernel centered on the local extreme point includes: calculating the gradient variance in the neighborhood corresponding to the local extreme point, wherein the gradient variance reflects the degree of drastic change in local gray level; The gradient variance is mapped to a Gaussian kernel size parameter. When the gradient variance is large, a smaller Gaussian kernel size parameter is obtained to preserve the edge features of subtle defects. When the gradient variance is small, a larger Gaussian kernel size parameter is obtained to diffuse and suppress flat and blurred regions. A two-dimensional Gaussian kernel function is constructed based on the Gaussian kernel size parameters. The two-dimensional Gaussian kernel function is multiplied with the local extremum point, and convolution diffusion is performed on the scale mask to obtain an adaptive spatial mask with local differentiation enhancement.

[0013] Preferably, the step of correcting the convolution sampling position of the coarse alignment feature map based on the spatial offset includes: in the cascaded deformable convolutional network, the first deformable convolutional layer generates a base offset based on the absolute difference between the coarse alignment feature map and the high-level semantic features of the current frame image. The base offset and the coarse alignment feature map are input into the second deformable convolution layer. The second deformable convolution layer calculates the local feature correlation at the sampling position guided by the base offset and generates the residual offset based on the local feature correlation. The base offset and the residual offset are added element-wise to obtain the target space offset, and the sampling coordinates of the convolution kernel of the coarse alignment feature map are reset using the target space offset.

[0014] Preferably, the step of performing cross-group cross-fusion operation on the multiple parallel feature subgroups includes: expanding the spatial feature map of the current feature subgroup into a spatial feature sequence, and expanding the spatial feature maps of other feature subgroups into a context key sequence and a value sequence; Calculate the matrix multiplication between the spatial feature sequence and the transpose of the context key sequence to obtain the spatial correlation matrix; The spatial correlation matrix is ​​scaled and normalized, and the processed spatial correlation matrix is ​​multiplied by the value sequence to obtain cross-group spatial aggregation features; The cross-group spatial aggregation features are residually connected to the current feature subgroup, and the spatial location information of the cross-group spatial aggregation features is integrated through channel-wise convolution to complete the spatial perception cross-fusion between feature subgroups.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. A motion-aware mask is generated by calculating the inter-frame optical flow field between consecutive frames. The motion-aware mask is then multiplied and weighted element-wise with the multi-scale spatial feature map of the current frame image. This directly suppresses the background feature response of motion-blurred regions during feature extraction, preventing blurred pixels from being misidentified as normal textures. The high-level semantic features of the previous frame image after spatial alignment with the inter-frame optical flow field are then concatenated with the low-level detail features of the current frame image at the channel level and dimensionality-reduced. The edge gradients lost due to motion blur in the current frame image are reconstructed using information from the previous frame image that is not blurred or has different degrees of blurring. This breaks the isolation of single-frame spatial feature extraction, restores the discernibility of fine defect features, and solves the core problem of missed detection caused by motion blur of single-frame images under high-speed copper foil movement.

[0016] 2. During the recursive aggregation of temporal features, a cascaded deformable convolutional network is used to learn the spatial offset based on the difference between the coarsely aligned feature map and the high-level semantic features of the current frame image. This corrects the convolution sampling position, eliminates sub-pixel level spatial alignment errors, and avoids artifact interference introduced by spatial misalignment during temporal feature fusion. The channel-level stitched features are grouped and cross-fused, and dimensionality-reduced. Through cross-group channel attention weighting and pointwise convolution compression, redundant background channel information is filtered out, the core response of defect edge gradients is retained, the data dimension of the feature map is reduced, and the consumption of computational resources is reduced. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of the real-time detection and classification of copper foil surface defects based on machine vision according to the present invention; Figure 2 This is a flowchart of the inter-frame optical flow field calculation and motion-sensing mask generation process of the present invention. Figure 3 This is a flowchart of the multi-scale spatial feature weighting processing of the present invention; Figure 4 This is a flowchart of the temporal feature spatial alignment process of the present invention; Figure 5 This is a flowchart of the feature splicing and cross-group fusion dimensionality reduction process of the present invention; Figure 6 This is a flowchart illustrating the process of locating and classifying surface defects on copper foil according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please refer to Figure 1 This embodiment provides a machine vision-based method for real-time detection and classification of surface defects on copper foil. It extracts consecutive frame images of the acquired copper foil surface and calculates the inter-frame optical flow field between these consecutive frames. The copper foil surface images are acquired by a line scan camera at a fixed frame rate. During acquisition, the copper foil moves at a constant speed along the production line direction, and a fixed time interval exists between adjacent frames. The consecutive frame images include the current frame image. and preceding frame images ,in This represents the timestamp of image acquisition. The inter-frame optical flow field describes the displacement vector of each pixel in the image between two adjacent frames; each displacement vector contains a horizontal component. and vertical components ,in This represents the coordinates of a pixel in the image coordinate system.

[0020] Based on the amplitude and direction distribution of the inter-frame optical flow field, a motion-aware mask characterizing the degree of motion blur is generated. The formula for calculating the amplitude of the inter-frame optical flow field is:

[0021] in Represents pixels The optical flow amplitude at a given location. The optical flow amplitude is positively correlated with the degree of motion blur; the larger the optical flow amplitude, the greater the displacement of that pixel during the exposure time, and the more severe the motion blur. Motion-aware mask. The range of values ​​is The value 0 indicates that there is no motion blur in the region, and the value 1 indicates that the motion blur is most severe in the region.

[0022] A multi-scale spatial feature map of the current frame image is extracted. The motion-aware mask is then multiplied element-wise with the multi-scale spatial feature map of the current frame image and weighted to suppress background feature responses in motion-blurred regions of the current frame image. The multi-scale spatial feature map is extracted through different layers of a convolutional neural network, including low-level detail feature maps, mid-level semantic feature maps, and high-level semantic feature maps. The low-level detail feature map contains rich edge, texture, and gradient information, while the high-level semantic feature map contains abstract category and structural information. The extraction process of the multi-scale spatial feature map is implemented through a residual network, which contains multiple residual blocks. Each residual block consists of two convolutional layers and one skip connection.

[0023] A temporal feature recursive aggregation path is constructed in the feature pyramid network. High-level semantic features of the preceding frame image, after spatial alignment with the inter-frame optical flow field, are concatenated with the low-level detail features of the current frame image at the channel level. The concatenated features are then subjected to dimensionality reduction mapping to obtain a reconstructed feature map of the defect edge gradient. The feature pyramid network includes a bottom-up feature extraction path and a top-down feature fusion path, with the temporal feature recursive aggregation path and the top-down feature fusion path running in parallel. The high-level semantic features of the preceding frame image, after spatial alignment, are fused with the corresponding scale feature map of the current frame image. The spatial alignment operation is implemented based on the inter-frame optical flow field, mapping the feature map of the preceding frame image to the coordinate system of the current frame image.

[0024] The reconstructed feature map is used to locate and classify surface defects on copper foil. Defect localization is achieved by predicting the position and size of the defect bounding box, while defect classification is achieved by predicting the probability that a defect belongs to different categories. Copper foil surface defects include pinholes, scratches, wrinkles, oxide spots, indentations, etc. The localization and classification process is implemented through a detection head, which includes a classification branch and a regression branch. The classification branch outputs the category probability of each anchor frame, and the regression branch outputs the position offset and size scaling factor of each anchor frame. The description of copper foil surface defect types and features is shown in Table 1.

[0025] Table 1. Description of surface defects and characteristics of copper foil ; In this embodiment, the calculation of the inter-frame optical flow field is based on the assumption of constant brightness, that is, the brightness of the same object remains unchanged in different frame images. The mathematical expression for the constant brightness constraint is:

[0026] Expanding the right side of the above equation using a first-order Taylor series, we get:

[0027] in and These represent the images in and Spatial gradient of direction, This represents the gradient of the image over time. After simplification, we obtain the fundamental equation of optical flow:

[0028] The equation contains two unknowns. and It cannot be solved directly and requires the introduction of additional constraints.

[0029] The generation process of the motion-aware mask includes optical flow amplitude normalization and spatial smoothing. The formula for optical flow amplitude normalization is:

[0030] in and These represent the minimum and maximum optical flow amplitudes in the current frame image, respectively. The normalized optical flow amplitude is mapped to... The initial motion-aware mask is obtained by dividing the image into intervals. A Gaussian filter is then applied to the initial motion-aware mask to remove isolated noise points, resulting in the final motion-aware mask. The kernel size of the Gaussian filter is dynamically adjusted based on the image resolution and the degree of motion blur.

[0031] In the process of extracting multi-scale spatial feature maps, the input of the residual network is the current frame image. The output is feature maps at different scales. ,in It has the highest resolution and contains the richest underlying details. The lowest resolution feature map contains the most abstract, high-level semantic information. The resolution is the same as the input image resolution. The motion-aware mask is downsampled to the same resolution as the feature maps at each scale to obtain the scale mask. The scale mask is then element-wise multiplied with the feature map at the corresponding scale to obtain the weighted feature map. ,in This indicates an element-wise multiplication operation.

[0032] In the process of constructing the recursive aggregation path of temporal features, the high-level semantic features of the preceding frame images After spatial alignment, it is compared with the low-level detail features of the current frame image. Channel-level stitching is performed. Spatial alignment maps each pixel in the previous frame's feature map to the current frame's coordinate system based on the optical flow displacement vector. For pixels with non-integer coordinates after mapping, bilinear interpolation is used to calculate their pixel values. Channel-level stitching merges the two feature maps along the channel dimension to obtain a stitched feature map. Its channel number is the sum of the channel numbers of the two input feature maps.

[0033] Dimensionality reduction mapping is performed on the concatenated feature map to obtain the reconstructed feature map. The dimensionality reduction mapping process is achieved through pointwise convolution, with the kernel size of the pointwise convolution being... The number of output channels is the same as the number of channels in the low-level detail feature map of the current frame image. The mathematical expression for pointwise convolution is:

[0034] in This indicates the number of channels in the spliced ​​feature map. This represents the weight parameters of the pointwise convolution. The dimensionality reduction mapping process reduces the number of channels in the feature map, thereby reducing the complexity of subsequent calculations.

[0035] During defect localization and classification, multiple anchor boxes are generated on the reconstructed feature map, each corresponding to a possible defect location. The size and proportion of the anchor boxes are pre-defined based on the statistical characteristics of defects on the copper foil surface. For each anchor box, the classification branch outputs a... A probability vector of dimension, where The first dimension represents the number of defect categories, and the second dimension represents the background category. The regression branch outputs a 4-dimensional vector representing the anchor box's position offset and size scaling factor. Overlapping anchor boxes are removed using a non-maximum suppression algorithm to obtain the final defect detection result.

[0036] In this embodiment, a motion-aware mask is generated by calculating the inter-frame optical flow field between consecutive frames. This motion-aware mask is then multiplied and weighted element-wise with the multi-scale spatial feature map of the current frame image, directly suppressing background feature responses in motion-blurred regions during feature extraction. The high-level semantic features of the preceding frame image, after spatial alignment with the inter-frame optical flow field, are then concatenated with the low-level detail features of the current frame image at the channel level and subjected to dimensionality reduction mapping. Information from the preceding frame image, either unblurred or with varying degrees of blurring, is used to reconstruct the edge gradients lost due to motion blur in the current frame image. Based on this reconstructed feature map, the location and classification of surface defects on the copper foil are performed, solving the problem of missed detection due to the loss of fine defect features caused by motion blur in a single frame image during high-speed copper foil movement.

[0037] In a preferred embodiment, reference Figure 2 An optical flow energy function is constructed that incorporates brightness constancy and gradient constancy. A structural consistency constraint term is introduced into this function, calculated based on the image's local structure tensor. The optical flow energy function consists of a data term and a smoothing term. The data term includes brightness constraining and gradient constraining, while the smoothing term includes structural consistency constraints. The brightness constraining measures the brightness difference between corresponding pixels in two adjacent frames, the gradient constraining measures the gradient difference between corresponding pixels in two adjacent frames, and the structural consistency constraint measures the structural similarity of the neighborhood of corresponding pixels in two adjacent frames.

[0038] The initial optical flow field of the consecutive frame images is obtained by iteratively solving the optical flow energy function. The minimization problem of the optical flow energy function is transformed into the Euler-Lagrange equation using a variational method, and then numerically solved using the finite difference method. The iterative solution process employs a pyramid layering strategy, starting with solving the optical flow field from a low-resolution image, and then using the solution as the initial value for a high-resolution image, iterating upwards layer by layer until the original image resolution is reached. The pyramid layering strategy improves the robustness and accuracy of optical flow calculation and can handle large displacement motion scenarios.

[0039] The initial optical flow field is processed by median filtering and bilateral filtering to remove isolated noise points and maintain smooth optical flow edges, thus obtaining the target optical flow field. Median filtering is a nonlinear filtering method that can effectively remove impulse noise while preserving image edges. The kernel size of the median filter is... or The standard deviation of the bilateral filter is dynamically adjusted based on the noise level. It considers both the spatial distance and grayscale differences between pixels, preserving image edge details while removing noise. The spatial and grayscale standard deviations of the bilateral filter are dynamically adjusted according to the image content.

[0040] Based on the vector magnitude of each pixel in the target optical flow field, the motion blur kernel width of each pixel is calculated, and the motion blur kernel width is normalized to a preset numerical range to generate the motion-sensing mask. The motion blur kernel width has a linear relationship with the optical flow magnitude; the larger the optical flow magnitude, the larger the motion blur kernel width. The formula for calculating the motion blur kernel width is:

[0041] in and The linear coefficients are pre-calibrated based on the camera exposure time and the copper foil movement speed. The motion blur kernel width is normalized to... The motion-sensing mask is obtained by dividing the region. The normalization process uses a linear mapping method, mapping the minimum value of the motion blur kernel width to 0 and the maximum value to 1.

[0042] Calculate the local structure tensor of the neighborhood of corresponding pixels in the consecutive frames of the image, and obtain the eigenvalues ​​and eigenvectors of the local structure tensor. The local structure tensor describes the texture structure and orientation information of a local region of the image, and is defined as follows:

[0043] in Represents a local neighborhood centered at a pixel, typically a window of type 0 or 1. The local structure tensor is a... A symmetric positive semi-definite matrix, whose eigenvalues and ( This describes the degree of anisotropy of local textures.

[0044] The local anisotropy metric is calculated based on the eigenvalues, and the local anisotropy metric represents the directional intensity of the local texture. The formula for calculating the local anisotropy metric is as follows:

[0045] in It is a very small positive number to avoid the denominator being zero. Local anisotropy measure. The range of values ​​is ,when When, it indicates that the local area has an isotropic texture with no obvious directionality; when When the local area is an ideal linear texture, it has a strong directionality.

[0046] The weighting coefficients of the structural consistency constraint term are dynamically adjusted based on the local anisotropy metric. When the local anisotropy metric increases, the weighting coefficients are increased to strengthen the structural constraint; when the local anisotropy metric decreases, the weighting coefficients are decreased to avoid over-smoothing. The structural consistency constraint term with the weighting coefficients is then added to the optical flow energy function. (Weighting coefficients of the structural consistency constraint term) The calculation formula is:

[0047] in These are the basic weighting coefficients. Dynamically adjusted weighting coefficients can enhance structural constraints in textured regions, improving the accuracy of optical flow calculations; and weaken structural constraints in textured regions, avoiding excessive smoothing that leads to blurred optical flow edges.

[0048] refer to Figure 3 The motion-aware mask is downsampled to the same resolution as the feature maps at each scale in the multi-scale spatial feature map, obtaining the scale mask corresponding to each scale. The downsampling process uses bilinear interpolation to reduce the resolution of the motion-aware mask to the same level as the corresponding scale feature map. For the ... The feature map is scaled to a resolution equal to the input image resolution. Therefore, scale mask The resolution is also the resolution of the input image. The downsampling process preserves the spatial distribution characteristics of the motion-aware mask, ensuring that feature maps at different scales receive appropriate weighting.

[0049] Local extrema are extracted from the scale masks corresponding to each scale. An adaptive Gaussian weight kernel is then set around these local extrema, and spatial Gaussian weighted diffusion is applied to the scale masks to generate an adaptive spatial mask. Local extrema include local maxima and local minima. Local maxima correspond to the regions with the most severe motion blur, while local minima correspond to the regions with the least motion blur. Local extrema are extracted by comparing the grayscale values ​​of each pixel with its eight neighboring pixels. If the grayscale value of a pixel is greater than (or less than) the grayscale values ​​of all its neighboring pixels, then that pixel is a local maximum (or local minimum).

[0050] Calculate the gradient variance within the neighborhood corresponding to the local extremum point. The gradient variance reflects the drasticness of local gray-level changes. The formula for calculating the gradient variance is:

[0051] in This represents a neighborhood window of a local extremum point. This indicates the number of pixels within the window. Represents pixels gradient magnitude at that point This represents the average gradient magnitude within the window. A larger gradient variance indicates more dramatic grayscale changes in a local area, containing more detailed information; a smaller gradient variance indicates smoother grayscale changes in a local area, containing more flat background information.

[0052] The gradient variance is mapped to a Gaussian kernel size parameter. When the gradient variance is large, a smaller Gaussian kernel size parameter is obtained to preserve subtle defect edge features. When the gradient variance is small, a larger Gaussian kernel size parameter is obtained to diffuse and suppress flat, blurred regions. The Gaussian kernel size parameter is inversely proportional to the gradient variance, and the mapping formula is:

[0053] in The maximum Gaussian kernel size parameter, is the attenuation coefficient. This mapping formula ensures that a small Gaussian kernel is used in regions with large gradient variance to preserve subtle edge features; and a large Gaussian kernel is used in regions with small gradient variance to diffuse and suppress the feature response of flat background regions.

[0054] A two-dimensional Gaussian kernel function is constructed based on the Gaussian kernel size parameters. This two-dimensional Gaussian kernel function is multiplied by the local extrema, and then convolutional diffusion is performed on the scale mask to obtain an adaptive spatial mask with localized enhancement. The definition of the two-dimensional Gaussian kernel function is:

[0055] in This represents the coordinates relative to the center of the Gaussian kernel. Each local extremum is multiplied by its corresponding 2D Gaussian kernel function, and then convolution is performed on the scale mask to diffuse the influence of the local extremum to the surrounding area. The diffusion results of all local extremums are then superimposed to obtain the adaptive spatial mask. The adaptive spatial mask can dynamically adjust the weighting intensity according to the local image content, maintaining a high feature response in defect edge regions and suppressing background feature response in flat, blurred regions.

[0056] The adaptive spatial mask is multiplied channel-by-channel with the corresponding scale feature map in the multi-scale spatial feature map to obtain a weighted multi-scale feature map that suppresses motion blur background and enhances local abrupt features. The channel-by-channel multiplication operation applies the adaptive spatial mask to each channel of the feature map, ensuring that all channels receive appropriate weighting. In the weighted feature map, the background feature response of motion-blurred regions is suppressed, while local abrupt features such as defect edges are enhanced, improving the accuracy of subsequent defect detection. Table 2 shows a comparison of the mask weighting effects under different degrees of motion blur.

[0057] Table 2 Comparison of mask weighting effects under different degrees of motion blur. ; In this embodiment, an optical flow energy function incorporating constraints of brightness constancy, gradient constancy, and structural consistency is constructed, and an accurate inter-frame optical flow field is obtained through iterative solution. Median filtering and bilateral filtering are applied to the initial optical flow field to remove noise and maintain smooth optical flow edges. The motion blur kernel width is calculated based on the optical flow amplitude to generate a motion-aware mask. A structural consistency constraint term based on the local structure tensor is introduced into the optical flow energy function, and the constraint weights are dynamically adjusted according to the local anisotropy metric, improving the accuracy and robustness of optical flow calculation. The motion-aware mask is downsampled to different scales, local extrema are extracted, and an adaptive Gaussian weight kernel is set to generate an adaptive spatial mask. The adaptive spatial mask is multiplied channel-by-channel with the multi-scale spatial feature map to suppress motion-blurred background feature responses and enhance local abrupt features. This embodiment refines the technical details of optical flow calculation and motion-aware mask generation, further improving the feature processing effect in motion-blurred regions.

[0058] In a preferred embodiment, reference Figure 4 Based on the inter-frame optical flow field, bilinear interpolation is used to coarsely align the high-level semantic features of the preceding frame image to obtain a coarsely aligned feature map. During the bilinear interpolation coarse alignment process, for each pixel in the current frame image... The corresponding position of the optical flow in the preceding frame is calculated based on the inter-frame optical flow field. If the corresponding position is an integer coordinate, the pixel value at that position is directly taken from the previous frame image; if the corresponding position is a non-integer coordinate, the pixel value at that position is calculated using bilinear interpolation. The formula for bilinear interpolation is:

[0059] in The integer part at the corresponding position. and For the decimal part at the corresponding position, This is a high-level semantic feature map of the preceding frame image. Bilinear interpolation coarse alignment can eliminate most pixel-level spatial alignment errors, but it cannot eliminate sub-pixel-level alignment errors.

[0060] The coarse-aligned feature map and the high-level semantic features of the current frame image are input into a cascaded deformable convolutional network. The deformable convolutional network learns a spatial offset based on the difference between the coarse-aligned feature map and the high-level semantic features of the current frame image. The difference between a deformable convolutional network and a traditional convolutional network is that the deformable convolutional network can learn the offset of the convolutional kernel sampling position, allowing the convolutional kernel to adaptively adjust its sampling position according to the content of the input features. The cascaded deformable convolutional network contains two deformable convolutional layers: the first deformable convolutional layer generates the base offset, and the second deformable convolutional layer generates the residual offset.

[0061] Based on the spatial offset, the convolution sampling positions of the coarsely aligned feature map are corrected to obtain finely aligned high-level semantic features that eliminate sub-pixel alignment errors. These features are then used as the high-level semantic features of the preceding frame to participate in the channel-level stitching operation. The spatial offset includes the horizontal offset. And the vertical offset, the corrected convolution sampling position is By correcting the convolution sampling position, deformable convolutional networks can align the feature maps of the previous frame and the current frame at the sub-pixel level, eliminating the sub-pixel error left by coarse alignment of bilinear interpolation.

[0062] In the cascaded deformable convolutional network, the first deformable convolutional layer generates a base offset based on the absolute difference between the coarse alignment feature map and the high-level semantic features of the current frame image. First, the absolute difference between the coarse alignment feature map and the high-level semantic feature map of the current frame is calculated. ,in This is the high-level semantic feature map of the current frame image. The absolute difference map... Input the offset prediction branch of the first deformable convolution layer to generate the base offset. and The offset prediction branch consists of two convolutional layers. The first convolutional layer has 64 output channels, and the second convolutional layer has [number missing] output channels. ,in This represents the number of sampling points for the convolution kernel.

[0063] The base offset and the coarse-aligned feature map are input into a second deformable convolutional layer. This second layer calculates local feature correlations at the sampling positions guided by the base offset and generates residual offsets based on these correlations. The second layer first adjusts the sampling positions of the convolution kernel according to the base offset, and then extracts features at the adjusted sampling positions. The correlation between the extracted features and the corresponding features in the current frame's high-level semantic feature map is calculated, and residual offsets are generated based on these correlations. and The residual offset is used to further correct the base offset and improve alignment accuracy.

[0064] The base offset and the residual offset are added element-wise to obtain the target spatial offset. The sampling coordinates of the convolution kernel of the coarse-aligned feature map are then reset using the target spatial offset. The formula for calculating the target spatial offset is:

[0065] The coordinates of each sampling point of the convolution kernel are reset according to the target space offset, and then a convolution operation is performed on the coarse-aligned feature map to obtain the fine-aligned high-level semantic feature map. The finely aligned high-level semantic feature map eliminates sub-pixel level spatial alignment errors and corresponds precisely in spatial location to the high-level semantic feature map of the current frame image.

[0066] refer to Figure 5 The stitched feature map obtained after channel-level stitching is divided into multiple parallel feature subgroups along the channel dimension. The number of channels is Where is the number of channels in the finely aligned high-level semantic feature map. This represents the number of channels in the low-level detail feature map of the current frame image. The stitched feature map is divided along the channel dimension. There are feature subgroups, and the number of channels in each feature subgroup is . The partitioning of feature subgroups is continuous, that is, the first... Each feature subgroup contains channel indexes from arrive The passage.

[0067] A cross-group cross-fusion operation is performed on the multiple parallel feature subgroups. This operation generates the channel attention weights of the current feature subgroup by concatenating and fully connecting the channel global pooling vectors of the current feature subgroup with those of other feature subgroups. The channel global pooling operation compresses the two-dimensional feature map of each channel into a scalar, calculated using the following formula:

[0068] in For the first Feature subgroups, and These represent the height and width of the feature map, respectively. For channel indexing. Globally pool the channel vectors of the current feature subgroup. Channel global pooling vectors with all other feature subgroups By concatenating, we obtain the concatenated vector. Concatenated vectors Input to the fully connected layer to generate channel attention weights for the current feature subgroup. .

[0069] After weighting the current feature subgroup using the channel attention weights, all weighted feature subgroups are merged by channel convolution. Then, the number of channels in the merged feature map is compressed to match the low-level detail features of the current frame image through pointwise convolution, thus obtaining the reconstructed feature map. The calculation formula for the channel attention weighting operation is:

[0070] All weighted feature subgroups Merging along the channel dimension yields the merged feature map. The merged feature map is input into the pointwise convolutional layer, and the number of output channels of the pointwise convolutional layer is... The number of channels is consistent with the number of channels in the low-level detail feature map of the current frame image. The output of the pointwise convolutional layer is the reconstructed feature map. .

[0071] Expand the spatial feature map of the current feature subgroup into a spatial feature sequence, and expand the spatial feature maps of other feature subgroups into context key sequences and value sequences. The size of the spatial feature map is... Expand it to a length of Spatial feature sequence The spatial feature maps of other feature subgroups are also expanded into a length of... context key sequence and context value sequence .

[0072] Calculate the matrix multiplication between the spatial feature sequence and the transpose of the context key sequence to obtain the spatial correlation matrix. The formula for calculating the spatial correlation matrix is:

[0073] in , Represents the spatial feature sequence of the first The position and context key sequence of the th position The spatial correlation degree matrix is ​​calculated for each location. The matrix is ​​then scaled using a scaling factor of 0.5 to avoid the vanishing gradient problem caused by excessively large correlation values. The scaled spatial correlation degree matrix is ​​as follows:

[0074] The spatial correlation matrix is ​​scaled and normalized, and then multiplied by the value sequence to obtain cross-group spatial aggregation features. The scaled spatial correlation matrix... Each row is subjected to softmax normalization to obtain the normalized spatial correlation matrix. The normalized spatial correlation matrix With context value sequence Multiplication yields cross-group spatial aggregation features:

[0075] Cross-group spatial aggregation features By incorporating spatial context information from other feature subgroups, it is able to capture spatial dependencies between different feature subgroups.

[0076] The cross-group spatial aggregation features are residually connected to the current feature subgroup, and the spatial location information of the cross-group spatial aggregation features is integrated through channel-wise convolution to complete the spatial-aware cross-fusion between feature subgroups. The residual connection operation integrates the cross-group spatial aggregation features... With the current feature subgroup Adding them together yields the residual characteristics. Residual characteristics Input is a channel-wise convolutional layer, and the kernel size of the channel-wise convolutional layer is... This is used to integrate spatial location information. The output of the channel-wise convolutional layer is the feature subgroup after spatially perceptual cross-fusion. .

[0077] refer to Figure 6 The gradient magnitude and gradient direction of each pixel are extracted from the reconstructed feature map, and candidate defect anchor points are selected based on the gradient magnitude. The formulas for calculating the gradient magnitude and gradient direction are as follows:

[0078] in and These represent the reconstructed feature maps in... and Gradient of direction. Represents pixels The gradient direction angle at that point. Set the gradient magnitude threshold. The gradient magnitude is greater than The pixels are marked as candidate defect anchor points. Gradient magnitude threshold. The gradient magnitude is dynamically adjusted based on the statistical characteristics of the reconstructed feature map, typically taking 2-3 times the average gradient magnitude.

[0079] Centered on the candidate defect anchor points, a direction-sensitive rotating anchor frame is generated based on the statistical distribution of the gradient directions. For each candidate defect anchor point, the gradient directions of all pixels in its neighborhood are statistically analyzed to generate a gradient direction histogram. The direction corresponding to the peak value in the gradient direction histogram is taken as the main direction of the rotating anchor frame. The size of the rotating anchor frame is pre-set according to the statistical characteristics of the copper foil surface defects, including various different sizes and proportions. The angle range of the rotating anchor frame is... This is to cover all possible defect directions.

[0080] The reconstructed feature map and the rotated anchor frame are subjected to deformable region-of-interest (ROI) pooling to extract local defect features aligned with the geometry of the rotated anchor frame. The difference between deformable ROI pooling and traditional ROI pooling is that deformable ROI pooling can learn the offset of the pooling sampling position, allowing the sampling position to adaptively adjust according to the geometry of the rotated anchor frame. Deformable ROI pooling first maps the rotated anchor frame to the coordinate system of the reconstructed feature map, then generates a regular sampling grid within the mapped region. The offset of each sampling point is predicted, the sampling position is adjusted based on the offset, and finally, pooling is performed at the adjusted sampling position to obtain local defect features.

[0081] The local defect features are input into the classification branch and the regression branch. The classification branch outputs the category probability of the defect in the rotating anchor frame, and the regression branch outputs the position offset and size scaling factor of the rotating anchor frame. The classification branch consists of two fully connected layers. The first fully connected layer has 256 output channels, and the second fully connected layer has [number missing]. ,in This represents the number of defect categories. The regression branch also consists of two fully connected layers. The first fully connected layer has 256 output channels, and the second fully connected layer has 5 output channels, corresponding to the center coordinate offset of the rotating anchor frame, the width scaling factor, the height scaling factor, and the angle offset, respectively. Table 3 shows a comparison of the defect detection performance of different feature fusion methods.

[0082] Table 3 Comparison of defect detection performance of different feature fusion methods ; In this embodiment, bilinear interpolation is used to coarsely align the high-level semantic features of the preceding frame image based on the inter-frame optical flow field. Then, a cascaded deformable convolutional network is used to learn the spatial offset, correct the convolution sampling position, and obtain finely aligned high-level semantic features. The cascaded deformable convolutional network contains two deformable convolutional layers. The first layer generates the base offset, and the second layer generates the residual offset. The two are added together to obtain the target spatial offset. The finely aligned high-level semantic features are concatenated with the low-level detail features of the current frame image at the channel level, and then divided into multiple feature subgroups along the channel dimension. Cross-group cross-fusion operations are performed on the feature subgroups, including channel attention weighting and spatially aware cross-fusion. The fused feature subgroups are merged and dimensionality reduced by pointwise convolution to obtain a reconstructed feature map. Gradient magnitude and direction are extracted from the reconstructed feature map, candidate defect anchor points are screened, and rotational anchor boxes with direction sensitivity are generated. Local defect features are extracted by deformable region of interest pooling, input into the classification branch and regression branch, and the category probability and location parameters of the defect are output. This embodiment refines the technical details of temporal feature aggregation and defect detection classification, further improving the accuracy of temporal feature fusion and the performance of defect detection.

Claims

1. A method for real-time detection and classification of surface defects on copper foil based on machine vision, characterized in that, include: Extract consecutive frame images of the copper foil surface and calculate the inter-frame optical flow field between the consecutive frame images; Based on the amplitude and direction distribution of the inter-frame optical flow field, a motion-sensing mask characterizing the degree of motion blur is generated; Extract the multi-scale spatial feature map of the current frame image, and perform element-wise multiplication and weighting of the motion-aware mask and the multi-scale spatial feature map of the current frame image to suppress the background feature response of the motion-blurred region in the current frame image; In the feature pyramid network, a recursive aggregation path for temporal features is constructed. The high-level semantic features of the previous frame image after being aligned with the inter-frame optical flow field space are combined with the low-level detail features of the current frame image at the channel level. The combined features are then subjected to dimensionality reduction mapping to obtain a reconstructed feature map of the defect edge gradient. Based on the reconstructed feature map, the surface defects of the copper foil are located and classified.

2. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 1, characterized in that, The step of calculating the inter-frame optical flow field between the consecutive frame images includes: constructing an optical flow energy function that includes brightness constancy and gradient constancy, introducing a structural consistency constraint term into the optical flow energy function, and the structural consistency constraint term being calculated based on the local structure tensor of the image; The initial optical flow field of the consecutive frame images is obtained by iteratively solving the optical flow energy function. The initial optical flow field is subjected to median filtering and bilateral filtering to remove isolated noise points in the initial optical flow field and keep the optical flow edges smooth, thereby obtaining the target optical flow field. Based on the vector magnitude of each pixel in the target optical flow field, the motion blur kernel width of each pixel is calculated, and the motion blur kernel width is normalized to a preset value range to generate the motion sensing mask.

3. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 2, characterized in that, The step of performing element-wise multiplication and weighting of the motion-aware mask with the multi-scale spatial feature map of the current frame image includes: downsampling the motion-aware mask to the same resolution size as each scale feature map in the multi-scale spatial feature map to obtain the scale mask corresponding to each scale. Local extrema are extracted from the scale mask corresponding to each scale. An adaptive Gaussian weight kernel is set with the local extrema as the center. Spatial Gaussian weighted diffusion is performed on the scale mask to generate an adaptive spatial mask. The adaptive spatial mask is multiplied channel-by-channel with the feature map of the corresponding scale in the multi-scale spatial feature map to obtain a weighted multi-scale feature map that suppresses motion blur background and enhances local abrupt features.

4. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 1, characterized in that, The step of aligning the high-level semantic features of the preceding frame image with the inter-frame optical flow field includes: performing bilinear interpolation coarse alignment on the high-level semantic features of the preceding frame image based on the inter-frame optical flow field to obtain a coarse alignment feature map. The coarse alignment feature map and the high-level semantic features of the current frame image are input into a concatenated deformable convolutional network. The deformable convolutional network learns the spatial offset based on the difference between the coarse alignment feature map and the high-level semantic features of the current frame image. Based on the spatial offset, the convolution sampling position of the coarsely aligned feature map is corrected to obtain finely aligned high-level semantic features that eliminate sub-pixel alignment errors, which are then used as high-level semantic features of the preceding frame to participate in the channel-level stitching operation.

5. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 1, characterized in that, The steps for dimensionality reduction mapping of the spliced ​​features include: dividing the spliced ​​feature map obtained after channel-level splicing into multiple parallel feature subgroups along the channel dimension; A cross-group cross-fusion operation is performed on the multiple parallel feature subgroups. The cross-group cross-fusion operation generates the channel attention weights of the current feature subgroup by concatenating and fully connecting the channel global pooling vectors of the current feature subgroup with the channel global pooling vectors of other feature subgroups. After weighting the current feature subgroup using the channel attention weights, all weighted feature subgroups are merged by channel convolution, and the number of channels in the merged feature map is compressed to be consistent with the low-level detail features of the current frame image through pointwise convolution to obtain the reconstructed feature map.

6. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 1, characterized in that, The steps of locating and classifying copper foil surface defects based on the reconstructed feature map include: extracting the gradient magnitude and gradient direction of each pixel on the reconstructed feature map, and filtering candidate defect anchor points according to the gradient magnitude; Centered on the candidate defect anchor point, a direction-sensitive rotating anchor frame is generated based on the statistical distribution of the gradient direction. The reconstructed feature map and the rotating anchor frame are subjected to deformable region of interest pooling to extract local defect features that are aligned with the geometry of the rotating anchor frame; The local defect features are input into the classification branch and the regression branch. The classification branch outputs the category probability of the defect in the rotating anchor frame, and the regression branch outputs the position offset and size scaling factor of the rotating anchor frame.

7. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 2, characterized in that, The step of introducing a structural consistency constraint term into the optical flow energy function includes: calculating the local structure tensor of the corresponding pixel neighborhood in the consecutive frame images, and obtaining the eigenvalues ​​and eigenvectors of the local structure tensor; The local anisotropy metric is calculated based on the eigenvalues, and the local anisotropy metric represents the directional intensity of the local texture. The weight coefficient of the structural consistency constraint term is dynamically adjusted based on the local anisotropy metric value. When the local anisotropy metric value increases, the weight coefficient is increased to enhance the structural constraint. When the local anisotropy metric decreases, the weighting coefficient is reduced to avoid oversmoothing, and the structural consistency constraint term with the weighting coefficient is added to the optical flow energy function.

8. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 3, characterized in that, The step of setting an adaptive Gaussian weight kernel centered on the local extreme point includes: calculating the gradient variance in the neighborhood corresponding to the local extreme point, wherein the gradient variance reflects the degree of drastic change in local gray level; The gradient variance is mapped to a Gaussian kernel size parameter. When the gradient variance is large, a small Gaussian kernel size parameter is obtained to preserve the edge features of subtle defects. When the gradient variance is small, a large Gaussian kernel size parameter is obtained to diffuse and suppress flat and blurred regions. A two-dimensional Gaussian kernel function is constructed based on the Gaussian kernel size parameters. The two-dimensional Gaussian kernel function is multiplied with the local extremum point, and convolution diffusion is performed on the scale mask to obtain an adaptive spatial mask with local differentiation enhancement.

9. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 4, characterized in that, The step of correcting the convolution sampling position of the coarse alignment feature map based on the spatial offset includes: in the cascaded deformable convolutional network, the first deformable convolutional layer generates a base offset based on the absolute difference between the coarse alignment feature map and the high-level semantic features of the current frame image. The base offset and the coarse alignment feature map are input into the second deformable convolution layer. The second deformable convolution layer calculates the local feature correlation at the sampling position guided by the base offset and generates the residual offset based on the local feature correlation. The base offset and the residual offset are added element-wise to obtain the target space offset, and the sampling coordinates of the convolution kernel of the coarse alignment feature map are reset using the target space offset.

10. The method for real-time detection and classification of copper foil surface defects based on machine vision according to claim 5, characterized in that, The step of performing cross-group cross-fusion operation on the multiple parallel feature subgroups includes: expanding the spatial feature map of the current feature subgroup into a spatial feature sequence, and expanding the spatial feature maps of other feature subgroups into a context key sequence and a value sequence; Calculate the matrix multiplication between the spatial feature sequence and the transpose of the context key sequence to obtain the spatial correlation matrix; The spatial correlation matrix is ​​scaled and normalized, and the processed spatial correlation matrix is ​​multiplied by the value sequence to obtain cross-group spatial aggregation features; The cross-group spatial aggregation features are residually connected to the current feature subgroup, and the spatial location information of the cross-group spatial aggregation features is integrated through channel-wise convolution to complete the spatial perception cross-fusion between feature subgroups.

Images