A kind of plate surface defect online detection method and system based on multi-granularity reconstruction and feature memory matching

Abstract

The application discloses a kind of based on multi-granularity reconstruction and feature memory matching plate surface defect online detection method and system.The method collects normal sample image, trains three auto-encoders of spatial structure, scale semantics and color distribution to form reconstruction branch, and constructs global and local feature memory based on pre-training network.Online detection, pre-processing and block are carried out to plate image, each image block is sent into reconstruction branch and memory branch in parallel, respectively to obtain reconstruction error and feature distance Image-level score and pixel-level response map are obtained.The two types of results are normalized after robust, and the integrated abnormal score and abnormal map are obtained by weighted fusion.When the score exceeds threshold value, it is determined that there is a defect, and the pixel-level segmentation is combined to mark the defect area, and finally the whole plate abnormal heat map is spliced and the alarm information is output.The application does not need a large number of defect labeling samples, can detect unknown defects to realize pixel-level fine positioning, and is suitable for production line online quality detection.

Description

Technical Field

[0001] This invention relates to the field of sheet metal surface quality inspection technology, specifically to an online detection method and system for sheet metal surface defects based on multi-granularity reconstruction and feature memory matching. Background Technology

[0002] During the raw material handling, forming, processing, surface treatment, and transportation of sheet materials, various surface defects such as cracks, holes, scratches, contamination, water stains, color differences, breakage, and edge defects are easily generated. These defects not only affect the product's appearance quality and grading results but also impact the utilization rate of subsequent processing. Therefore, it is necessary to conduct stable and continuous quality inspections on the sheet materials on the production line.

[0003] Currently, the detection of surface defects in the sheet metal industry mainly relies on two methods: manual visual inspection and machine vision inspection. Manual visual inspection depends on the experience of quality inspectors and is easily affected by factors such as fatigue, subjective judgment, environmental noise, and production cycle time. This results in low inspection efficiency, poor consistency, and a high rate of missed and false detections, making it difficult to meet the requirements of continuous, automated, and mass production in modern sheet metal production lines. Current surface defect detection methods mainly include those based on optical imaging and manual rules, as well as those based on supervised deep learning.

[0004] The most commonly used method is the detection method based on optical imaging systems, CCD / line scan cameras, and manual rule-based judgment, such as the scheme disclosed in patent CN107589124A. When the board material passes through the detection area, the position sensor triggers image acquisition. Two sets of CCD industrial line scan cameras, one on the front and one on the back, are used in conjunction with an LED light source to perform online scanning imaging of the board surface. Then, the computer completes the defect location, extraction, and classification, and compares it with a standard defect database to finally achieve defect alarm and product classification. However, this type of method is relatively sensitive to lighting conditions and has limited generalization ability to changes in color, texture, and material.

[0005] Another approach employs supervised deep learning methods. Existing technologies include patents that disclose surface defect detection methods for sheet metal based on improved YOLOv5l networks, such as CN114359235A. These methods pre-train on publicly available defect datasets and combine them with real-world sheet metal data for manual annotation and pseudo-label fine-tuning, enabling the identification and detection of various sheet metal surface defects and improving detection efficiency and automation to some extent. While these methods typically rely on a large number of labeled defect samples and can identify known defect categories, their adaptability to unknown defects, composite defects, and defects with ambiguous boundaries remains limited.

[0006] Therefore, the main problems of existing technologies include: strong dependence on lighting environment and artificial rules, and limited generalization ability to natural texture, color and material variations of board materials; strong dependence on large-scale high-quality defect annotation samples for supervised detection methods, resulting in high data acquisition and annotation costs; insufficient adaptability to unknown defects, composite defects and defects with blurred boundaries; and difficulty in achieving pixel-level fine positioning based on detection boxes, thus making it difficult to simultaneously meet the requirements of high robustness detection and fine positioning in complex scenarios on board material surfaces. Summary of the Invention

[0007] The purpose of this invention is to provide an online detection method and system for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching, so as to solve the technical problems of existing sheet metal surface defect detection methods, which are highly dependent on lighting environment and artificial rules, have limited generalization ability, rely on a large number of defect annotation samples, and are difficult to simultaneously handle the detection of unknown defects and pixel-level fine positioning.

[0008] To achieve the above objectives, the technical solution provided by this invention is: an online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching, comprising the following steps: S1: Collect normal sample images of the board surface, train a multi-granularity autoencoder using the normal sample images and construct a feature memory library, and calibrate the image-level judgment threshold, pixel-level segmentation threshold, image-level fusion weight, and pixel-level fusion weight; the multi-granularity autoencoder includes three autoencoders: spatial structure, scale semantics, and color distribution; the feature memory library extracts features of normal samples based on a pre-trained network with frozen parameters, and constructs a global feature memory library for image-level scoring and a local feature memory library for pixel-level localization respectively; S2: Collect surface images of the board to be inspected on the production line, perform preprocessing, and then divide the board into grid blocks according to preset dimensions to obtain multiple image blocks; S3: Each image block is fed in parallel into the multi-granularity autoencoder branch and the feature memory branch; in the multi-granularity autoencoder branch, the image block is input into three autoencoders respectively to obtain three reconstructed images and reconstruction errors. After dynamic fusion, the image-level score of reconstruction error and the pixel-level response map of reconstruction error are output; in the feature memory branch, the same image block is input into the pre-trained network to extract features. The feature deviation distance from the global feature memory and the local feature memory is calculated through K-nearest neighbor retrieval, and the feature distance image-level score and the feature distance pixel-level response map are output. S4: Robustly normalize the reconstruction error image-level score and the feature distance image-level score, then weight and fuse them according to the image-level fusion weights to obtain a comprehensive image-level anomaly score; Robustly normalize the reconstruction error pixel-level response map and the feature distance pixel-level response map, then weight and fuse them according to the pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map; When the comprehensive image-level anomaly score exceeds the image-level judgment threshold, it is determined that there is a defect in the current image block, and the comprehensive pixel-level anomaly map is binarized using a pixel-level segmentation threshold to mark the defect area; S5: Stitch together the defect areas of all image blocks according to their spatial locations, perform boundary compensation by distance-weighted fusion on overlapping areas, form a whole-board level anomaly heat map, and output the defect location and alarm information including the comprehensive image level anomaly score.

[0009] To optimize the above technical solution, the specific measures also include: In step S1, the process of training a multi-granularity autoencoder using normal sample images is as follows: A spatial structure autoencoder, a scale semantic autoencoder, and a color distribution autoencoder are trained separately, all using only normal sample images. The optimizer is Adam. The loss function of the spatial structure autoencoder includes mean squared error, an L1 auxiliary term, and the KL divergence between the encoder posterior distribution and the Gaussian prior. The loss functions of the scale semantic autoencoder and the color distribution autoencoder include a weighted combination of mean squared error and an L1 auxiliary term. After the three autoencoders are trained, their parameters are frozen, and then the dynamic fusion module is trained using normal sample images. During the training process, entropy regularization constraints on the weight distribution are introduced.

[0010] The spatial structure autoencoder is a convolutional variational autoencoder, used to characterize the disruption of board surface continuity caused by cracks, holes and edge defects; the scale semantic autoencoder is a residual pyramid fusion autoencoder, used to characterize cross-scale anomalies such as knots, texture misalignment and local structural incoordination; the color distribution autoencoder is a multi-branch fusion attention autoencoder, used to characterize color anomalies such as water stains, contamination, color difference and uneven coating.

[0011] In step S1, the construction of the feature memory specifically includes: using a pre-trained network with frozen parameters to extract intermediate layer features from all normal sample images; performing local neighborhood average pooling on the extracted features; upsampling the features at different levels and then concatenating them along the channel dimension to form a multi-scale patch-level feature representation; performing random subsampling on the collected patch features and retaining a representative compact subset as the final feature memory.

[0012] In step S3, the reconstruction error pixel-level response map is generated as follows: for the input image and the reconstruction image of each autoencoder, the mean square error map and the structural similarity error map are calculated pixel by pixel; the mean square error map and the structural similarity error map are weighted and fused according to a preset weight to obtain a hybrid error map; after performing multi-scale average pooling on the hybrid error map, the maximum value of each pixel is taken to obtain the reconstruction error pixel-level response map of that autoencoder.

[0013] In step S2, the preprocessing includes adaptive illumination correction and reflection suppression processing, specifically: global brightness prediction and exposure compensation, local illumination component estimation based on a large-scale Gaussian smoothing kernel, generation of a continuous-value reflection mask based on the local brightness ratio, and partitioned adaptive correction based on the mask to perform illumination normalization correction on non-reflective areas and neighborhood weighted smoothing interpolation compensation on reflective areas. After correction, the image is cropped into image blocks according to a preset window, and the spatial position correspondence of the images in the corrected image is preserved.

[0014] In step S4, the robust normalization includes determining normalization parameters based on the score distribution of normal samples or validation samples, and mapping the reconstruction error image-level score, feature distance image-level score, reconstruction error pixel-level response map, and feature distance pixel-level response map to a unified numerical range; the weighted fusion includes generating a comprehensive image-level anomaly score according to the image-level fusion weights, and generating a comprehensive pixel-level anomaly map according to the pixel-level fusion weights; the defect region generation includes extracting high-response regions based on the comprehensive pixel-level anomaly map, and post-processing the high-response regions to obtain the defect regions.

[0015] After step S4, there is also a defect attribute auxiliary discrimination step: extract five types of visual features, namely area, shape, texture direction, color difference and edge continuity, for the located abnormal area, and use a multi-level decision tree for attribute labeling; the first level is divided into linear defects, patch defects and irregular defects according to aspect ratio and roundness, and the second level is further subdivided and judged in each branch by combining edge sharpness, texture direction deviation and color difference.

[0016] As another important technical solution, this invention also provides an online detection system for plate surface defects based on multi-granularity reconstruction and feature memory matching, used to implement the online detection method for plate surface defects based on multi-granularity reconstruction and feature memory matching as described above, including: The sample training and memory construction module is used to collect normal sample images of the board surface, train a multi-granularity autoencoder and build a feature memory, and calibrate the image-level decision threshold, pixel-level segmentation threshold, image-level fusion weight and pixel-level fusion weight. The online image acquisition and block preprocessing module is used to acquire surface images of the board material to be inspected on the production line, perform preprocessing, and then divide the image into grid blocks according to preset dimensions to obtain multiple image blocks. The dual-path parallel anomaly detection module is used to feed each image block into a multi-granularity autoencoder branch and a feature memory branch in parallel. The multi-granularity autoencoder branch outputs an image-level score and a pixel-level response map of the reconstruction error. The feature memory branch extracts image block features based on a pre-trained network, calculates the feature deviation distance through feature matching, and outputs an image-level score and a pixel-level response map of the feature distance. The fusion judgment and threshold segmentation module is used to robustly normalize the reconstruction error image-level score and the feature distance image-level score, and then fuse them according to the image-level fusion weights to obtain a comprehensive image-level anomaly score; it also robustly normalizes the reconstruction error pixel-level response map and the feature distance pixel-level response map, and then fuses them according to the pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map; when the comprehensive image-level anomaly score exceeds the image-level judgment threshold, it is determined that there is a defect in the current image block, and the comprehensive pixel-level anomaly map is binarized using a pixel-level segmentation threshold to mark the defect area; The whole-board reconstruction and alarm output module is used to stitch together the defect areas of all image blocks according to their spatial locations, perform boundary compensation on overlapping areas, form a whole-board level anomaly heat map, and output the defect location and alarm information.

[0017] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for online detection of surface defects in sheet metal based on multi-granularity reconstruction and feature memory matching.

[0018] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute an online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching as described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention uses two detection paths in parallel: a multi-granularity autoencoder and a feature memory matching. This enables model training and detection preparation to be completed with only normal samples. It is more suitable for the actual situation in the production line of sheet metal where there are diverse defect morphologies and difficult sample collection, and reduces the cost of data preparation and model maintenance.

[0020] This invention models the structural continuity, cross-scale texture variation, and color distribution of the board surface using a multi-granularity autoencoder. The feature memory branch captures any abnormal patterns that deviate from the normal feature distribution. After the two are fused, they can produce a stable response to various known or unknown defects such as cracks, holes, knots, water stains, and color differences. The reliability of detection is especially high for complex defects with unclear boundaries or irregular shapes.

[0021] This invention employs gridded segmentation for large-size sheet metal inspection and distance-weighted fusion for boundary compensation in overlapping areas, avoiding splicing artifacts and ultimately generating a continuous, whole-sheet-level anomaly heatmap. Simultaneously, based on multi-level decision tree-based defect attribute assistance, it can provide a description of defect type trends without requiring additional labeled samples, offering interpretable reference information for on-site quality inspectors.

[0022] While maintaining high detection accuracy, this invention has low requirements for equipment, materials, environment, and personnel skills. It has good replicability and cross-scenario transferability, making it suitable for promotion and application in the sheet metal processing industry and even in the broader field of surface quality inspection. Attached Figure Description

[0023] Figure 1 : A schematic diagram of the system device structure in an embodiment of the present invention.

[0024] Figure 2 : This is a schematic diagram of the process in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0026] In some implementations, such as Figure 2 As shown, this invention provides an online detection method for surface defects in sheet metal based on multi-granularity reconstruction and feature memory matching, comprising the following steps: S1: Acquire normal sample images of the board surface, train a multi-granularity autoencoder using the normal sample images and construct a feature memory library, calibrate image-level judgment threshold, pixel-level segmentation threshold, image-level fusion weight, and pixel-level fusion weight; the multi-granularity autoencoder includes three autoencoders: spatial structure, scale semantics, and color distribution; the feature memory library extracts features of normal samples based on a pre-trained network with frozen parameters, and constructs a global feature memory library for image-level scoring and a local feature memory library for pixel-level localization respectively; after the board enters the detection area with the conveyor mechanism, the trigger sensor sends an image acquisition signal, and the industrial camera continuously images the current board surface to obtain normal sample images; In some implementations, the spatial structure autoencoder, scale semantic autoencoder, and color distribution autoencoder are trained independently using the same basic optimization configuration, with the optimizer employing Adam (learning rate 100%). The weight decays to The batch size is 64, training lasts for 100 epochs, and the learning rate is scheduled using a cosine annealing (Cosine AnnealingLR) strategy. Gradient clipping (maximum norm 1.0) is applied uniformly during training, and mixed precision training is enabled to accelerate computation. An early stopping mechanism is also introduced: training is terminated early when the validation loss does not decrease for 12 consecutive epochs, and the system automatically saves the weight file with the optimal validation loss.

[0027] In some implementations, a spatial structure autoencoder (CVAE) is used to characterize surface continuity disruptions caused by cracks, holes, and edge defects. It employs an encoder-latent space-decoder structure, with the encoder consisting of four Conv–BN–LeakyReLU modules. The number of channels progressively increases from 32 to 64 to 128 to 256, used to gradually extract and compress the input image. The latent space is a low-dimensional feature representation space formed after the image structure information is compressed by the encoder. In other words, after the image is processed by the encoder, the features are encoded into a set of machine-understandable arrays. In this embodiment, its dimension is set to 128. Unlike ordinary autoencoders that directly obtain fixed feature vectors, CVAE models the mean of the latent representation at the bottleneck layer. With log variance Latent variables are obtained through reparameterization techniques:

[0028] in, This represents the latent variables obtained from sampling. This represents the mean vector of the latent representation. The standard deviation vector of the latent representation. It indicates that it follows a standard normal distribution. N(0, I Random noise, N (0, I () indicates that the mean is 0 and the covariance matrix is ​​the identity matrix. I The standard normal distribution This indicates element-wise multiplication.

[0029] The decoder then with Using the transposed convolution as input, the image is gradually restored to obtain the reconstructed image. When the input image contains cracks, holes, or missing corners, its local structure deviates from the normal plate surface distribution, increasing the model reconstruction error and reflecting spatial structural anomalies. The training loss function of the spatial structure autoencoder (CVAE) is:

[0030] in, Indicates the input image With reconstructed image The mean square error between them The L1 norm measure representing the absolute difference between the two pixels is used as an L1 auxiliary term. This enhances the model's sensitivity to edge and local detail errors; For encoder posterior With Gaussian prior KL divergence between them is the KL term tradeoff coefficient.

[0031] In some implementations, the scale-semantic autoencoder RPF-AE is used to characterize cross-scale anomalies such as knots, texture misalignments, and local structural inconsistencies. It uses a ResNet18 pre-trained on ImageNet as the residual encoder to extract multi-scale features from different layers. By fusing features from each layer in a top-down manner through a Feature Pyramid Network (FPN), a multi-scale feature representation with stronger expressive power is formed. (256 channels, 64×64). The decoder sequentially performs two levels of transposed convolution upsampling (256 to 128 to 64, stride 2) to restore the original resolution, and then performs two layers of convolutional refinement (64 to 32 to 3) to output the reconstructed image.

[0032] The training loss function for the scale semantic autoencoder RPF-AE is:

[0033] In some implementations, the color distribution autoencoder MBFA-AE is used to characterize color anomalies such as water stains, contamination, color difference, and uneven coating. The input image is decomposed into three single-channel sub-images according to the RGB color channels, and these sub-images are fed into three parallel branch encoders (homogeneous but heterogeneous, with identical network structures but independent weights). Each branch encoder consists of a 3-level Conv–BN–LeakyReLU module, with 1 to 32 to 64 to 128 channels, using a 4×4 convolutional kernel and a stride of 2. The three branch encoders extract the encoded features of the red, green, and blue channels respectively, denoted as... , and Because color anomalies such as water stains, contamination, color difference, and uneven coating have different response intensities across different color channels, this embodiment introduces a channel attention module to adaptively weight and fuse the three encoded features before they enter the shared decoder, resulting in fused features:

[0034] in, Indicates weight, ,satisfy The RGB image is dynamically generated by the channel attention module (global average pooling + double-layer fully connected + Softmax) based on the input image. The fused features are input to a shared decoder (2-level ConvTranspose–BN–ReLU + terminal ConvTranspose–Sigmoid, with the number of channels ranging from 128 to 64 to 32 to 3) and reconstructed into an RGB image.

[0035] In some implementations, the feature memory branch does not require training network parameters and is constructed using a one-time feature extraction and storage method. The specific steps are as follows: Using a pre-trained WideResNet50-2 with frozen parameters as the feature extraction backbone, forward inference was performed on all normal training samples to extract features from three intermediate layers: layer1 (256 channels, spatial resolution 64×64), layer2 (512 channels, spatial resolution 32×32), and layer3 (1024 channels, spatial resolution 16×16), taking into account both shallow spatial details and high-level semantic context information.

[0036] Local neighborhood average pooling (3×3 kernel) is performed on the extracted features to enhance the ability of each patch to perceive information from the surrounding area, forming a patch-level feature representation with spatial context awareness.

[0037] The 32×32 branch concatenates the features of layer2 and the upsampled features of layer3 to form a 1536-dimensional feature vector; the 64×64 branch concatenates the features of layer1, the upsampled features of layer2 and layer3 to form a 1792-dimensional feature vector.

[0038] In some implementations, to control the retrieval overhead during the inference phase, random subsampling is performed on the collected features (with a maximum memory size of 50,000 patches per scale), retaining only a representative, compact subset as the final memory. .

[0039] Global and local feature memories are constructed at two patch scales: 32×32 and 64×64. The 32×32 scale focuses on semantic discrimination information for image-level anomaly scoring, while the 64×64 scale focuses on spatial resolution for pixel-level anomaly localization.

[0040] Calibrate image-level fusion weights on the validation set Pixel-level fusion weights Image-level decision threshold Pixel-level segmentation threshold And post-processing parameters (morphological kernel size, region growth step size, etc.).

[0041] The overall optimization objective of the calibration is:

[0042] in, The anomaly score measures the overall ability of the score to distinguish between defective pixels and normal pixels; PRO (Per-Region Overlap) treats each independent defective region as an equally weighted evaluation unit and calculates the average coverage ratio of the prediction results for each defective region. This is a weighting factor.

[0043] In some implementations, the percentile cutoff boundary used for normalization is calculated from the score distribution of normal validation samples after training and is saved independently in a calibration file according to product category, along with parameters such as fusion weights and thresholds. After calibration, the system packages all trained model weights, memory, and calibration parameters and distributes them to the edge.

[0044] S2: Collect surface images of the board to be inspected on the production line, perform preprocessing, and then divide the board into grid blocks according to preset dimensions to obtain multiple image blocks; Preprocessing includes: calculating the global mean brightness of the input image. , compared with the preset standard reference brightness Compare them.

[0045] like If the exposure deviation threshold is reached, then exposure compensation will be applied to the entire image.

[0046] in, Input image in pixel coordinates The original grayscale / brightness value at that location; For exposure compensation at pixel coordinates The grayscale / brightness value at the specified location; global brightness is corrected to the standard range to eliminate overall exposure shift caused by light source aging, ambient light interference, or differences in board reflectivity; if the deviation is within the allowable range, this step is skipped and proceeds directly to subsequent processing. When performing exposure compensation, = When the exposure deviation is within the acceptable range and exposure compensation is skipped, = .

[0047] In some implementations, The input image is then used in the subsequent illumination correction process, employing a large-scale Gaussian smoothing kernel. ( A low-pass filter is applied to the image, with values ​​ranging from 1 / 8 to 1 / 4 of the shorter side, to extract the local illumination estimate. This component reflects the gradual change in brightness caused by the non-uniform spatial distribution of the light source and the macroscopic curvature of the board surface, and its expression is:

[0048] Threshold determined by reflection Generate continuous-value reflective masks using truncation functions :

[0049] Among them, the reflection detection threshold Take 1.2, Areas close to 1 are highly reflective, while areas close to 0 are under normal lighting. For each pixel, the local brightness ratio is used. When the local brightness ratio significantly exceeds 1 (i.e., the pixel brightness is much higher than the local average level), it is determined to be a reflective area. It is a manually added, extremely small positive number to prevent the denominator from being zero. Using a continuous mask instead of a binary mask enables a smooth transition between reflective and normal areas, avoiding abrupt boundary changes in the corrected image.

[0050] In some implementations, for non-reflective areas ( Perform illumination normalization: divide the pixel value by the local illumination component and multiply by the global mean. This eliminates spatial lighting unevenness, ensuring that all areas of the board are illuminated under uniform lighting conditions. For reflective areas, since the original pixel information has been distorted due to saturation, a smaller-scale Gaussian kernel is used. ( Weighted smooth interpolation compensation is applied to neighboring pixels to restore the approximate appearance of the reflective area using the texture and brightness information of the surrounding normal area; for intermediate transition areas, linear interpolation is performed between the two correction results based on the mask value.

[0051] Perform the corrected image Range truncation prevents numerical overflow, resulting in the final corrected output image. :

[0052] in, Indicates the input image The global average grayscale value is used to restore the non-reflective areas after local illumination normalization to a grayscale level consistent with the overall brightness of the original image. The expression is:

[0053] in, For image pixel domain; This indicates that the standard deviation is used. The result after smoothing the input image using a Gaussian kernel; This means that the correction result will be limited to the grayscale range of [0, 255] to avoid pixel value overflow.

[0054] In some implementations, the corrected output image Based on its actual length and width dimensions, it is divided into grid blocks of 256×256 window size, which can be divided along the width direction into The block (rounded up) can be divided along the height direction into... Blocks (rounded up), each corrected output image generates a total of training image patches (of which) , These represent the width and height in pixels of the original image, respectively; for areas with edges less than 256 pixels, ... After mirror padding, the images are incorporated into the blocks. All image blocks are uniformly normalized using ImageNet mean and standard deviation. Data augmentation strategies are enabled during subsequent training, including random horizontal flipping (probability 0.5) and random rotation (±10).

[0055] S3: Each image block is fed in parallel into the multi-granularity autoencoder branch and the feature memory branch; the multi-granularity autoencoder branch inputs the image block into three autoencoders respectively to obtain three reconstructed images and reconstruction errors, and outputs the image-level score of reconstruction error and the pixel-level response map of reconstruction error after dynamic fusion; the feature memory branch inputs the same image block into the pre-trained network to extract features, calculates the feature deviation distance from the global feature memory and the local feature memory through K-nearest neighbor retrieval, and outputs the feature distance image-level score and the feature distance pixel-level response map.

[0056] Image patches are input into three autoencoders. In CVAE, the input images are convolutionally encoded, latent space reparameterized sampled, and decoded to output spatially reconstructed images. In RPF-AE, multi-scale features are extracted using a ResNet18 encoder, fused by FPN, and then decoded to output a scale-semantic reconstructed image. In MBFA-AE, the images are encoded separately for each of the RGB channels, then fused using channel attention adaptive weighting, and finally reconstructed by a shared decoder, outputting a color distribution reconstructed image. .

[0057] In some implementations, the current input image block is denoted as B; B is the preprocessed corrected image from step S2. The pixel value at pixel coordinates (x, y) is denoted as B(x, y) obtained by capturing the image within a preset window. The current input image patch is compared pixel-by-pixel with the three reconstructed images, and the corresponding MSE reconstruction error is calculated. This is then processed using multi-scale average pooling (with window sizes corresponding to...). Then, the maximum value for each pixel is taken to enhance the response sensitivity to defects of different sizes.

[0058] in, For the first i The reconstructed image output by the autoencoder, where These correspond to the three reconstruction branches: CVAE, RPF-AE, and MBFA-AE, respectively. Indicates the first i The path reconstructs the image in pixel coordinates ( x,y The pixel value at ().

[0059] Calculate the original reconstruction error for each pixel:

[0060] Three-way reconstruction error , Adaptive weights are calculated by the GAT dynamic fusion module. The specific process is as follows: The Graph Attention Network (GAT) represents the three-way reconstruction error as graph nodes. Establish fully connected adjacency relationships for nodes. and nodes The attention coefficient is calculated as follows:

[0061] in, To share linear transformation parameters, For attention vectors, This indicates a splicing operation. For nodes The set of neighbors.

[0062] The merged node is represented as follows:

[0063] Weighted fusion yields a pixel-level response map of the reconstruction error:

[0064] Reconstruction error image-level fraction The calculation formula is:

[0065] in, The maximum image-level score for reconstruction error The average value of each pixel ( (2% of the total number of pixels) The global mean. This is the mixing coefficient. This strategy focuses the scoring on locally high-response areas while preserving a baseline level of global background.

[0066] In some implementations, this invention employs a two-layer cascaded GAT structure, with four attention heads per layer and a hidden dimension of 64. The weight prediction head concatenates the GAT output with the original node features using residuals, then outputs three fused weights via a fully connected layer and a learnable temperature-scalable Softmax layer. Since the training data consists entirely of normal samples, the optimization objective of GAT is to minimize the fused anomaly score, thereby generating a significantly higher score response for anomaly samples than normal samples during the detection phase. To prevent the fused weights from collapsing to rely solely on a single branch, an entropy regularization constraint (regularization coefficient) is introduced during training to regulate the weight distribution. The GAT fusion module was trained for 50 epochs.

[0067] The same image patch is fed into WideResNet50-2 with frozen parameters to extract intermediate layer features. The patch test feature at a certain spatial location of the current image patch is denoted as... ; Patch features at each spatial location are obtained through Nearest neighbor search ( Calculate the feature deviation distance between it and the memory bank:

[0068] in, For the mid-range test features of the memory bank The K most recent feature vectors, For Euclidean distance, Represents a set Any normal patch feature vector in the.

[0069] In some implementations, the set of anomaly distances for all patch features is calculated, and the T largest distances are selected and averaged, where T is a preset number or a positive integer determined according to a preset proportion of the total number of patches, and this average is used as the feature distance image-level score. The anomaly distances of each patch are rearranged according to their spatial position in the feature map to form a low-resolution feature distance map. ,right Perform bilinear interpolation upsampling followed by Gaussian smoothing to obtain the pixel-level response map of the feature distance. The expression is: .

[0070] in, This is a bilinear interpolation upsampling operation. A low-resolution feature distance map is formed by rearranging the K nearest neighbor anomaly distances corresponding to 64×64 patch features according to their spatial locations. (·) indicates Gaussian smoothing with a standard deviation of σ; The larger the value of the median, the more significant the deviation between the patch features at that location and the normal memory features, and the more likely it is to correspond to a defect area.

[0071] S4: Robustly normalize the reconstruction error image-level score and the feature distance image-level score, then weight and fuse them according to the image-level fusion weights to obtain a comprehensive image-level anomaly score; Robustly normalize the reconstruction error pixel-level response map and the feature distance pixel-level response map, then weight and fuse them according to the pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map; When the comprehensive image-level anomaly score exceeds the image-level judgment threshold, it is determined that there is a defect in the current image block, and the comprehensive pixel-level anomaly map is binarized using a pixel-level segmentation threshold to mark the defect area; In some implementations, since the dimensions and dynamic range of the reconstruction error image-level score and the feature distance image-level score differ significantly, direct fusion may lead to one pathway dominating the final score. Therefore, robust normalization based on percentiles is introduced before fusion.

[0072]

[0073] in, and These are the first and second digits of the normal sample score distribution during the training phase. and the percentiles (take) After truncation, it is mapped to min-max normalization. Interval.

[0074] The comprehensive image-level anomaly score is expressed as:

[0075] Pixel-level anomaly map: Before fusion, the system's pixel-level response map to reconstruction errors. and feature distance pixel-level response map Image-by-image normalization is performed separately to map the value ranges of both to a consistent range, thus eliminating the dimensional differences between the reconstruction error path and the feature memory matching path. The two types of pixel-level response maps after normalization are then weighted and fused according to pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map.

[0076] in, This represents the pixel-level response map of the reconstruction error after normalization. γ represents the pixel-level response map of the normalized feature distance; γ is the pixel-level fusion weight of the reconstruction error response map. , which represents the pixel-level fusion weights of the feature distance response map. This is the final pixel-level anomaly map used for defect localization.

[0077] S5: Stitch together the defect areas of all image blocks according to their spatial locations, perform boundary compensation by distance-weighted fusion on overlapping areas, form a whole-board level anomaly heat map, and output the defect location and alarm information including the comprehensive image level anomaly score.

[0078] In some implementations, for large-sized wood panels, the system divides the panel surface into multiple image blocks for separate detection. The detection results of each image block are then stitched together to reconstruct a full-panel anomaly heatmap. During the image segmentation stage, a preset width overlap region is set between adjacent image blocks, with the overlap width being 12.5% ​​to 25% of the image block's side length. After each image block completes its detection, the pixel-level anomaly scores within the overlap region are stitched together using a distance-weighted fusion method: the final anomaly score for each pixel is a weighted average of the detection results from its respective image blocks, calculated based on the pixel's distance to its block boundary. Results closer to the block center receive higher weights, while results closer to the block boundary receive lower weights. This strategy ensures a smooth transition of anomaly maps between adjacent image blocks within the overlap region, avoiding stitching artifacts caused by feature truncation at block boundaries, thus forming a continuous full-panel anomaly heatmap.

[0079] Following step S4, a defect attribute-assisted discrimination step is also included: Threshold segmentation is performed on the fused pixel-level anomaly map (threshold segmentation). This separates the high-response region from the background; morphological opening and closing operations are performed sequentially (structuring element size is...). The former suppresses discrete noise points, while the latter fills small holes within the defect region. A region growing algorithm with progressive threshold decay is used to gradually expand the defect boundary starting from the candidate seed region. Finally, the grown region is area filtered to output the region mask and bounding box of the defect location.

[0080] Five visual features—area, shape, texture direction, color difference, and edge continuity—are extracted from the located anomaly regions, and attribute annotation is performed using a multi-level decision tree. The first level classifies abnormal areas into three categories based on aspect ratio and roundness: linear defects, patchy defects, and irregular defects. The second level further refines the judgment within each branch by combining edge sharpness, texture direction deviation, brightness difference direction, and area ratio. For example, a linear area with sharp edges along the wood grain direction is labeled as a crack, a patchy area with gentle edges and a darker appearance is labeled as pollution or water stains, and a large area with small color difference is labeled as color difference. This discrimination process is based on rule reasoning of geometric and visual features and does not rely on supervised training of defect category labeled samples.

[0081] In some embodiments, the present invention includes a model collaborative update step between the edge and the cloud: a synchronization strategy combining event triggering and timed batching is adopted between the edge and the cloud. Specifically, during the detection process, the edge uploads samples judged as abnormal and samples with a comprehensive image-level anomaly score in the low confidence range (e.g., close to the image-level judgment threshold) to the cloud in real time via event triggering; detection logs and statistical information are uploaded in batches at preset intervals. After the cloud completes the model update, it sends the new version of the multi-granularity autoencoder weights, feature memory, and calibration parameters (including image-level judgment threshold, pixel-level segmentation threshold, image-level fusion weights, and pixel-level fusion weights) to the edge in the form of incremental difference packets. The edge loads the updated parameters during the current detection interval without interrupting the online detection process. The above communication process is based on HTTPS protocol transmission and adopts a breakpoint resume mechanism to cope with network fluctuations, ensuring the integrity of the transmission of model files and abnormal sample data. Figure 1 As shown.

[0082] The invention also includes result display and sorting execution steps: On the industrial control display terminal, the original image of the board to be inspected, the overall board anomaly heat map, the annotation results of defect attribute auxiliary judgment, and the comprehensive image-level anomaly score are displayed on the same screen. When the comprehensive image-level anomaly score exceeds the preset alarm threshold, an on-site audible and visual alarm and a software interface pop-up alarm are triggered. At the same time, a graded control signal is output to the downstream sorting mechanism through the industrial IO interface: defect-free boards are introduced into the qualified product channel, and defective boards are introduced into the defective product channel; for boards whose comprehensive image-level anomaly score is within the boundary range (e.g., a preset fuzzy range near the image-level judgment threshold), they are marked as pending re-inspection and are manually confirmed a second time.

[0083] Example 1: The present invention was verified using a test set of wood-based panel surfaces. The test set consisted of 79 images. The results of the testing and evaluation are shown in Table 1. Table 1 Overall performance indicators of the present invention

[0084] Table 2. Identification accuracy of different sample types

[0085] To verify the impact of each component on the detection effect, comparative experiments were conducted by removing the feature memory matching branch, the dynamic fusion module, and the reconstruction branches of different granularities. The experimental results are shown in Table 3.

[0086] Table 3 Ablation Experiment Results

[0087] Experimental results show that in scenarios with complex wood grain backgrounds, blurred boundary defects, and composite defects, methods with a single detection branch or lacking key components have weak characterization capabilities for abnormal areas, easily leading to decreased defect detection capabilities or inaccurate localization. In contrast, this invention, through the fusion and judgment of dual-path detection results, can more accurately identify and locate surface defects in boards.

[0088] As another important technical solution, this invention also provides an online detection system for plate surface defects based on multi-granularity reconstruction and feature memory matching, used to implement the online detection method for plate surface defects based on multi-granularity reconstruction and feature memory matching as described above, including: The sample training and memory construction module is used to collect normal sample images of the board surface, train a multi-granularity autoencoder and build a feature memory, and calibrate the image-level decision threshold, pixel-level segmentation threshold, image-level fusion weight and pixel-level fusion weight. The online image acquisition and block preprocessing module is used to acquire surface images of the board material to be inspected on the production line, perform preprocessing, and then divide the image into grid blocks according to preset dimensions to obtain multiple image blocks. The dual-path parallel anomaly detection module is used to feed each image block into a multi-granularity autoencoder branch and a feature memory branch in parallel. The multi-granularity autoencoder branch outputs an image-level score and a pixel-level response map of the reconstruction error. The feature memory branch extracts image block features based on a pre-trained network, calculates the feature deviation distance through feature matching, and outputs an image-level score and a pixel-level response map of the feature distance. The fusion judgment and threshold segmentation module is used to robustly normalize the reconstruction error image-level score and the feature distance image-level score, and then fuse them according to the image-level fusion weights to obtain a comprehensive image-level anomaly score; it also robustly normalizes the reconstruction error pixel-level response map and the feature distance pixel-level response map, and then fuses them according to the pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map; when the comprehensive image-level anomaly score exceeds the image-level judgment threshold, it is determined that there is a defect in the current image block, and the comprehensive pixel-level anomaly map is binarized using a pixel-level segmentation threshold to mark the defect area; The whole-board reconstruction and alarm output module is used to stitch together the defect areas of all image blocks according to their spatial locations, perform boundary compensation on overlapping areas, form a whole-board level anomaly heat map, and output the defect location and alarm information.

[0089] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching as described above.

[0090] In another embodiment of the present invention, a computer-readable storage medium is provided, storing a computer program that causes a computer to execute an online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching as described above.

[0091] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. An online detection method for surface defects in sheet metal based on multi-granularity reconstruction and feature memory matching, characterized in that, Includes the following steps: S1: Collect normal sample images of the board surface, train a multi-granularity autoencoder and build a feature memory library, and calibrate the image-level decision threshold, pixel-level segmentation threshold, image-level fusion weight and pixel-level fusion weight; S2: Collect surface images of the board to be inspected on the production line, perform preprocessing, and then divide the board into grid blocks according to preset dimensions to obtain multiple image blocks; S3: Each image patch is fed in parallel into the multi-granularity autoencoder branch and the feature memory branch; the multi-granularity autoencoder branch outputs the reconstruction error image-level score and the reconstruction error pixel-level response map; the feature memory branch extracts image patch features based on the pre-trained network, calculates the feature deviation distance through feature matching, and outputs the feature distance image-level score and the feature distance pixel-level response map. S4: Robustly normalize the reconstruction error image-level score and the feature distance image-level score, then weight and fuse them according to the image-level fusion weights to obtain a comprehensive image-level anomaly score; Robustly normalize the reconstruction error pixel-level response map and the feature distance pixel-level response map, then weight and fuse them according to the pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map; When the comprehensive image-level anomaly score exceeds the image-level judgment threshold, it is determined that there is a defect in the current image block, and the comprehensive pixel-level anomaly map is binarized using a pixel-level segmentation threshold to mark the defect area; S5: Stitch together the defect areas of all image blocks according to their spatial locations, perform boundary compensation on overlapping areas, form a whole-board level anomaly heat map, and output the defect location and alarm information.

2. The online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching according to claim 1, characterized in that: In step S1, the training of the multi-granularity autoencoder specifically involves the following process: The multi-granularity autoencoder includes a spatial structure autoencoder, a scale semantic autoencoder, and a color distribution autoencoder. All three autoencoders are trained using normal sample images, and the optimizer is Adam. The loss function of the spatial structure autoencoder includes mean squared error, an L1 auxiliary term, and the KL divergence between the encoder posterior distribution and the Gaussian prior. The loss functions of the scale semantic autoencoder and the color distribution autoencoder contain a weighted combination of mean squared error and an L1 auxiliary term. After the three autoencoders are trained, their parameters are frozen, and then the dynamic fusion module is trained using normal sample images. During the training process, entropy regularization constraints on the weight distribution are introduced.

3. The online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching according to claim 1, characterized in that: In step S1, the construction of the feature memory specifically includes: extracting intermediate layer features from all normal sample images using a pre-trained network with frozen parameters; performing local neighborhood average pooling on the extracted features; upsampling the features at different levels and then concatenating them along the channel dimension to form a multi-scale patch-level feature representation; performing random subsampling on the collected patch features and retaining a representative compact subset as the final feature memory, thereby constructing a global feature memory for image-level scoring and a local feature memory for pixel-level localization.

4. The online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching according to claim 2, characterized in that: The spatial structure autoencoder is a convolutional variational autoencoder, used to characterize the disruption of board surface continuity caused by cracks, holes and edge defects; the scale semantic autoencoder is a residual pyramid fusion autoencoder, used to characterize cross-scale anomalies such as knots, texture misalignment and local structural incoordination; the color distribution autoencoder is a multi-branch fusion attention autoencoder, used to characterize color anomalies such as water stains, contamination, color difference and uneven coating.

5. The online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching according to claim 1, characterized in that: In step S2, the preprocessing includes adaptive illumination correction and reflection suppression processing, specifically: global brightness prediction and exposure compensation, local illumination component estimation based on a large-scale Gaussian smoothing kernel, generation of a continuous-value reflection mask based on the local brightness ratio, and partitioned adaptive correction based on the mask to perform illumination normalization correction on non-reflective areas and neighborhood weighted smoothing interpolation compensation on reflective areas. After correction, the image is cropped into image blocks according to a preset window, and the spatial position correspondence of the images in the corrected image is preserved.

6. The online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching according to claim 1, characterized in that: In step S4, the robust normalization includes determining normalization parameters based on the score distribution of normal samples or validation samples, and mapping the reconstruction error image-level score, feature distance image-level score, reconstruction error pixel-level response map, and feature distance pixel-level response map to a unified numerical range; the weighted fusion includes generating a comprehensive image-level anomaly score according to the image-level fusion weights, and generating a comprehensive pixel-level anomaly map according to the pixel-level fusion weights; the defect region generation includes extracting high-response regions based on the comprehensive pixel-level anomaly map, and post-processing the high-response regions to obtain the defect regions.

7. The online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching according to claim 1, characterized in that: After step S4, a defect attribute auxiliary discrimination step is also included: extracting five types of visual features, namely area, shape, texture direction, color difference and edge continuity, for the located abnormal area, and using a multi-level decision tree for attribute labeling; the first level is divided into linear defects, patch defects and irregular defects according to aspect ratio and roundness, and the second level is further subdivided and judged in each branch by combining edge sharpness, texture direction deviation and color difference.

8. An online detection system for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching, used to implement the online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching as described in any one of claims 1 to 7, characterized in that, include: The sample training and memory construction module is used to collect normal sample images of the board surface, train a multi-granularity autoencoder and build a feature memory, and calibrate the image-level decision threshold, pixel-level segmentation threshold, image-level fusion weight and pixel-level fusion weight. The online image acquisition and block preprocessing module is used to acquire surface images of the board material to be inspected on the production line, perform preprocessing, and then divide the image into grid blocks according to preset dimensions to obtain multiple image blocks. The dual-path parallel anomaly detection module is used to feed each image block into a multi-granularity autoencoder branch and a feature memory branch in parallel. The multi-granularity autoencoder branch outputs an image-level score and a pixel-level response map of the reconstruction error. The feature memory branch extracts image block features based on a pre-trained network, calculates the feature deviation distance through feature matching, and outputs an image-level score and a pixel-level response map of the feature distance. The fusion judgment and threshold segmentation module is used to robustly normalize the reconstruction error image-level score and the feature distance image-level score, and then fuse them according to the image-level fusion weights to obtain a comprehensive image-level anomaly score; it also robustly normalizes the reconstruction error pixel-level response map and the feature distance pixel-level response map, and then fuses them according to the pixel-level fusion weights to obtain a comprehensive pixel-level anomaly map; when the comprehensive image-level anomaly score exceeds the image-level judgment threshold, it is determined that there is a defect in the current image block, and the comprehensive pixel-level anomaly map is binarized using a pixel-level segmentation threshold to mark the defect area; The whole-board reconstruction and alarm output module is used to stitch together the defect areas of all image blocks according to their spatial locations, perform boundary compensation on overlapping areas, form a whole-board level anomaly heat map, and output the defect location and alarm information.

9. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute an online detection method for surface defects of sheet metal based on multi-granularity reconstruction and feature memory matching as described in any one of claims 1 to 7.

Images