Metal product surface flaw detection method and system based on machine vision

Abstract

The invention provides a metal product surface flaw detection method and system based on machine vision, and belongs to the technical field of machine vision. The method comprises the following steps: constructing a multi-modal image data set through bright field image acquisition, dark field image acquisition and three-dimensional point cloud data acquisition of the surface of a detected metal product, and registering the multi-modal image data set; based on the registration multi-modal image data of pixel-level alignment, pixel-level defect segmentation is carried out by adopting a double-flow diffusion Transform model, and a defect segmentation map is output; and on the basis of the defect segmentation image and the registered multi-modal image data, through segmentation image binaryzation and region extraction, multi-dimensional defect feature extraction and quantification, defect classification and report generation, a final defect detection report is output, and the metal product surface defect detection method is completed. The invention effectively solves the core pain points of low precision, poor robustness and insufficient generalization ability for unknown defects in metal surface flaw detection, and provides an automatic detection method with high reliability and high precision.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically relating to a method and system for detecting surface defects in metal products based on machine vision. Background Technology

[0002] Automated defect detection on the surface of metal products is a crucial aspect of industrial quality control; currently, mainstream methods still rely on manual visual inspection or traditional machine vision algorithms. Manual visual inspection is inefficient, prone to fatigue, highly subjective, and inconsistent in its standards. Traditional machine vision methods typically rely on two-dimensional images under a single light source, using algorithms such as threshold segmentation and edge detection for defect identification. This method has the following inherent drawbacks: the reflective properties of metal surfaces are complex, and a single lighting condition cannot comprehensively highlight all types of defects. Normal textures, oil stains, and water stains on the product surface are easily misjudged as defects. Feature extraction capabilities are limited; traditional algorithms struggle to effectively distinguish subtle defects from background noise, resulting in low detection rates for defects with low contrast to the background and varied shapes. The lack of three-dimensional information prevents the detection of defects such as pits and protrusions that are highly correlated with surface height. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides a method and system for detecting surface defects in metal products based on machine vision, reducing false alarms caused by environmental interference.

[0004] To achieve the above objectives, the following steps are included: S1. Multimodal image data acquisition and registration: A multimodal image dataset is constructed by acquiring bright field images, dark field images, and three-dimensional point cloud data of the surface of the tested metal product. The multimodal image dataset is then registered to obtain the registered multimodal image dataset. Specifically, the following steps are included: S1.1 Multimodal image data acquisition: After acquiring bright field images, dark field images and three-dimensional point cloud data of the surface of the tested metal product, the bright field image sequence, dark field image and three-dimensional point cloud data are output through hardware synchronization. The bright-field image acquisition specifically involves using an area array industrial camera with a resolution of no less than 2 million pixels, in conjunction with a ring LED light source, to image the metal product being tested at an angle perpendicular to the surface of the metal product. To overcome the strong reflectivity of metal surfaces, this invention employs multi-exposure image fusion technology: the camera is controlled to continuously acquire three images at the same shooting point with short exposure time, standard exposure time, and long exposure time respectively. Then, a weighted fusion algorithm based on the Laplacian pyramid is used to fuse the dark details that are not overexposed in the short exposure image, the bright details that are not underexposed in the long exposure image, and the midtone information of the standard exposure image to output a bright field image.

[0005] In this invention, bright-field images are mainly used to detect color changes, spots, oxidation, and corrosion; The dark field image acquisition is specifically as follows: using an area array industrial camera with a resolution of no less than 2 million pixels, the camera is placed at a low angle of 10° to 30° with the surface being measured, and a strip LED light source is used to illuminate the surface being measured in a grazing manner (the angle between the light direction and the tangent direction of the surface is less than 10°) to output a dark field image.

[0006] The specific acquisition of the three-dimensional point cloud data is as follows: a line laser scanner based on the principle of laser triangulation is used; a blue laser line with a wavelength of 450nm is projected onto the metal surface, and the deformation of the laser line is captured by a camera located at a fixed angle (30° in this invention) to the laser; the camera pixel coordinates are converted into three-dimensional coordinates by laser plane calibration and light stripe center extraction algorithm (Steger algorithm), and the three-dimensional point cloud data is output after scanning line by line.

[0007] The hardware synchronization specifically involves receiving encoder signals via a programmable logic controller (PLC). When the workpiece under test moves to the detection station, the PLC simultaneously sends hardware trigger signals to the bright-field camera, dark-field camera, and line laser scanner to ensure that the three sets of data—bright-field image, dark-field image, and 3D point cloud data—are acquired at the same physical location on the workpiece, ensuring that the images acquired in different modes for the same detection area are completely aligned in time and space.

[0008] This invention performs noise reduction, flattening correction (eliminating uneven illumination), and distortion correction on the acquired bright field images, dark field images, and 3D point cloud data.

[0009] S1.2 Multimodal Data Joint Calibration: Based on bright-field images, dark-field images, and 3D point cloud data, the initial transformation parameter matrix is ​​output through calibration plate preparation, 2D camera calibration, 3D scanner calibration, and joint calibration. The calibration plate is specifically prepared as follows: the substrate of the calibration plate is alumina ceramic with a checkerboard pattern on the surface with a side length of 5 mm. At the center of each checkerboard corner, a conical pit with a diameter of 50 μm and a depth of 25 μm is made using micromachining technology as a three-dimensional feature point.

[0010] The two-dimensional camera calibration specifically involves: placing the calibration board in different positions and orientations within the field of view, and acquiring no fewer than 15 images for both the bright-field and dark-field cameras; identifying the two-dimensional pixel coordinates of the checkerboard corner points, and using the Zhang Zhengyou calibration method to solve for the camera's intrinsic parameter matrix (including focal length, principal point coordinates, and distortion coefficients) and extrinsic parameter matrix (rotation and translation vectors) relative to the calibration board coordinate system at each placement.

[0011] The 3D scanner calibration specifically involves: at the same location, the 3D scanner scans the surface of the calibration plate, and through a 3D point cloud segmentation and fitting algorithm, identifies and locates the 3D spatial center coordinates of each conical pit. The transformation relationship from the 3D scanner coordinate system to the calibration plate coordinate system is then solved using least squares fitting.

[0012] The joint calibration specifically involves using the calibration board coordinate system as a common reference system. This invention defines the transformation from the bright-field camera to the calibration board as follows: The transformation from a 3D scanner to a calibration plate is The initial transformation parameter matrix from the 3D scanner to the bright-field camera is then... The calculation expression is as follows: Similarly, calculate the transformation from a dark field camera to a bright field camera.

[0013] S1.3 Multimodal Feature Matching: Based on bright field images, dark field images, 3D point cloud data and initial transformation parameter matrix, the matching is performed through field of view division and key point detection, hardening depth local feature descriptor extraction, feature matching and mismatch removal, and output of a set of matching feature point pairs. Specifically, the following steps are included: S1.3.1 Field of View Division and Key Point Detection: Based on bright field images, dark field images, and 3D point cloud data, through grid division, image key point detection, and point cloud key point detection, output the 2D key point set of the bright field image, the 2D key point set of the dark field image, and the 3D key point set of the 3D point cloud in each sub-region. The specific mesh division method of this invention involves using a bright-field image as a reference, uniformly dividing its field of view into a 10×10 grid, with each grid defined as a sub-region. Image keypoint detection is then performed. Within each sub-region, the Shi-Tomasi corner detection algorithm is executed in parallel on both the bright-field and dark-field images to extract strong corner points as candidate keypoints, outputting a 2D keypoint set for both the bright-field and dark-field images. Regarding point cloud keypoint detection, in the corresponding 3D point cloud sub-region, the point cloud is converted into a 2D grayscale height map (Z-coordinate mapped to pixel grayscale value) and a 2D average curvature map. Subsequently, the Harris corner detection algorithm is used on this 2D map to extract feature points, and through coordinate mapping relationships, the corresponding 3D spatial coordinates in the 3D point cloud are found, forming 3D point cloud keypoints, resulting in a 3D point cloud keypoint set.

[0014] S1.3.2 Hardening Depth Local Feature Descriptor Extraction: Input the 2D keypoint set of the bright field image, the 2D keypoint set of the dark field image, and the 3D keypoint set of the 3D point cloud. After training through the feature extraction network and loss function calculation, and after L2 normalization, output the 512-dimensional hardening depth local feature descriptor corresponding to each keypoint. The feature extraction network is a VGG16 network structure with 13 convolutional layers and 3 fully connected layers; The hardening depth local feature descriptor is extracted from the third convolutional layer of the fourth convolutional block in the VGG16 network structure. This invention defines a feature extraction network with a VGG16 network pre-trained on ImageNet as its backbone, using the output of the conv4_3 convolutional layer as a dense initial depth descriptor. The loss function is defined as follows: In the formula, For triplet loss, for an anchor image patch, the goal is to minimize its distance in feature space from positive samples (image patches at the same physical location under slight illumination changes or deformations) and maximize its distance from negative samples (image patches at different physical locations). For consistency loss, for the same physical keypoint, the cosine similarity of its feature vectors extracted from the bright-field and dark-field images is maximized after passing through a shared projection layer. The consistency loss weight value is set within the preset range [0.1, 2.0]. Subsequently, online descriptor computation is performed. For each keypoint detected in S1.3.1, the image patch around it is cropped and input into the fine-tuned VGG16 network to extract its conv4_3 features. After L2 normalization, the 512-dimensional hardened depth local feature descriptor corresponding to each keypoint is output.

[0015] S1.3.3 Feature Matching and Mismatch Removal: Based on the hardened depth local feature description operator and the initial transformation parameter matrix, matching is performed between bright field image and dark field image and between image and point cloud, outputting a set of matching feature point pairs; This invention inputs a 512-dimensional hardened depth local feature descriptor corresponding to each key point. First, the bright field image and dark field image are matched. Fast Approximate Nearest Neighbor Search (FLANN) is used to perform bidirectional matching of the descriptors of all sub-regions in the bright field and dark field images. Then, the Random Sample Consensus Algorithm (RANSAC) is used with the Fundamental Matrix as the geometric model to remove mismatched point pairs that do not conform to the epipolar constraints. Secondly, using the initial transformation parameter matrix After projecting the 3D keypoints onto the brightfield image plane, within the neighborhood of each projected point (with a preset radius of 10 pixels), the point with the closest descriptor cosine distance among the 2D keypoints in the brightfield image is found to form a 2D-3D matching pair, resulting in a set of matching feature point pairs.

[0016] S1.4, Pixel-level alignment: Based on the matching feature point pair set, initial transformation parameter matrix, dark field image and 3D point cloud data, the pixel-level aligned registration multimodal image data is output through local nonlinear transformation model calculation and image resampling and multi-resolution feathering fusion. S1.4.1 Calculation of Local Nonlinear Transformation Model: Based on the set of matching feature point pairs, the model is solved by using the moving least squares model and the adaptive regularization loss function, and the local MLS transformation model of all sub-regions is output. This invention is based on a set of matching feature point pairs within a sub-region, and establishes a Moving Least Squares (MLS) deformation model for each sub-region. For each matching point pair, the average curvature of the 3D point cloud within its local neighborhood is calculated; therefore, this invention defines an adaptive regularization weight function, the calculation expression of which is: In the formula, Control points The average curvature at that point, As a preset scale factor, The preset attenuation rate.

[0017] This invention and The value should be adjusted appropriately. Within the range [0.1, 1.0], Adjusting within the range of [20.0, 60.0] can achieve better registration results than the globally fixed regularization method; This invention uses all matching point pairs within a sub-region as control points, aims to minimize reprojection error, and combines adaptive regularization weight function calculation to solve for the parameters of the MLS transform function of that sub-region, outputting the local MLS transform model for all sub-regions. S1.4.2 Image resampling and multi-resolution feathering fusion: Based on the local MLS transform model of all sub-regions, dark field images and 3D point cloud data are resampled and feathered to output pixel-level aligned registered multimodal image data. The resampling includes dark field image resampling and 3D point cloud data resampling; The pixel-level aligned registration multimodal image data includes: bright field image, resampled dark field image, two-dimensional height map, and two-dimensional intensity map; The dark field image resampling method of the present invention is as follows: for each target pixel in the bright field image coordinate system, the source coordinates corresponding to it in the original dark field image are calculated using the local MLS transformation model of the sub-region where it is located; the pixel value is obtained from the original dark field image using bilinear interpolation and filled into the target position.

[0018] The specific method for resampling 3D point cloud data in this invention is as follows: First, an initial transformation parameter matrix is ​​used. Coarse alignment is performed; then, for each 3D point, a displacement correction is calculated using the local MLS transform model of the sub-region where it is projected onto the brightfield image, and fine correction is performed on the 3D point. Finally, the corrected 3D point cloud is projected onto the brightfield image plane to generate a 2D height map and a 2D intensity map that correspond one-to-one with the pixels of the brightfield image. Finally, a feathering fusion operation is performed, specifically: the overlapping area of ​​the resampling results of adjacent sub-regions is decomposed into three layers of Gaussian pyramids. On each layer of the Laplacian pyramid, linear weighted fusion is performed based on the distance of the pixel to the boundary of the two sub-regions (the closer to the boundary of sub-region A, the greater the weight; and vice versa). The fused Laplacian pyramids are then reconstructed using Gaussian pyramids to generate the final seamless registered image, outputting pixel-level aligned registered multimodal image data (bright field image, resampled dark field image, 2D height map, and 2D intensity map).

[0019] S2. Pixel-level defect segmentation using a dual-stream diffusion Transformer model: Based on pixel-level aligned multimodal image data, pixel-level defect segmentation is performed using a dual-stream diffusion Transformer model, and a defect segmentation map is output. The dual-stream diffusion Transformer model includes: an ideal reference image generation stream, a feature space difference module, and a diffusion-driven segmentation stream; Specifically, the following steps are included: S2.1 Ideal Reference Image Generation Stream: Input pixel-level aligned registered multimodal image data, and output an ideal reference image through a defect-aware editor and an ideal image generator; S2.1.1, Defect-Aware Editor: Input pixel-level aligned registered multimodal image data, and output a visual token sequence through modality-specific extraction, multimodal feature fusion and tokenization; This invention employs three convolutional coding subnetworks with identical structures but non-shared weights to extract modality-specific features, processing pixel-level aligned registered multimodal images. The network consists of a bright field image channel, a dark field image channel, and a 3D derived (height map / intensity map) channel. Each sub-network comprises four convolutional blocks, each containing a 3x3 convolutional layer, a batch normalization layer, and a ReLU activation function; convolutions with stride=2 are used for downsampling; finally, each sub-network outputs a 256-channel feature map. This invention forces each sub-network to focus on learning the unique feature representation of its corresponding imaging modality by enforcing non-sharing of weights: the bright-field sub-network focuses on color and macroscopic texture variations; the dark-field sub-network is highly sensitive to microscopic geometric undulations; and the three-dimensional sub-network directly captures the physical height information of the surface morphology.

[0020] The weights of the three sub-networks are independently updated and optimized through backpropagation during the end-to-end training of the dual-stream diffusion Transformer model. Specifically, during model training, the independent weight parameters of the three sub-networks are calculated and updated respectively by minimizing the overall objective function and using gradient descent.

[0021] The multimodal feature fusion and tokenization of this invention specifically involves concatenating the feature maps output from the three sub-networks along the channel dimension to obtain a 768-channel feature map. Subsequently, a fusion module incorporating 1x1 convolutions and channel attention mechanisms reduces the dimensionality of the channels and fuses them, ultimately outputting a 256-channel multimodal fused feature map. Multimodal feature map fusion Flattening the spatial dimensions yields feature vectors, which are used as visual token sequences, resulting in the output visual token sequence. .

[0022] S2.1.2 Ideal Image Generator: Input visual token sequence and pixel-level aligned registered multimodal image, and output ideal reference image through conditional Transformer decoder network structure, defect repair guided attention mechanism and preset training strategy; The ideal image generator of this invention is a conditional Transformer decoder network structure that starts with a learnable CLS token and predicts image patch sequences in an autoregressive manner.

[0023] The CLS Token is a vector that is randomly initialized and co-optimized during training. Its dimension is the same as the embedding dimension of the visual token. During training and inference, the CLS Token is placed at the beginning of the decoder input sequence to aggregate global context information and serve as the initial state for the autoregressive generation of image patch sequences.

[0024] The defect repair-guided attention mechanism includes: defect confidence gating generation and attention modulation; The present invention sets the defect confidence gating generation as a lightweight defect confidence prediction subnetwork (consisting of two 1x1 convolutional layers) to generate a single-channel defect confidence gating map. The size of the image is the same as Similarly, the value at each position is between 0 and 1, representing the prior probability that a defect exists at that location.

[0025] Attention modulation specifically involves the following: In each layer of cross-attention in the Transformer decoder, the query is the decoder's own hidden state, and the key and value are the encoder's output. Before calculating attention weights, this invention uses a gating graph. Modulate the attention score. Specifically, for The key corresponding to the position marked as having a high defect probability > 0.5 is given an attention score of 10 before the Softmax calculation. This significantly reduces the model's attention to these "untrustworthy" tokens, forcing the model to rely more on the "healthy prior" it has learned on a large number of flawless samples to generate the content of that region.

[0026] The preset training strategy of this invention includes two training phases: a first phase of pre-training and a second phase of fine-tuning. The first stage of pre-training involves training the generator stream as an autoencoder only on a large number of flawless metal surface samples, minimizing the output ideal reference image. Registered multimodal images aligned to pixel level The L1 loss and perceptual loss between them enable the model to learn to reconstruct realistic normal surfaces.

[0027] The second stage of fine-tuning involves introducing flawed samples and constructing a fine-tuning loss function, calculated as follows: In the formula, To reconstruct the loss function, For an ideal reference image, Flawless, realistic images with manual annotations. To guide losses for defect repair, Preset the weight value for the loss in defect repair guidance; S2.2 Feature Space Difference Module: Based on pixel-level aligned registered multimodal images and ideal reference images, it outputs defect residual feature maps through multi-scale feature pyramid difference layering, cross-modal interaction layer and adversarial defect residual discriminator. Specifically, the following steps are included: S2.2.1 Multi-scale feature pyramid difference layering: Input pixel-level aligned registered multimodal image and ideal reference image, extract feature maps of three scales by using a U-Net encoder with shared weights, calculate gated difference features, and output multi-scale gated difference features. The multi-scale gated differential features include: high-frequency detail features, intermediate features, and global context features; This invention uses a shared-weight U-Net encoder (without a decoder) to process pixel-level aligned registered multimodal images. and ideal reference image Feature extraction was performed to obtain feature maps at three scales. , where scales s∈{1,2,3} represent features downsampled by 8 times, 16 times, and 32 times, respectively; Then, the present invention performs gated difference calculation at each scale s, and the expression for calculating the original difference is: In the formula, For multimodal images registered at three scales, An ideal reference image for three scales; Will and By concatenating along the channel dimension, a spatially adaptive gated weight map is generated through a lightweight gating network (consisting of a 1x1 convolution and a sigmoid activation function). The expression for calculating the gated difference feature is as follows: In the formula, For the original difference, A spatially adaptive gated weight graph; Output multi-scale gated differential features , For high-frequency detail features, It is a medium-level feature. This is a global context feature.

[0028] S2.2.2 Cross-modal interaction layer: Input multi-scale gated differential features, and output the initial defect residual feature map through bidirectional cross attention, feature fusion and convolution compression; This invention inputs multi-scale gated differential features and uses bidirectional cross-attention to focus on intermediate features. As a query High-frequency detail features serve as both keys and values. Through a cross-attention module, intermediate features are integrated to fuse subtle boundary texture differences, resulting in enhanced features. .

[0029] The feature fusion of the present invention will , , Upsampled to the same size with 256 channels, then concatenated along the channel dimension, followed by convolutional compression. A 1x1 convolutional layer is then used to fuse and reduce the dimensionality of the concatenated features, generating an initial defect residual feature map, which is then output. .

[0030] S2.2.3 Adversarial Flaw Residual Discriminator: Input the initial flaw residual feature map, perform adversarial training through a lightweight convolutional discriminator network, and output the flaw residual feature map; The lightweight convolutional discriminator network includes four convolutional layers and one output layer; S2.3, Diffusion-driven segmentation flow: Input defect residual feature map, recover the segmentation map through a conditional denoising U-Net, and then output defect segmentation map through conditional injection mechanism and iterative denoising; This invention employs a conditional denoising U-Net to perform an iterative denoising process. This process starts with a pure random noise map of the same size as the segmentation map, and after T iterations (T=1000), gradually removes noise and recovers the segmentation map. The conditional injection mechanism of this invention is as follows: in each layer of the denoising U-Net, the defect residual feature map The network is injected through the Spatial Adaptive Layer Normalization (SPADE) module. Specifically, the SPADE module uses... After passing through a small convolutional sub-network, scaling coefficients are dynamically generated for the normalized features of this layer in U-Net. and offset coefficient The calculation expression is: The iterative denoising method of this invention specifically involves U-Net using the current noisy segmentation image. Current time step code and conditional defect residual feature map Given the input, predict the added noise. Then, based on the predefined scheduler (DDIM scheduler), the next step of calculating a cleaner segmentation graph is performed. The calculation expression is: In the formula, This is the current noisy segmentation image. Encode the current time step. To predict the added noise, It is random noise. For the first scheduler parameters, For the second scheduler parameters, The parameters are for the third scheduler; when the iteration reaches t=0, the final defect segmentation map is output. .

[0031] S3. Defect detection and localization based on defect segmentation map: Based on defect segmentation map and registered multimodal image data, the method of detecting surface defects of metal products is completed by segmentation map binarization and region extraction, multi-dimensional defect feature extraction and quantification, defect classification and report generation, and outputting the final defect detection report. S3.1 Segmentation image binarization and region extraction: Input the defect segmentation image, perform adaptive threshold binarization using Gaussian weighted average local thresholding method, and perform morphological post-processing and connected component analysis to output a set of candidate defect regions; For each pixel in an image, this invention calculates the weighted average of the pixel probability values ​​within a neighborhood window (a circular window with a radius of 15 pixels). The weights are generated by a Gaussian function. If the probability value of the pixel is higher than (average value - constant C), it is set to 1 (defect); otherwise, it is set to 0 (background). A binary mask is then output. The constant C is an adjustable parameter with a preset value of 0.05; The morphological post-processing of the binary mask in this invention is as follows: a 3x3 rectangle is used first for erosion and then for dilation to eliminate tiny noise points, and then a 5x5 rectangle is used first for dilation and then for erosion to fill the small holes inside the defect area and smooth the defect boundary. This invention applies an 8-connected component labeling algorithm to the post-processed binary mask to identify all interconnected pixel regions. Each independent connected component is initially defined as a candidate defect region. Where k = 1, 2, 3, ..., K, K is the total number of candidate defective regions, and k is the index; output the set of candidate defective regions. S3.2 Multi-dimensional defect feature extraction and quantization: Input the candidate defect region set and the registered multimodal image data, and output a multi-dimensional feature vector through geometric dimension feature calculation, intensity dimension feature calculation and texture dimension feature calculation; This invention addresses each candidate defect region. Quantitative features are calculated using three dimensions: geometric dimension features, intensity dimension features, and texture dimension features. Finally, the multi-dimensional feature vector corresponding to each defective region is output.

[0032] S3.3 Defect Classification and Report Generation: Input a multi-dimensional feature vector, and output the final defect detection report through a predefined defect classification rule base and grading standards; Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By fusing bright field, dark field and three-dimensional point cloud data, the present invention constructs a multimodal image, which effectively overcomes the inherent problems of complex reflection of metal surface and limited information of single imaging mode, and realizes the comprehensive capture of various defects such as fine scratches, low contrast pits and oxide spots, and significantly reduces the false negative rate.

[0033] (2) This invention uses an innovative dual-stream diffusion Transformer model for defect segmentation, transforming the segmentation task into a conditional generation problem. This model achieves strong generalization ability and robustness against noise, illumination changes and unknown defect morphology by dynamically generating ideal reference images, accurately locating differences, and using diffusion models for iterative denoising. It can achieve extremely fine defect boundary localization and effectively suppress over-detection and false detection.

[0034] (3) By constructing a fully automatic, end-to-end intelligent detection solution from multimodal data acquisition to final segmentation and positioning, this invention combines advanced generative AI with discriminative tasks, which not only improves the accuracy of identifying complex defects, but also enhances the system's adaptability to different production lines and different metal products, providing reliable technical support for the digital and intelligent upgrading of industrial quality control. Attached Figure Description

[0035] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a machine vision-based method for detecting surface defects in metal products according to the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example 1 like Figure 1 As shown, a machine vision-based method for detecting surface defects in metal products includes: S1. Multimodal image data acquisition and registration: A multimodal image dataset is constructed by acquiring bright field images, dark field images, and three-dimensional point cloud data of the surface of the tested metal product. The multimodal image dataset is then registered to obtain the registered multimodal image dataset. Specifically, the following steps are included: S1.1 Multimodal image data acquisition: After acquiring bright field images, dark field images and three-dimensional point cloud data of the surface of the tested metal product, the bright field image sequence, dark field image and three-dimensional point cloud data are output through hardware synchronization. The bright-field image acquisition specifically involves using an area array industrial camera with a resolution of no less than 2 million pixels, in conjunction with a ring LED light source, to image the metal product being tested at an angle perpendicular to the surface of the metal product. To overcome the strong reflection of metal surfaces, this invention employs multi-exposure image fusion technology: the camera is controlled to continuously acquire three images at the same shooting point with short exposure time (0.1ms), standard exposure time (1ms), and long exposure time (10ms). Then, a weighted fusion algorithm based on the Laplacian pyramid is used to fuse the dark details that are not overexposed in the short exposure image, the bright details that are not underexposed in the long exposure image, and the midtone information of the standard exposure image to output a bright field image.

[0040] In this invention, bright-field images are mainly used to detect color changes, spots, oxidation, corrosion, etc. The dark field image acquisition is specifically as follows: using an area array industrial camera with a resolution of no less than 2 million pixels, the camera is placed at a low angle of 10° to 30° with the surface being measured, and a strip LED light source is used to illuminate the surface being measured in a grazing manner (the angle between the light direction and the tangent direction of the surface is less than 10°) to output a dark field image.

[0041] In this embodiment, by setting up the camera and light source, any microscopic unevenness will scatter light onto the camera lens, thereby forming a bright defect signal against a dark background.

[0042] The specific acquisition of the three-dimensional point cloud data is as follows: a line laser scanner based on the principle of laser triangulation is used; a blue laser line with a wavelength of 450nm is projected onto the metal surface, and the deformation of the laser line is captured by a camera located at a fixed angle (30° in this embodiment) to the laser; the camera pixel coordinates are converted into three-dimensional world coordinates through laser plane calibration and light stripe center extraction algorithm (Steger algorithm), and the three-dimensional point cloud data is output after scanning line by line.

[0043] The hardware synchronization specifically involves receiving encoder signals via a programmable logic controller (PLC). When the workpiece under test moves to the detection station, the PLC simultaneously sends hardware trigger signals to the bright-field camera, dark-field camera, and line laser scanner to ensure that the three sets of data—bright-field image, dark-field image, and 3D point cloud data—are acquired at the same physical location on the workpiece, ensuring that the images acquired in different modes for the same detection area are completely aligned in time and space.

[0044] This invention performs noise reduction, flattening correction (eliminating uneven illumination), and distortion correction on the acquired bright field images, dark field images, and 3D point cloud data.

[0045] S1.2 Multimodal Data Joint Calibration: Based on bright-field images, dark-field images, and 3D point cloud data, the initial transformation parameter matrix is ​​output through calibration plate preparation, 2D camera calibration, 3D scanner calibration, and joint calibration. The calibration plate is prepared as follows: the substrate of the calibration plate is alumina ceramic, and a checkerboard pattern with a side length of 5mm is formed on the surface by precision etching. At the center of each checkerboard corner, a conical pit with a diameter of 50μm and a depth of 25μm is made using micromachining technology as a three-dimensional feature point.

[0046] The two-dimensional camera calibration specifically involves: placing the calibration board in different positions and orientations within the field of view, and acquiring no fewer than 15 images for both the bright-field and dark-field cameras; identifying the two-dimensional pixel coordinates of the checkerboard corner points, and using the Zhang Zhengyou calibration method to solve for the camera's intrinsic parameter matrix (including focal length, principal point coordinates, and distortion coefficients) and extrinsic parameter matrix (rotation and translation vectors) relative to the calibration board coordinate system at each placement.

[0047] The 3D scanner calibration specifically involves: at the same location, the 3D scanner scans the surface of the calibration plate, and through a 3D point cloud segmentation and fitting algorithm, identifies and locates the 3D spatial center coordinates of each conical pit. The transformation relationship from the 3D scanner coordinate system to the calibration plate coordinate system is then solved using least squares fitting.

[0048] The joint calibration specifically involves using the calibration board coordinate system as a common reference system. This invention defines the transformation from the bright-field camera to the calibration board as follows: The transformation from a 3D scanner to a calibration plate is The initial transformation parameter matrix from the 3D scanner to the bright-field camera is then... The calculation expression is as follows: Similarly, calculate the transformation from a dark field camera to a bright field camera.

[0049] S1.3 Multimodal Feature Matching: Based on bright field images, dark field images, 3D point cloud data and initial transformation parameter matrix, the matching is performed through field of view division and key point detection, hardening depth local feature descriptor extraction, feature matching and mismatch removal, and output of a set of matching feature point pairs. Specifically, the following steps are included: S1.3.1 Field of View Division and Key Point Detection: Based on bright field images, dark field images, and 3D point cloud data, through grid division, image key point detection, and point cloud key point detection, output the 2D key point set of the bright field image, the 2D key point set of the dark field image, and the 3D key point set of the 3D point cloud in each sub-region. The specific mesh division method of this invention involves using a bright-field image as a reference, uniformly dividing its field of view into a 10×10 grid, with each grid defined as a sub-region. Image keypoint detection is then performed. Within each sub-region, the Shi-Tomasi corner detection algorithm is executed in parallel on both the bright-field and dark-field images to extract strong corner points as candidate keypoints, outputting a 2D keypoint set for both the bright-field and dark-field images. Regarding point cloud keypoint detection, in the corresponding 3D point cloud sub-region, the point cloud is converted into a 2D grayscale height map (Z-coordinate mapped to pixel grayscale value) and a 2D average curvature map. Subsequently, the Harris corner detection algorithm is used on this 2D map to extract feature points, and through coordinate mapping relationships, the corresponding 3D spatial coordinates in the 3D point cloud are found, forming 3D point cloud keypoints, resulting in a 3D point cloud keypoint set.

[0050] S1.3.2 Hardening Depth Local Feature Descriptor Extraction: Input the 2D keypoint set of the bright field image, the 2D keypoint set of the dark field image, and the 3D keypoint set of the 3D point cloud. After training through the feature extraction network and loss function calculation, and after L2 normalization, output the 512-dimensional hardening depth local feature descriptor corresponding to each keypoint. The feature extraction network is a VGG16 network structure with 13 convolutional layers and 3 fully connected layers. The hardened deep local feature descriptor is taken from the 3rd convolutional layer of the 4th convolutional block in the VGG16 network structure. This invention defines a feature extraction network with a VGG16 network pre-trained on ImageNet as its backbone, using the output of the conv4_3 convolutional layer as a dense initial depth descriptor. The loss function is defined as follows: In the formula, For triplet loss, for an anchor image patch, the goal is to minimize its distance in feature space from positive samples (image patches at the same physical location under slight illumination changes or deformations) and maximize its distance from negative samples (image patches at different physical locations). For consistency loss, for the same physical keypoint, the cosine similarity of its feature vectors extracted from the bright-field and dark-field images is maximized after passing through a shared projection layer. In this embodiment, the consistency loss weight value is set within a preset range of [0.1, 2.0]. Specifically, within the value range [0.1, 2.0], a search is performed with a step size of 0.1, and finally, 1.0, which makes the model achieve the highest feature matching accuracy on the independent validation set, is selected as the final weight value. Subsequently, online descriptor computation is performed. For each keypoint detected in S1.3.1, the image patch around it is cropped and input into the fine-tuned VGG16 network to extract its conv4_3 features. After L2 normalization, the 512-dimensional hardened depth local feature descriptor corresponding to each keypoint is output.

[0051] S1.3.3 Feature Matching and Mismatch Removal: Based on the hardened depth local feature description operator and the initial transformation parameter matrix, matching is performed between bright field image and dark field image and between image and point cloud, outputting a set of matching feature point pairs; This invention inputs a 512-dimensional hardened depth local feature descriptor corresponding to each key point. First, the bright field image and dark field image are matched. Fast Approximate Nearest Neighbor Search (FLANN) is used to perform bidirectional matching of the descriptors of all sub-regions in the bright field and dark field images. Then, the Random Sample Consensus Algorithm (RANSAC) is used with the Fundamental Matrix as the geometric model to remove mismatched point pairs that do not conform to the epipolar constraints. Secondly, using the initial transformation parameter matrix After projecting the 3D keypoints onto the brightfield image plane, within the neighborhood of each projected point (with a preset radius of 10 pixels), the point with the closest descriptor cosine distance among the 2D keypoints in the brightfield image is found to form a 2D-3D matching pair, resulting in a set of matching feature point pairs.

[0052] S1.4, Pixel-level alignment: Based on the matching feature point pair set, initial transformation parameter matrix, dark field image and 3D point cloud data, the pixel-level aligned registration multimodal image data is output through local nonlinear transformation model calculation and image resampling and multi-resolution feathering fusion. S1.4.1 Calculation of Local Nonlinear Transformation Model: Based on the set of matching feature point pairs, the model is solved by using the moving least squares model and the adaptive regularization loss function, and the local MLS transformation model of all sub-regions is output. This invention is based on a set of matching feature point pairs within a sub-region, and establishes a Moving Least Squares (MLS) deformation model for each sub-region. For each matching point pair, the average curvature of the 3D point cloud within its local neighborhood is calculated; therefore, this invention defines an adaptive regularization weight function, the calculation expression of which is: In the formula, Control points The average curvature at that point, As a preset scale factor, In this embodiment, the preset attenuation rate is used. Set to 0.5. Setting it to 40.0, this parameter set has been verified to be suitable for common metal surfaces (such as steel plates and aluminum) and their common curvature distribution ranges (mean curvature). (Mostly between 0 and 0.1), which can stably produce registration results with low reprojection error (less than 0.3 pixels) and no significant distortion.

[0053] This invention and The value should be adjusted appropriately. Within the range [0.1, 1.0], Adjusting within the range of [20.0, 60.0] can achieve better registration results than the globally fixed regularization method; This invention uses all matching point pairs within a sub-region as control points, aims to minimize reprojection error, and combines adaptive regularization weight function calculation to solve for the parameters of the MLS transform function of that sub-region, outputting the local MLS transform model for all sub-regions. S1.4.2 Image resampling and multi-resolution feathering fusion: Based on the local MLS transform model of all sub-regions, dark field images and 3D point cloud data are resampled and feathered to output pixel-level aligned registered multimodal image data. The resampling includes dark field image resampling and 3D point cloud data resampling; The pixel-level aligned registration multimodal image data includes: bright field image, resampled dark field image, two-dimensional height map, and two-dimensional intensity map; The dark field image resampling method of the present invention is as follows: for each target pixel in the bright field image coordinate system, the source coordinates corresponding to it in the original dark field image are calculated using the local MLS transformation model of the sub-region where it is located; the pixel value is obtained from the original dark field image using bilinear interpolation and filled into the target position.

[0054] The specific method for resampling 3D point cloud data in this invention is as follows: First, an initial transformation parameter matrix is ​​used. Coarse alignment is performed; then, for each 3D point, a displacement correction is calculated using the local MLS transform model of the sub-region where it is projected onto the brightfield image, and fine correction is performed on the 3D point. Finally, the corrected 3D point cloud is projected onto the brightfield image plane to generate a 2D height map and a 2D intensity map that correspond one-to-one with the pixels of the brightfield image. Finally, a feathering fusion operation is performed, specifically: the overlapping area of ​​the resampling results of adjacent sub-regions is decomposed into three layers of Gaussian pyramids. On each layer of the Laplacian pyramid, linear weighted fusion is performed based on the distance of the pixel to the boundary of the two sub-regions (the closer to the boundary of sub-region A, the greater the weight; and vice versa). The fused Laplacian pyramids are then reconstructed using Gaussian pyramids to generate the final seamless registered image, outputting pixel-level aligned registered multimodal image data (bright field image, resampled dark field image, 2D height map, and 2D intensity map).

[0055] S2. Pixel-level defect segmentation using a dual-stream diffusion Transformer model: Based on pixel-level aligned multimodal image data, pixel-level defect segmentation is performed using a dual-stream diffusion Transformer model, and a defect segmentation map is output. The dual-stream diffusion Transformer model includes: an ideal reference image generation stream, a feature space difference module, and a diffusion-driven segmentation stream; Specifically, the following steps are included: S2.1 Ideal Reference Image Generation Stream: Input pixel-level aligned registered multimodal image data, and output an ideal reference image through a defect-aware editor and an ideal image generator; S2.1.1, Defect-Aware Editor: Input pixel-level aligned registered multimodal image data, and output a visual token sequence through modality-specific extraction, multimodal feature fusion and tokenization; This invention employs three convolutional coding subnetworks with identical structures but non-shared weights to extract modality-specific features, processing pixel-level aligned registered multimodal images. The network consists of bright field image channels, dark field image channels, and 3D derived (height map / intensity map) channels. Each sub-network comprises four convolutional blocks, each containing a 3x3 convolutional layer, a batch normalization layer, and a ReLU activation function. Convolutions with stride=2 are used for downsampling; ultimately, each sub-network outputs a 256-channel feature map. This invention forces each sub-network to focus on learning the unique feature representation of its corresponding imaging modality by enforcing non-sharing of weights: the bright-field sub-network focuses on color and macroscopic texture variations; the dark-field sub-network is highly sensitive to microscopic geometric undulations; and the three-dimensional sub-network directly captures the physical height information of the surface morphology.

[0056] The weights of the three sub-networks are independently updated and optimized through backpropagation during the end-to-end training of the dual-stream diffusion Transformer model. Specifically, during model training, the independent weight parameters of the three sub-networks are calculated and updated respectively by minimizing the overall objective function and using gradient descent.

[0057] The multimodal feature fusion and tokenization of this invention specifically involves concatenating the feature maps output from the three sub-networks along the channel dimension to obtain a 768-channel feature map. Subsequently, a fusion module incorporating 1x1 convolutions and channel attention mechanisms reduces the dimensionality of the channels and fuses them, ultimately outputting a 256-channel multimodal fused feature map. Multimodal feature map fusion Flattening the spatial dimensions yields feature vectors, which are used as visual token sequences, resulting in the output visual token sequence. .

[0058] S2.1.2 Ideal Image Generator: Input visual token sequence and pixel-level aligned registered multimodal image, and output ideal reference image through conditional Transformer decoder network structure, defect repair guided attention mechanism and preset training strategy; The ideal image generator of this invention is a conditional Transformer decoder network structure that starts with a learnable CLS token and predicts image patch sequences in an autoregressive manner.

[0059] The CLS Token is a vector that is randomly initialized and co-optimized during training. Its dimension is the same as the embedding dimension of the visual token. During training and inference, the CLS Token is placed at the beginning of the decoder input sequence to aggregate global context information and serve as the initial state for the autoregressive generation of image patch sequences.

[0060] The defect repair-guided attention mechanism includes: defect confidence gating generation and attention modulation; The present invention sets the defect confidence gating generation as a lightweight defect confidence prediction subnetwork (consisting of two 1x1 convolutional layers) to generate a single-channel defect confidence gating map. The size of the image is the same as Similarly, the value at each position is between 0 and 1, representing the prior probability that a defect exists at that location.

[0061] Attention modulation specifically involves the following: In each layer of cross-attention in the Transformer decoder, the query is the decoder's own hidden state, and the key and value are the encoder's output. Before calculating attention weights, this invention uses a gating graph. Modulate the attention score. Specifically, for The Key corresponding to the position marked as having a high defect probability > 0.5 is given an attention score before Softmax calculation by subtracting a large constant (set to 10 in this embodiment), thereby significantly reducing the model's attention to these "untrustworthy" Tokens and forcing the model to rely more on the "healthy prior" it has learned on a large number of flawless samples to generate the content of that region.

[0062] The preset training strategy of this invention includes two training phases: a first phase of pre-training and a second phase of fine-tuning. The first stage of pre-training involves training the generator stream as an autoencoder only on a large number of flawless metal surface samples, minimizing the output ideal reference image. Registered multimodal images aligned to pixel level The L1 loss and perceptual loss between them enable the model to learn to reconstruct realistic normal surfaces.

[0063] The second stage of fine-tuning involves introducing flawed samples and constructing a fine-tuning loss function, calculated as follows: In the formula, To reconstruct the loss function, For an ideal reference image, Flawless, realistic images with manual annotations. To guide losses for defect repair, To preset the defect repair guidance loss weight value, this embodiment sets it to 0.1, within the range of [0.01, 1.0]. Adjustments can be made; if the model's repair of defective areas is too conservative (i.e., insufficient), the size can be appropriately increased. To enhance the strength of the guiding signal; if the repair causes image distortion or introduces artifacts, the intensity can be appropriately reduced. ; S2.2 Feature Space Difference Module: Based on pixel-level aligned registered multimodal images and ideal reference images, it outputs defect residual feature maps through multi-scale feature pyramid difference layering, cross-modal interaction layer and adversarial defect residual discriminator. Specifically, the following steps are included: S2.2.1 Multi-scale feature pyramid difference layering: Input pixel-level aligned registered multimodal image and ideal reference image, extract feature maps of three scales by using a U-Net encoder with shared weights, calculate gated difference features, and output multi-scale gated difference features. The multi-scale gated differential features include: high-frequency detail features, intermediate features, and global context features; This invention uses a shared-weight U-Net encoder (without a decoder) to process pixel-level aligned registered multimodal images. and ideal reference image Feature extraction was performed to obtain feature maps at three scales. , where scales s∈{1,2,3} represent features downsampled by 8 times, 16 times, and 32 times, respectively; Then, the present invention performs gated difference calculation at each scale s, and the expression for calculating the original difference is: In the formula, For multimodal images registered at three scales, An ideal reference image for three scales; Will and By concatenating along the channel dimension, a spatially adaptive gated weight map is generated through a lightweight gating network (consisting of a 1x1 convolution and a sigmoid activation function). The expression for calculating the gated difference feature is as follows: In the formula, For the original difference, A spatially adaptive gated weight graph; Output multi-scale gated differential features , For high-frequency detail features, It is a medium-level feature. This is a global context feature.

[0064] S2.2.2 Cross-modal interaction layer: Input multi-scale gated differential features, and output the initial defect residual feature map through bidirectional cross attention, feature fusion and convolution compression; This invention inputs multi-scale gated differential features and uses bidirectional cross-attention to focus on intermediate features. As a query High-frequency detail features serve as both keys and values. Through a cross-attention module, intermediate features are integrated to fuse subtle boundary texture differences, resulting in enhanced features. .

[0065] The feature fusion of the present invention will , , Upsampled to the same size with 256 channels, then concatenated along the channel dimension, followed by convolutional compression. A 1x1 convolutional layer is then used to fuse and reduce the dimensionality of the concatenated features, generating an initial defect residual feature map, which is then output. .

[0066] S2.2.3 Adversarial Flaw Residual Discriminator: Input the initial flaw residual feature map, perform adversarial training through a lightweight convolutional discriminator network, and output the flaw residual feature map; The lightweight convolutional discriminator network includes four convolutional layers and one output layer; S2.3, Diffusion-driven segmentation flow: Input defect residual feature map, recover the segmentation map through a conditional denoising U-Net, and then output defect segmentation map through conditional injection mechanism and iterative denoising; This invention employs a conditional denoising U-Net to perform an iterative denoising process. This process starts with a pure random noise map of the same size as the segmentation map, and after T iterations (T=1000), gradually removes noise and recovers the segmentation map. The conditional injection mechanism of this invention is as follows: in each layer of the denoising U-Net, the defect residual feature map The network is injected through the Spatial Adaptive Layer Normalization (SPADE) module. Specifically, the SPADE module uses... After passing through a small convolutional sub-network, scaling coefficients are dynamically generated for the normalized features of this layer in U-Net. and offset coefficient The calculation expression is: The iterative denoising method of this invention specifically involves U-Net using the current noisy segmentation image. Current time step code and conditional defect residual feature map Given the input, predict the added noise. Then, based on the predefined scheduler (DDIM scheduler in this embodiment), the next step of calculating a cleaner segmentation graph is performed. The calculation expression is: In the formula, This is the current noisy segmentation image. Encode the current time step. To predict the added noise, It is random noise. For the first scheduler parameters, For the second scheduler parameters, The parameters are for the third scheduler; when the iteration reaches t=0, the final defect segmentation map is output. .

[0067] S3. Defect detection and localization based on defect segmentation map: Based on defect segmentation map and registered multimodal image data, the method of detecting surface defects of metal products is completed by segmentation map binarization and region extraction, multi-dimensional defect feature extraction and quantification, defect classification and report generation, and outputting the final defect detection report. S3.1 Segmentation image binarization and region extraction: Input the defect segmentation image, perform adaptive threshold binarization using Gaussian weighted average local thresholding method, and perform morphological post-processing and connected component analysis to output a set of candidate defect regions; For each pixel in an image, this invention calculates the weighted average of the pixel probability values ​​within a neighborhood window (a circular window with a radius of 15 pixels). The weights are generated by a Gaussian function. If the probability value of the pixel is higher than (average value - constant C), it is set to 1 (defect); otherwise, it is set to 0 (background). A binary mask is then output. The constant C is an adjustable parameter with a preset value of 0.05; The morphological post-processing of the binary mask in this invention is as follows: a 3x3 rectangle is used first for erosion and then for dilation to eliminate tiny noise points, and then a 5x5 rectangle is used first for dilation and then for erosion to fill the small holes inside the defect area and smooth the defect boundary. This invention applies an 8-connected component labeling algorithm to the post-processed binary mask to identify all interconnected pixel regions. Each independent connected component is initially defined as a candidate defect region. Where k = 1, 2, 3, ..., K, K is the total number of candidate defective regions, and k is the index; output the set of candidate defective regions. S3.2 Multi-dimensional defect feature extraction and quantization: Input the candidate defect region set and the registered multimodal image data, and output a multi-dimensional feature vector through geometric dimension feature calculation, intensity dimension feature calculation and texture dimension feature calculation; This invention addresses each candidate defect region. Quantitative features are calculated using three dimensions: geometric dimension features, intensity dimension features, and texture dimension features. Geometric dimension feature calculation: based on Given the pixel coordinates, calculate the area, perimeter, minimum bounding matrix, aspect ratio, and circularity.

[0068] Intensity dimension feature calculation: based on For the corresponding pixel values ​​in the multimodal dataset, calculate the mean, standard deviation, and maximum value of the pixel values ​​in the bright field image, dark field image, and intensity map, respectively; and calculate the mean, minimum, and peak-to-valley difference (maximum value - minimum value) of the height in the height map to quantify the degree of convexity.

[0069] Texture dimension feature calculation: based on The outer ring region is calculated on the brightfield image. The gray-level co-occurrence matrix of the surrounding normal background region is used to extract contrast and energy features from the two GLCMs respectively, and the texture anomalousness is calculated. Finally, the multi-dimensional feature vector corresponding to each defective region is output.

[0070] S3.3 Defect Classification and Report Generation: Input a multi-dimensional feature vector, and output the final defect detection report through a predefined defect classification rule base and grading standards; In this embodiment, the predefined defect classification rule base is: Scratches: If (aspect ratio > 5) AND (dark field intensity > 200) THEN, classify as scratches; Pits: If (area > 10) AND (average depth < -0.1mm) THEN, then classify as pits; Oxidation: IF (texture contrast > 2.0) AND (area > 15) THEN Classified as oxidation; The rating criteria in this embodiment are as follows: Scratch severity: <2mm in length → minor; 2-5mm → moderate; >5mm → severe; Pitting severity: <1mm² → Slight; 1-4mm² → Moderate; >4mm² → Severe; Oxidation severity: <2 mm² → Slight; 2-8 mm² → Moderate; >8 mm² → Severe; Finally, a structured inspection report is output, which includes, but is not limited to: the overall quality judgment of the sample (pass / fail), the precise pixel-level coordinates and bounding boxes of each defect, the specific defect type (such as scratches, pits, oxidation, etc.), the quantified dimensions (such as area, length), and the rule-based severity level, thereby fully realizing the automated detection, location and classification of surface defects of the inspected metal products.

[0071] Example 2 This invention also provides a machine vision-based surface defect detection system for metal products, and a method for implementing this system, comprising: Data registration module: used to acquire bright field images, dark field images and 3D point cloud data of the surface of the tested metal product, construct multimodal image data, and register the multimodal image data; The defect segmentation module is used to perform pixel-level defect segmentation on registered multimodal image data using a dual-stream diffusion Transformer model to obtain a defect segmentation map. Defect Detection Module: Based on the defect segmentation map, it is used to detect and locate surface defects of the tested metal products and output the final defect detection report.

[0072] A further implementation method includes a data registration module comprising: Camera calibration unit: Loads and uses a calibration board image that combines two-dimensional checkerboard and three-dimensional microscopic feature points, executes Zhang Zhengyou calibration method, calculates and stores the intrinsic parameters of each sensor and the extrinsic parameter matrix relative to the bright field camera; Feature matching module: Divides the field of view into a grid, performs hardened depth local feature descriptor extraction and matching in parallel within each sub-region (based on pre-trained VGG16 skeleton), and uses the RANSAC algorithm to remove mismatches; Image resampling unit: Based on matching point pairs, it calculates the moving least squares (MLS) transformation model for each sub-region and resamples the dark field image and 3D point cloud data to the bright field image coordinate system to achieve pixel-level alignment; A further implementation method is that, in the defect segmentation module, the structure of the dual-flow diffusion Transformer model includes: Ideal reference image generation stream: used to generate ideal reference images from registration-based multimodal image data; Feature space difference module: used to calculate the difference between the registered multimodal image data and the ideal reference image, and generate defect residual feature maps; Diffusion-driven segmentation flow: Using a denoising U-Net, an iterative denoising process controlled by the DDIM scheduler is executed, conditioned on the defect residual feature map, to recover a clean defect segmentation map from random noise.

[0073] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for detecting surface flaws of a metal product based on machine vision, characterized by, The method comprises the following steps: S1, multi-modal image data acquisition and registration: through bright field image acquisition, dark field image acquisition and three-dimensional point cloud data acquisition of the surface of the measured metal product, a multi-modal image data set is constructed, and after registration of the multi-modal image data set, a registered multi-modal image data set is obtained; S2, pixel-level defect segmentation by using a double-flow diffusion Transformer model: based on the pixel-level registration multi-modal image data, pixel-level defect segmentation is performed by using a double-flow diffusion Transformer model, and a defect segmentation map is output; S3, defect detection and positioning based on the defect segmentation map: based on the defect segmentation map and the registered multi-modal image data, through binaryzation and region extraction of the segmentation map, multi-dimensional defect feature extraction and quantization, defect classification and report generation, a final defect detection report is output, and the metal product surface defect detection method is completed.

2. The method according to claim 1, wherein In S1, multi-modal image data acquisition and registration, the following steps are specifically included: S1.1, multi-modal image data acquisition: after bright field image acquisition, dark field image acquisition and three-dimensional point cloud data acquisition of the surface of the measured metal product, through hardware synchronization, a time and space aligned bright field image sequence, a dark field image and three-dimensional point cloud data are output; The bright field image acquisition specifically comprises: using a surface array industrial camera with a resolution of not less than 2 million pixels, cooperating with a ring-shaped LED light source, and imaging at an angle perpendicular to the surface of the measured metal product; The dark field image acquisition specifically comprises: using a surface array industrial camera with a resolution of not less than 2 million pixels, placing the camera at a low angle position of 10° to 30° with the measured surface, using a strip-shaped LED light source to irradiate the measured surface at an angle of less than 10° between the light direction and the tangent direction of the surface, and outputting a dark field image; The three-dimensional point cloud data acquisition specifically comprises: using a line laser scanner based on the laser triangulation principle, projecting a blue laser line with a wavelength of 450 nm onto the metal surface, and capturing the deformation of the laser line through a camera and a laser at a fixed angle of 30°; S1.2, multi-modal data joint calibration: based on the bright field image, the dark field image and the three-dimensional point cloud data, through calibration plate preparation, two-dimensional camera calibration, three-dimensional scanner calibration and joint calibration, an initial transformation parameter matrix is output; The calibration plate preparation specifically comprises: the calibration plate substrate is an alumina ceramic, and the surface is a chessboard pattern with a side length of 5 mm; The two-dimensional camera calibration specifically comprises: placing the calibration plate in different positions and postures in the field of view, and the bright field and dark field cameras respectively acquire not less than 15 images; by identifying the two-dimensional pixel coordinates of the chessboard corner points, the intrinsic matrix of the camera and the extrinsic matrix relative to the coordinate system of the calibration plate in each placement are solved by using Zhang Zhengyou's calibration method; The three-dimensional scanner calibration specifically comprises: under the same position, the three-dimensional scanner scans the surface of the calibration plate; The joint calibration is specifically: taking the calibration plate coordinate system as a common reference system, the application sets the transformation of the bright field camera to the calibration plate as , the transformation of the three-dimensional scanner to the calibration plate as , and the calculation expression of the initial transformation parameter matrix of the three-dimensional scanner to the bright field camera is as follows: S1.3, multi-modal feature matching: based on the bright field image, the dark field image, the three-dimensional point cloud data and the initial transformation parameter matrix, through field division and key point detection, hardened depth local feature descriptor extraction, feature matching and false match elimination, a matching feature point pair set is output; S1.4, pixel-level alignment: based on the matching feature point pair set, the initial transformation parameter matrix, the dark field image and the three-dimensional point cloud data, the pixel-level alignment of the registration multi-modal image data is outputted by local nonlinear transformation model calculation, image resampling and multi-resolution feathering fusion.

3. The method according to claim 2, wherein In S1.3, the multi-modal feature matching specifically includes the following steps: S1.3.1, field of view division and key point detection: based on the bright field image, the dark field image and the three-dimensional point cloud data, the bright field image 2D key point set, the dark field image 2D key point set and the three-dimensional point cloud 3D key point set in each sub-region are outputted by grid division, image key point detection and point cloud key point detection; S1.3.2, hard depth local feature descriptor extraction: after the input of the bright field image 2D key point set, the dark field image 2D key point set and the three-dimensional point cloud 3D key point set is trained by the feature extraction network and the loss function calculation, the 512-dimensional hard depth local feature descriptor corresponding to each key point is outputted after L2 normalization; The feature extraction network is a VGG16 network structure with 13 convolutional layers and 3 fully connected layers; The hard depth local feature descriptor is outputted by the third convolutional layer of the fourth convolutional block in the VGG16 network structure; The calculation expression of the loss function is as follows: In the formula, is a triplet loss, is a consistency loss, is a preset consistency loss weight value; S1.3.3, feature matching and false matching elimination: based on the hard depth local feature descriptor and the initial transformation parameter matrix, the matching feature point pair set is outputted by bright field image-dark field image matching and image-point cloud matching.

4. The method of claim 2, wherein the method further comprises: In S1.4, the following steps are specifically included: S1.4.1, local nonlinear transformation model calculation: based on the matching feature point pair set, the local MLS transformation model of all sub-regions is outputted by solving the model through the moving least squares model and the adaptive regularization loss function calculation; The adaptive regularization weight function has the calculation expression as follows: In the formula, is the average curvature at the control point , is a preset scale factor, is a preset decay rate; S1.4.2, image resampling and multi-resolution feathering fusion: based on the local MLS transformation model of all sub-regions, the dark field image and the three-dimensional point cloud data, the pixel-level alignment of the registration multi-modal image data is outputted by resampling and feathering fusion; The resampling includes dark field image resampling and three-dimensional point cloud data resampling; The pixel-level alignment of the registration multi-modal image data includes the bright field image, the resampled dark field image, the two-dimensional height map and the two-dimensional intensity map.

5. The method of claim 1, wherein the method further comprises: In S2, the double-flow diffusion Transformer model is used for pixel-level defect segmentation, and the following steps are specifically included: S2.1, ideal reference image generation flow: inputting the pixel-level alignment of the registration multi-modal image data, the ideal reference image is outputted by the defect perception editor and the ideal image generator; S2.2, feature space difference module: based on the pixel-level alignment of the registration multi-modal image and the ideal reference image, the defect residual feature map is outputted by the multi-scale feature pyramid difference layer, the cross-modal interaction layer and the adversarial defect residual discriminator. S2.3, diffusion driven segmentation flow: input the defect residual feature map, restore the segmentation map through a conditional denoising U-Net, and output the defect segmentation map through the conditional injection mechanism and iterative denoising.

6. The method of claim 5, wherein the method further comprises: In the S2.1, the ideal reference image generation flow specifically comprises the following steps: S2.1.1, defect perception editor: input the pixel-level aligned registered multi-modal image data, output the visual Token sequence through the modal-specific extraction, multi-modal feature fusion and Tokenization; The modal-specific feature extraction comprises: three convolutional coding subnetworks with the same structure but not sharing the weights of the three subnetworks; each subnetwork is composed of four convolutional blocks, and each convolutional block comprises a 3x3 convolutional layer, a batch normalization layer and a ReLU activation function; S2.1.2, ideal image generator: input the visual Token sequence and the pixel-level aligned registered multi-modal image, output the ideal reference image through the conditional Transformer decoder network structure, the defect repair guided attention mechanism and the preset training strategy; The defect repair guided attention mechanism comprises: defect confidence gating and attention modulation; The flaw confidence gate is specifically a lightweight flaw confidence prediction subnetwork formed by two 1x1 convolutional layers to generate a single-channel flaw confidence gate map ; The attention modulation specifically refers to: […]. The key corresponding to the position marked as having a high defect probability > 0.5 is given an attention score minus a constant of 10 before Softmax calculation; The training strategy comprises two training stages, namely a first stage pre-training and a second stage fine-tuning.

7. The method of claim 5, wherein the method further comprises: In the S2.2, the feature space difference module specifically comprises the following steps: S2.2.1, multi-scale feature pyramid difference layer: input the pixel-level aligned registered multi-modal image and the ideal reference image, calculate the gated difference feature after extracting the feature maps of three scales by using a U-Net encoder sharing the weights, and output the multi-scale gated difference feature; The multi-scale gated difference feature comprises: high-frequency detail feature, intermediate feature and global context feature; The expression of the gated difference feature is: wherein is the original difference, is a spatially adaptive gating weight map; S2.2.2, cross-modal interaction layer: input the multi-scale gated difference feature, output the initial defect residual feature map through the bidirectional cross-attention, feature fusion and convolution compression; S2.2.3, adversarial defect residual discriminator: input the initial defect residual feature map, output the defect residual feature map through the adversarial training of the lightweight convolutional discriminator network; The lightweight convolutional discriminator network comprises four convolutional layers and one output layer.

8. The method of claim 1, wherein the method further comprises: In the S3, the segmentation map binarization and region extraction specifically comprises the following steps: S3.1, segmentation map binarization and region extraction: input the defect segmentation map, perform adaptive threshold binarization by using the Gaussian weighted average local threshold method, and perform morphological post-processing and connected component analysis, to output a candidate defect region set; S3.2, multi-dimensional defect feature extraction and quantization: input the candidate defect region set and the registered multi-modal image data, output the multi-dimensional feature vector through the geometric dimension feature calculation, intensity dimension feature calculation and texture dimension feature calculation; S3.3, defect classification and report generation: input the multi-dimensional feature vector, output the final defect detection report through the pre-defined defect classification rule library and the grade evaluation standard.

9. The method of claim 6, wherein the method further comprises: In the S2.1.2, the second stage fine-tuning introduces the defect-containing sample, constructs a fine-tuning loss function, and the calculation expression is as follows: In the formula, is a reconstruction loss function, is an ideal reference image, is a manually labeled flawless real image, is a flaw repair guide loss, is a preset flaw repair guide loss weight value.

10. A machine vision based metal product surface flaw detection system for implementing the method of any one of claims 1-9, characterized by, The application relates to a metal product surface defect detection method and device. The method comprises the following steps: A data registration module is used to collect bright-field images, dark-field images and three-dimensional point cloud data of a measured metal product surface, construct multi-modal image data, and register the multi-modal image data; An imperfection segmentation module is used to adopt a double-flow diffusion Transformer model to perform pixel-level imperfection segmentation on the registered multi-modal image data, and obtain an imperfection segmentation image; An imperfection detection module is used to complete detection and positioning of the surface imperfections of the measured metal product based on the imperfection segmentation image, and output a final imperfection detection report. The data registration module comprises a camera calibration unit, a feature matching module and an image resampling unit. The double-flow diffusion Transformer model comprises an ideal reference image generation flow, a feature space difference module and a diffusion-driven segmentation flow.

Images