Metal product surface defect detection method and system based on light field imaging

Abstract

The invention discloses a metal product surface defect detection method and system based on light field imaging. The detection method comprises the following steps: acquiring a light field image; based on the light field image, focus stack image features and full-focus image features are acquired; based on the focus stack image feature and the full-focus image feature, obtaining a residual feature outside a focal plane; acquiring defect features based on the residual features outside the focal plane; fusing the defect features and the full-focus image features to obtain comprehensive features; based on the comprehensive features, a defect prediction map is obtained, and the defect prediction map is used for defect positioning and detection. According to the method, the focusing stack and the full-focusing image are effectively processed, so that high-precision detection on the surface defects of the metal product is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial image detection, and in particular relates to a method and system for detecting surface defects of metal products based on light field imaging. Background Art

[0002] The surface quality of metal products (such as automotive parts, mobile phone mid-frames, and precision machinery parts) directly impacts their mechanical properties, corrosion resistance, and service life. However, during the manufacturing process, metal surfaces may develop various defects, such as scratches, cracks, and pores, due to processing techniques or environmental factors. Traditionally, surface defects in metal products rely on observation by experienced personnel using microscopes or magnifying glasses, which is inefficient and highly subjective. Automated inspection based on machine vision typically relies on two-dimensional images, which are susceptible to interference from metal surface reflections and complex textures, resulting in a high rate of false detection. Summary of the Invention

[0003] To solve the above technical problems, the present invention proposes a metal product surface defect detection method and system based on light field imaging, which realizes high-precision detection of metal product surface defects by effectively processing the focus stack and the full focus image.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A method for detecting surface defects of metal products based on light field imaging, comprising:

[0006] Acquire a light field image;

[0007] Based on the light field image, acquiring focus stack image features and all-focus image features;

[0008] acquiring residual features outside the focal plane based on the focused stack image features and the all-focused image features;

[0009] Obtaining defect features based on the residual features outside the focal plane;

[0010] Fusing the defect feature with the all-focus image feature to obtain a comprehensive feature;

[0011] Based on the comprehensive features, a defect prediction map is obtained, wherein the defect prediction map is used for defect location and detection.

[0012] Optionally, based on the focused stack image features and the all-focused image features, residual features outside the focal plane are obtained:

[0013] A focus stack in the light field image is subtracted from the all-focus image of the light field image to obtain residual features outside the focal plane.

[0014] Optionally, obtaining defect features based on the residual features outside the focal plane includes:

[0015] Performing Fourier transform on the residual features outside the focal plane to obtain frequency domain features;

[0016] Performing wavelet transform and texture feature extraction on the residual features outside the focal plane to obtain wavelet features and texture features;

[0017] generating a spatial attention map based on the wavelet features and the texture features;

[0018] Based on the spatial attention map and in combination with the frequency domain features, a dynamic frequency mask is obtained;

[0019] The defect feature is obtained based on the dynamic frequency mask.

[0020] Optionally, obtaining the defect feature based on the dynamic frequency mask includes:

[0021] Based on the dynamic frequency mask, obtaining high-frequency features and low-frequency features;

[0022] Frequency weighted convolution and edge enhancement fusion processing are performed on the high-frequency features and the low-frequency features to obtain the defect features.

[0023] Optionally, fusing the defect feature with the all-focus image feature to obtain a comprehensive feature includes:

[0024] Processing the defect features and the all-focus image features using an encoder to obtain multi-scale features;

[0025] Constructing a graph structure of the multi-scale features using a graph convolutional network;

[0026] Fusion is performed based on the graph structure to obtain the comprehensive features.

[0027] Optionally, performing fusion based on the graph structure to obtain the comprehensive features includes:

[0028] Performing feature updating on the graph structure to obtain a first feature;

[0029] Perform convolution and attention enhancement on the first feature to obtain the second feature;

[0030] Performing dimensionality reduction and upsampling on the second feature to obtain a third feature;

[0031] Aggregating the second features to obtain aggregated features;

[0032] The comprehensive feature is obtained based on the aggregated feature and the third feature.

[0033] Optionally, obtaining the comprehensive feature based on the aggregate feature and the third feature includes:

[0034] shallowly fusing the third feature to obtain shallow features;

[0035] Deeply integrating the third feature to obtain a deep feature;

[0036] Fusing the deep features and shallow features to obtain initial comprehensive features;

[0037] The initial comprehensive features and the aggregated features are fused to obtain the comprehensive features.

[0038] Optionally, obtaining a defect prediction map based on the comprehensive features includes:

[0039] Performing convolution mapping on the comprehensive features to obtain a single-channel feature map;

[0040] The single-channel feature map is up-sampled to the target resolution by bilinear interpolation to obtain the defect prediction map.

[0041] Optionally, obtaining a defect prediction map based on the comprehensive features includes:

[0042] Performing convolution mapping on the comprehensive features to obtain a single-channel feature map;

[0043] The single-channel feature map is up-sampled to the target resolution by bilinear interpolation to obtain the defect prediction map.

[0044] The present invention also provides a metal product surface defect detection system based on light field imaging, comprising: an image acquisition module, an image processing module and a defect detection module;

[0045] The acquisition module is used to acquire light field images of the surface of the metal product;

[0046] The image processing module is configured to obtain focus stack image features and all-focus image features based on the light field image; obtain residual features outside the focal plane based on the focus stack image features and all-focus image features; obtain defect features based on the residual features outside the focal plane; and fuse the defect features with the all-focus image features to obtain comprehensive features;

[0047] The defect detection module is used to obtain a defect prediction map based on the comprehensive features, wherein the defect prediction map is used for defect location and detection.

[0048] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a method for detecting surface defects of metal products is implemented.

[0049] Compared with the prior art, the present invention has the following advantages and technical effects:

[0050] 1. High-Precision Inspection: By subtracting the focused stack from the fully focused image, the feature representation of minute defects (such as scratches, pits, and cracks) is effectively enhanced. Compared to traditional RGB image inspection methods, this method can capture micron-level defects and improve inspection accuracy, making it particularly suitable for industrial scenarios with complex textures and lighting conditions.

[0051] 2. Enhanced frequency domain features: Through frequency domain analysis, high-frequency and low-frequency features are decomposed. High-frequency features capture the edges and details of defects, while low-frequency features preserve the overall background information and enhance the salient features of the defect area (such as edges and texture changes). This mechanism effectively suppresses background noise interference and improves the model's robustness in non-uniform lighting and complex surface backgrounds.

[0052] 3. Adaptability to complex environments: This method can effectively suppress background noise, highlight defect areas, adapt to complex industrial environments, and improve the robustness of detection.

[0053] 4. Broad application prospects: This method has broad application prospects in the fields of metal product quality control, industrial manufacturing, etc. It can significantly improve production efficiency and product quality, and provide strong technical support and guarantee for industrial manufacturing enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0055] Figure 1 This is a flow chart of a method for detecting surface defects of metal products based on light field imaging according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0057] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0058] This embodiment proposes a method for detecting surface defects of metal products based on light field imaging. Figure 1 As shown, the specific steps include:

[0059] Acquire a light field image;

[0060] Based on the light field image, the focus stack image features and the all-focus image features are obtained;

[0061] Obtaining residual features outside the focal plane based on the focus stack image features and the all-focus image features;

[0062] Obtain defect features based on residual features outside the focal plane;

[0063] Fuse defect features and all-focus image features to obtain comprehensive features;

[0064] Based on the comprehensive features, a defect prediction map is obtained, wherein the defect prediction map is used for defect location and detection.

[0065] Specifically, the metal product surface defect detection method includes: S1 fully focused image minus focused stack: The input focused stack and fully focused image are processed, and the focused stack is subtracted from the fully focused image to obtain residual features outside the focal plane. These features highlight the differences between different focal planes and the overall clear image, emphasizing the unique depth and spatial information of defects, providing a basis for subsequent analysis.

[0066] S2 frequency domain decomposition: Based on the residual features outside the focal plane obtained in the previous step, frequency domain analysis is performed to decompose them into high-frequency and low-frequency features. High-frequency features capture the edges and details of the defect, while low-frequency features retain the overall background information. The attention mechanism then highlights the defect area, resulting in clearer defect features.

[0067] S3 Feature Fusion: Based on the defect features obtained in the previous step, they are fused with the features of the fully focused image at multiple levels. This process constructs a relationship network between the residual features outside the focal plane and the fully focused image features, analyzes the connections between the features, and then integrates this information to generate comprehensive features. These comprehensive features combine the depth information of the light field image with the color and texture information of the fully focused image, making them more suitable for defect detection in complex scenes.

[0068] S4 Prediction Map Generation: Based on the comprehensive features obtained in the previous step, they are mapped and magnified to generate a high-resolution defect prediction map. This map clearly shows the location and shape of the defects and can be used to accurately locate and identify defects on the metal surface.

[0069] Furthermore, based on the focus stack image features and the all-focus image features, obtaining the residual features outside the focal plane includes:

[0070] The focus stack in the light field image is subtracted from the fully focused image of the light field image to obtain the residual features out of focus.

[0071] Specifically, the input data includes focus stack Ifocal_stack ∈R B×C×H×W (where B is the batch size, C is the number of channels, and H×W is the spatial resolution) and the fully focused image I all_focus ∈R B×C×H×W The focus stack contains multiple sub-field images at different focal planes, and the fully focused image is a clear image integrated from all focal planes.

[0072] Subtract the focused stack from the fully focused image:

[0073] I sub =I all_focus -I focal_stack

[0074] Among them, I sub The residual features outside the focal plane after subtraction. This operation obtains the differences between each sub-field of view by subtracting the fully focused image from the focused stack, highlighting the unique information of the defect in depth and space, and providing input for subsequent frequency domain decomposition.

[0075] Furthermore, based on the residual features outside the focal plane, defect features are obtained including:

[0076] Perform Fourier transform on the residual features outside the focal plane to obtain frequency domain features;

[0077] Perform wavelet transform and texture feature extraction on the residual features outside the focus plane to obtain wavelet features and texture features;

[0078] Generate spatial attention map based on wavelet features and texture features;

[0079] Based on the spatial attention map and combined with frequency domain features, a dynamic frequency mask is obtained;

[0080] Obtain defect features based on dynamic frequency mask.

[0081] Specifically, S2 frequency domain decomposition:

[0082] The I obtained for S1 sub Perform frequency domain decomposition to extract high-frequency and low-frequency features and enhance the prominence of defect areas. The specific operations are as follows:

[0083] S21. Perform Fourier transform on the input features:

[0084] X fft =FFT(I sub ,norm=ortho)

[0085] Among them, FFT is the orthogonal normalized two-dimensional Fourier transform, which generates the frequency domain representation X fft .

[0086] S22. Wavelet transform:

[0087] to I sub Apply the discrete wavelet transform (DWT) to obtain high-frequency and low-frequency components at multiple resolutions:

[0088] X dwt =DWT(I sub ,wavelet=db1,level=L)

[0089] Where dbl is the Haar wavelet basis, L = 2 is the decomposition level, and the approximate component (low frequency) X is generated. approx dwt and detail component (high frequency) X detail dwt ={X h dwt , X v dwt , X d dwt}, representing the high-frequency features in the horizontal, vertical and diagonal directions respectively. The output dimension is X dwt ∈R B×N×C′×H′×W , where H′, W′ decreases due to the decomposition level, and C′ is determined by the wavelet component.

[0090] S23. Multi-scale LBP:

[0091] to I sub The spatial domain features of the multi-scale local binary pattern (LBP) are applied to extract texture features:

[0092] X lbp =LBP(I sub ,{R1,R2},P)

[0093] Among them, R1=1, R2=3 are different radii, P=8 is the number of neighborhood points, and the multi-scale texture feature X is generated. lbp ∈R B ×N×C″×H×W , where C″ is determined by the number of LBP patterns. LBP generates a binary pattern by comparing the grayscale values of the central pixel with the neighboring pixels to capture the texture characteristics of the defect.

[0094] S24. Generate a spatial attention map att_map to guide the construction of a dynamic frequency mask:

[0095] att_map=Attention(Concat(I sub ,X edge ,X dwt ,X lbp ))

[0096] Among them, X edgeIt is the edge feature extracted by edge convolution, Concat means channel concatenation, Attention is a single input module that combines channel and spatial information to enhance the feature expression output att_map.

[0097] S25. Generate dynamic frequency mask M based on spatial attention dyn (k) is used to decompose high-frequency and low-frequency components:

[0098] M dyn (k) = AdaptiveMask(X fft ,k i ,att_map)

[0099] Among them, k i ∈[2,4] is the frequency parameter.

[0100] S26. Use the mask to decompose the frequency components and obtain the high frequency component X high With the low frequency component X low :

[0101]

[0102]

[0103]

[0104] Among them, the initial iFFT is the inverse Fourier transform, and the final low-frequency component is X 2 low .

[0105] S27. Feature fusion through frequency-weighted convolution and edge enhancement:

[0106]

[0107] Among them, σ is the Sigmoid activation function, W i is the frequency weight convolution kernel, X freq Enhance features for the final frequency.

[0108] Furthermore, based on the dynamic frequency mask, defect features are obtained including:

[0109] Based on dynamic frequency mask, high-frequency features and low-frequency features are obtained;

[0110] Frequency-weighted convolution and edge enhancement fusion are performed on high-frequency features and low-frequency features to obtain defect features.

[0111] Furthermore, the defect features and the all-focus image features are fused to obtain comprehensive features including:

[0112] Use the encoder to process defect features and all-focus image features to obtain multi-scale features;

[0113] Use graph convolutional networks to construct graph structures with multi-scale features;

[0114] Fusion is performed based on graph structure to obtain comprehensive features.

[0115] Specifically, S3: Multi-scale feature fusion:

[0116] The X obtained by S2 freq And the input all-focus image feature Y all_focus ∈R B×C×H×W Perform multi-level fusion to generate comprehensive features. The specific operations are as follows:

[0117] S31. Process X through the encoder freq and Y all_focus , generate multi-scale features:

[0118] X=[X0,X1,X2,X3], Y=[Y1,Y2,Y3,Y4].

[0119] Among them, X i ,,Y i ∈R B×Ci×Hi×Wi They are the different scale representations of the residual features out of the focal plane after frequency enhancement and the fully focused image features.

[0120] S32.Graph structure construction:

[0121] For each scale i, X i and Y i Constructed as a graph structure G i =(V i , E i ):

[0122] Node V i :X i and Y i The feature points (pixels or regions) of are taken as nodes, and each node represents a feature vector. For example, for X i ∈R B×Ci×Hi×Wi , the feature vector X for each pixel position (h,w) i [:,:,h,w]∈R Ci As a node.

[0123] Edge E i : Build edges based on the spatial neighborhood relationship or similarity between feature points. For example, if two nodes v m , v n If the cosine similarity of the feature vector of is greater than the threshold θ, then add an edge (v m ,vn) .

[0124] Generate graph structure representation G i , node characteristic moment Z i ∈R ∣Vi∣×(Ci+Ci) (By splicing X i and Y i features), adjacency matrix A i ∈R ∣Vi∣×∣Vi∣ .

[0125] S33. Graph Convolutional Network (GCN) Fusion:

[0126] For Figure G i Apply GCN for feature update:

[0127]

[0128] Among them, A^ i =D -1 / 2 i (A i +I)D -1 / 2 i is the normalized adjacency matrix, D i is the degree matrix, W gcn ∈R (Ci+Ci)×Cout is the learnable weight, σ is the ReLU activation function, and the output F i gcn ∈R ∣Vi∣×Cout 。 i gcn Reshape back to the spatial dimension and get F i gcn ∈R B ×Cout×Hi×Wi , where C out =64.

[0129] S34. Enhance features through 3x3 convolution and attention mechanism:

[0130]

[0131] Among them, Conv is a 3x3 convolution (input C out , output 64 channels). Dimensionality reduction and upsampling:

[0132]

[0133] Generate F i out ∈R B×64×64×64 .

[0134] S35. Aggregate multi-scale features:

[0135]

[0136] Generate F fuse ∈R B×64×64×64 . Further enhancements:

[0137] F′ fuse =Attention(Conv(F fuse )),

[0138] F″ fuse =Conv(F′ fuse ,64,64),

[0139] Among them, Conv is a 3x3 convolution.

[0140] S36. Multi-stage fusion.

[0141] Furthermore, based on the graph structure, fusion is performed to obtain comprehensive features including:

[0142] Update the features of the graph structure to obtain the first feature;

[0143] Perform convolution and attention enhancement on the first feature to obtain the second feature;

[0144] Reduce the dimension and upsample the second feature to obtain the third feature;

[0145] Aggregate the second feature to obtain the aggregated feature;

[0146] Based on the aggregated features and the third features, comprehensive features are obtained.

[0147] Furthermore, based on the aggregated features and the third feature, the comprehensive features are obtained including:

[0148] Shallow fusion of the third feature to obtain shallow features;

[0149] Deep fusion of the third feature to obtain deep features;

[0150] Fuse deep features and shallow features to obtain initial comprehensive features;

[0151] The initial comprehensive features and aggregated features are integrated to obtain comprehensive features.

[0152] Specifically, S36. Multi-stage fusion includes:

[0153] S361. Fusion of shallow features:

[0154]

[0155] Among them, Conv is a 3x3 convolution.

[0156] S362. Fusion of Deep Features:

[0157]

[0158] S363. Fusion of shallow and deep layers:

[0159] F mid =Attention(Conv(Concat(F left ,F right ))

[0160] S364. Final fusion:

[0161]

[0162] Among them, Conv is a 3x3 convolution, generating F final ∈R B×64×64×64 .

[0163] Furthermore, based on the comprehensive features, the defect prediction map is obtained including:

[0164] Perform convolution mapping on the comprehensive features to obtain a single-channel feature map;

[0165] Perform bilinear interpolation upsampling on the single-channel feature map to the target resolution to obtain the defect prediction map.

[0166] Specifically, the S4 defect prediction result graph is generated:

[0167] F obtained for S3 final Perform convolution mapping:

[0168] F out =Conv(F final ,K=3,padding=1)

[0169] Among them, Conv is a 3×3 convolution, the number of output channels is 1, and a single-channel feature map F is generated. out ∈R B×1×64×64 . Then, it is upsampled to the target resolution via bilinear interpolation:

[0170] P defect =Interpolate(F out (256,256),mode=bilinear)

[0171] Generate the final defect prediction map P defect ∈R B×1×256×256 , used for locating and identifying surface defects.

[0172] This embodiment also provides a metal product surface defect detection system based on light field imaging, comprising: an image acquisition module, an image processing module and a defect detection module;

[0173] An acquisition module, used for acquiring light field images of the surface of metal products;

[0174] An image processing module is used to obtain focus stack image features and all-focus image features based on the light field image; obtain residual features outside the focal plane based on the focus stack image features and the all-focus image features; obtain defect features based on the residual features outside the focal plane; and fuse the defect features with the all-focus image features to obtain comprehensive features;

[0175] The defect detection module is used to obtain a defect prediction map based on comprehensive features, wherein the defect prediction map is used for defect location and detection.

[0176] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which implements a method for detecting surface defects of metal products when the program is executed by a processor.

[0177] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for detecting surface defects of metal products based on light field imaging, characterized in that: include: Acquire a light field image; Based on the light field image, acquiring focus stack image features and all-focus image features; acquiring residual features outside the focal plane based on the focused stack image features and the all-focused image features; Obtaining defect features based on the residual features outside the focal plane; Fusing the defect feature with the all-focus image feature to obtain a comprehensive feature; Based on the comprehensive features, a defect prediction map is obtained, wherein the defect prediction map is used for defect location and detection.

2. The method for detecting surface defects of metal products based on light field imaging according to claim 1, characterized in that: Based on the focus stack image features and the all-focus image features, obtaining residual features outside the focal plane includes: A focus stack in the light field image is subtracted from the all-focus image of the light field image to obtain residual features outside the focal plane.

3. The method for detecting surface defects of metal products based on light field imaging according to claim 1, characterized in that: Obtaining defect features based on the residual features outside the focal plane includes: Performing Fourier transform on the residual features outside the focal plane to obtain frequency domain features; Performing wavelet transform and texture feature extraction on the residual features outside the focal plane to obtain wavelet features and texture features; generating a spatial attention map based on the wavelet features and the texture features; Based on the spatial attention map and in combination with the frequency domain features, a dynamic frequency mask is obtained; The defect feature is obtained based on the dynamic frequency mask.

4. The method for detecting surface defects of metal products based on light field imaging according to claim 3, characterized in that: Acquiring the defect feature based on the dynamic frequency mask includes: Based on the dynamic frequency mask, obtaining high-frequency features and low-frequency features; Frequency weighted convolution and edge enhancement fusion processing are performed on the high-frequency features and the low-frequency features to obtain the defect features.

5. The method for detecting surface defects of metal products based on light field imaging according to claim 1, characterized in that: The defect features and the all-focus image features are fused to obtain comprehensive features including: Processing the defect features and the all-focus image features using an encoder to obtain multi-scale features; Constructing a graph structure of the multi-scale features using a graph convolutional network; Fusion is performed based on the graph structure to obtain the comprehensive features.

6. The method for detecting surface defects of metal products based on light field imaging according to claim 5, characterized in that: The fusion based on the graph structure to obtain the comprehensive features includes: Performing feature updating on the graph structure to obtain a first feature; Perform convolution and attention enhancement on the first feature to obtain the second feature; Performing dimensionality reduction and upsampling on the second feature to obtain a third feature; Aggregating the second features to obtain aggregated features; The comprehensive feature is obtained based on the aggregated feature and the third feature.

7. The method for detecting surface defects of metal products based on light field imaging according to claim 6, characterized in that: Based on the aggregated feature and the third feature, obtaining the comprehensive feature includes: shallowly fusing the third feature to obtain shallow features; Deeply integrating the third feature to obtain a deep feature; Fusing the deep features and shallow features to obtain initial comprehensive features; The initial comprehensive features and the aggregated features are fused to obtain the comprehensive features.

8. The method for detecting surface defects of metal products based on light field imaging according to claim 1, characterized in that: Based on the comprehensive features, obtaining a defect prediction map includes: Performing convolution mapping on the comprehensive features to obtain a single-channel feature map; Perform bilinear interpolation upsampling on the single-channel feature map to the target resolution to obtain the defect prediction map.

9. A metal product surface defect detection system based on light field imaging, characterized in that: include: Image acquisition module, image processing module and defect detection module; The acquisition module is used to acquire light field images of the surface of the metal product; The image processing module is configured to obtain focus stack image features and all-focus image features based on the light field image; and obtain residual features outside the focal plane based on the focus stack image features and all-focus image features; Obtaining defect features based on the residual features outside the focal plane; Fusing the defect feature with the all-focus image feature to obtain a comprehensive feature; The defect detection module is used to obtain a defect prediction map based on the comprehensive features, wherein the defect prediction map is used for defect location and detection.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images