A metal product surface defect detection method and system based on light field imaging

Abstract

The application discloses a metal product surface defect detection method and system based on light field imaging. The detection method comprises the following steps: acquiring a light field image; acquiring a focus stack image feature and a full-focus image feature based on the light field image; acquiring a residual feature outside a focal plane based on the focus stack image feature and the full-focus image feature; acquiring a defect feature based on the residual feature outside the focal plane; fusing the defect feature and the full-focus image feature to acquire a comprehensive feature; and acquiring a defect prediction map based on the comprehensive feature, wherein the defect prediction map is used for defect positioning and detection. Through effective processing of the focus stack and the full-focus image, high-precision detection of the metal product surface defect is realized.

Description

Technical Field

[0001] This invention belongs to the field of industrial image inspection technology, and in particular relates to a method and system for detecting surface defects in metal products based on light field imaging. Background Technology

[0002] The surface quality of metal products (such as automotive parts, mobile phone frames, and precision mechanical parts) directly affects their mechanical properties, corrosion resistance, and service life. However, during the manufacturing process, metal surfaces may develop various defects such as scratches, cracks, and porosity due to processing techniques or environmental factors. Traditional methods for detecting surface defects in metal products rely on experienced personnel observing them through microscopes or magnifying glasses, which is inefficient and highly subjective. Machine vision-based automated inspection, on the other hand, typically relies on two-dimensional images, which are easily affected by surface reflections and complex textures, leading to a high false detection rate. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a method and system for detecting surface defects in metal products based on light field imaging. By effectively processing the focusing stack and the full-focus image, high-precision detection of surface defects in metal products can be achieved.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A method for detecting surface defects in metal products based on light field imaging, comprising:

[0006] Acquire light field images;

[0007] Based on the light field image, obtain the focused stack image features and the full-focus image features;

[0008] Based on the focused stack image features and the full-focus image features, residual features outside the focal plane are obtained;

[0009] Defect features are obtained based on the residual features outside the focal plane;

[0010] The defect features and the full-focus image features are fused to obtain comprehensive features;

[0011] Based on the comprehensive features, a defect prediction map is obtained, wherein the defect prediction map is used for defect location and detection.

[0012] Optionally, based on the focused stack image features and the full-focus image features, residual features outside the focal plane are obtained:

[0013] The residual features outside the focal plane are obtained by subtracting the focus stack in the light field image from the fully focused image of the light field image.

[0014] Optionally, based on the residual features outside the focal plane, obtaining defect features includes:

[0015] Perform a Fourier transform on the residual features outside the focal plane to obtain the frequency domain features;

[0016] Wavelet transform and texture feature extraction are performed on the residual features outside the focal plane to obtain wavelet features and texture features;

[0017] Based on the wavelet features and texture features, a spatial attention map is generated;

[0018] Based on the spatial attention map and combined with the frequency domain features, a dynamic frequency mask is obtained;

[0019] The defect features are obtained based on the dynamic frequency mask.

[0020] Optionally, obtaining the defect features based on the dynamic frequency mask includes:

[0021] Based on the dynamic frequency mask, high-frequency features and low-frequency features are obtained;

[0022] The high-frequency and low-frequency features are subjected to frequency-weighted convolution and edge enhancement fusion processing to obtain the defect features.

[0023] Optionally, the defect features and the full-focus image features are fused to obtain comprehensive features, including:

[0024] The encoder is used to process the defect features and the full-focus image features to obtain multi-scale features;

[0025] The graph structure of the multi-scale features is constructed using a graph convolutional network;

[0026] The integrated features are obtained by fusing the graph structure.

[0027] Optionally, the fusion based on the graph structure to obtain the comprehensive features includes:

[0028] The graph structure is updated to obtain a first feature;

[0029] The first feature is subjected to convolution and attention enhancement to obtain the second feature;

[0030] The second feature is then subjected to dimensionality reduction and upsampling to obtain the third feature;

[0031] Aggregate the second feature to obtain the aggregated feature;

[0032] Based on the aggregated features and the third feature, the comprehensive features are obtained.

[0033] Optionally, based on the aggregated features and the third feature, obtaining the comprehensive features includes:

[0034] Shallow fusion of the third feature yields shallow features;

[0035] Deep fusion of the third feature is performed to obtain deep features;

[0036] By fusing the deep and shallow features, an initial comprehensive feature is obtained;

[0037] The initial integrated features and aggregated features are combined to obtain the integrated features.

[0038] Optionally, obtaining the defect prediction map based on the comprehensive features includes:

[0039] The comprehensive features are convolutionally mapped to obtain single-channel feature maps;

[0040] The single-channel feature map is upsampled to the target resolution using bilinear interpolation to obtain the defect prediction map.

[0041] Optionally, obtaining the defect prediction map based on the comprehensive features includes:

[0042] The comprehensive features are convolutionally mapped to obtain single-channel feature maps;

[0043] The single-channel feature map is upsampled to the target resolution using bilinear interpolation to obtain the defect prediction map.

[0044] The present invention also provides a surface defect detection system for metal products based on light field imaging, comprising: an image acquisition module, an image processing module, and a defect detection module;

[0045] The acquisition module is used to acquire light field images of the surface of metal products;

[0046] The image processing module is used to obtain focused stack image features and full-focus image features based on the light field image; obtain residual features outside the focal plane based on the focused stack image features and full-focus image features; obtain defect features based on the residual features outside the focal plane; and fuse the defect features and the full-focus image features to obtain comprehensive features.

[0047] The defect detection module is used to obtain a defect prediction map based on the comprehensive features, wherein the defect prediction map is used for defect localization and detection.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for detecting surface defects in metal products.

[0049] Compared with the prior art, the present invention has the following advantages and technical effects:

[0050] 1. High-precision detection: By subtracting the focus stack from the fully focused image, the feature representation capability of minute defects (such as scratches, dents, cracks, etc.) is effectively enhanced. Compared with traditional RGB image detection methods, this invention can capture micron-level defects, improve detection accuracy, and is especially suitable for industrial scenarios with complex textures and lighting conditions.

[0051] 2. Enhanced Frequency Domain Features: Through frequency domain analysis, high-frequency and low-frequency features are decomposed. High-frequency features capture the edges and details of defects, while low-frequency features preserve the overall information of the background, enhancing the salient features of the defect area (such as edge and texture variations). This mechanism effectively suppresses background noise interference and improves the robustness of the model under non-uniform lighting and complex surface backgrounds.

[0052] 3. Adaptability to complex environments: This method can effectively suppress background noise, highlight defect areas, adapt to complex industrial environments, and improve the robustness of detection.

[0053] 4. Broad application prospects: This method has broad application prospects in fields such as metal product quality control and industrial manufacturing. It can significantly improve production efficiency and product quality, and provide strong technical support and guarantee for industrial manufacturing enterprises. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a flowchart of a method for detecting surface defects in metal products based on light field imaging, according to an embodiment of the present invention. Detailed Implementation

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0057] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0058] This embodiment proposes a method for detecting surface defects in metal products based on light field imaging, such as... Figure 1 As shown, the specific steps include:

[0059] Acquire light field images;

[0060] Based on the light field image, obtain the features of the focused stacked image and the full-focus image;

[0061] Based on the features of the focused stacked image and the full-focus image, residual features outside the focal plane are obtained;

[0062] Defect features are obtained based on residual features outside the focal plane;

[0063] By fusing defect features and full-focus image features, a comprehensive feature is obtained;

[0064] Based on comprehensive features, a defect prediction map is obtained, which is used for defect localization and detection.

[0065] Specifically, the method for detecting surface defects in metal products includes: S1 Full-focus image minus focus stack: The input focus stack and full-focus image are processed, and the focus stack is subtracted from the full-focus image to obtain residual features outside the focal plane. These features highlight the differences between different focal planes and the overall clear image, emphasizing the unique information of defects in depth and space, and providing a basis for subsequent analysis.

[0066] S2 Frequency Domain Decomposition: Based on the residual features outside the focal plane obtained in the previous step, frequency domain analysis is performed to decompose them into high-frequency and low-frequency features. High-frequency features capture the edges and details of defects, while low-frequency features retain the overall information of the background. Then, an attention mechanism is used to highlight the defect area, resulting in clearer defect features.

[0067] S3 Feature Fusion: Based on the defect features obtained in the previous step, these features are fused with the features of the full-focus image at multiple levels. By constructing a relationship network between the out-of-focus residual features and the full-focus image features, the connections between the features are analyzed, and then the information is integrated to obtain comprehensive features. These comprehensive features combine the depth information of the light field image and the color and texture information of the full-focus image, enabling them to better adapt to defect detection in complex scenes.

[0068] S4 Prediction Map Generation: Based on the comprehensive features obtained in the previous step, these features are mapped and magnified to generate a high-resolution defect prediction map. This map clearly shows the location and shape of the defect, and can be used to accurately locate and identify defects on metal surfaces.

[0069] Furthermore, based on the features of the focused stacked image and the full-focus image, the residual features outside the focal plane are obtained, including:

[0070] By subtracting the focus stack in the light field image from the fully focused image of the light field image, the residual features outside the focal plane can be obtained.

[0071] Specifically, the input data includes the focus stack I.focal_stack ∈R B×C×H×W (Where B is the batch size, C is the number of channels, and H×W is the spatial resolution) and the fully focused image I all_focus ∈R B×C×H×W The focus stack contains sub-field images with multiple different focal planes, and the full-focus image is the sharp image obtained by integrating all the focal planes.

[0072] Subtract the focus stack from the fully focused image:

[0073] I sub =I all_focus -I focal_stack

[0074] Among them, I sub The residual features outside the focal plane after subtraction. This operation obtains the differences between each subfield of view by subtracting the full-focus image from the focus stack, highlighting the unique information of the defects in depth and space, and providing input for subsequent frequency domain decomposition.

[0075] Furthermore, based on the residual features outside the focal plane, the defect features are obtained as follows:

[0076] Perform a Fourier transform on the residual features outside the focal plane to obtain the frequency domain features;

[0077] Wavelet transform and texture feature extraction are performed on the residual features outside the focal plane to obtain wavelet features and texture features;

[0078] Spatial attention maps are generated based on wavelet features and texture features;

[0079] Based on spatial attention maps and combined with frequency domain features, dynamic frequency masks are obtained.

[0080] Defect characteristics are obtained based on dynamic frequency masks.

[0081] Specifically, S2 frequency domain decomposition:

[0082] The I obtained from S1 sub Frequency domain decomposition is performed to extract high-frequency and low-frequency features, enhancing the salience of the defect area. The specific steps are as follows:

[0083] S21. Perform Fourier transform on the input features:

[0084] X fft =FFT(I sub (norm = ortho)

[0085] Wherein, FFT is an orthogonally normalized two-dimensional Fourier transform that generates the frequency domain representation X. fft .

[0086] S22. Wavelet Transform:

[0087] to I sub Apply Discrete Wavelet Transform (DWT) to obtain high-frequency and low-frequency components at multiple resolutions:

[0088] X dwt =DWT(I sub ,wavelet=db1,level=L)

[0089] Where dbl is the Haar wavelet basis, L=2 is the decomposition level, and the approximate component (low frequency) X is generated. approx dwt and detail components (high frequency) X detail dwt ={X h dwt X v dwt X d dwt} represent the high-frequency features in the horizontal, vertical, and diagonal directions, respectively. The output dimension is X. dwt ∈R B×N×C′×H′×W H′ and W′ decrease due to the decomposition hierarchy, while C′ is determined by the wavelet components.

[0090] S23. Multiscale LBP:

[0091] to I sub The spatial domain features are extracted using multi-scale local binary modeling (LBP).

[0092] X lbp =LBP(I sub ,{R1,R2},P)

[0093] Where R1=1, R2=3 represent different radii, P=8 represents the number of neighborhood points, and multi-scale texture features X are generated. lbp ∈R B ×N×C″×H×W C″ is determined by the number of LBP modes. LBP generates a binary mode by comparing the gray values ​​of the center pixel with those of its neighboring pixels, thereby capturing the texture characteristics of defects.

[0094] S24. Generate a spatial attention map att_map, which is used to guide the construction of the dynamic frequency mask:

[0095] att_map = Attention(Concat(I sub ,X edge ,X dwt ,X lbp ))

[0096] Among them, X edgeIt extracts edge features through edge convolution, Concat represents channel concatenation, and Attention is a single-input module that combines channel and spatial information to enhance the feature representation output att_map.

[0097] S25. Generating Dynamic Frequency Masks M Based on Spatial Attention dyn (k) is used to decompose high-frequency and low-frequency components:

[0098] M dyn (k) = AdaptiveMask(X) fft ,k i ,att_map)

[0099] Where, k i ∈[2,4] represents the frequency parameter.

[0100] S26. Use masking to decompose the frequency components to obtain the high-frequency component X. high With low-frequency component X low :

[0101]

[0102]

[0103]

[0104] Among them, the initial iFFT is the inverse Fourier transform, and the final low-frequency component is X. 2 low .

[0105] S27. Feature fusion via frequency-weighted convolution and edge enhancement:

[0106]

[0107] Where σ is the Sigmoid activation function, W i X is the frequency-weighted convolution kernel. freq This is for the final frequency enhancement feature.

[0108] Furthermore, based on dynamic frequency masks, the defect features obtained include:

[0109] High-frequency and low-frequency features are obtained based on dynamic frequency masks;

[0110] Defect features are obtained by performing frequency-weighted convolution and edge enhancement fusion on high-frequency and low-frequency features.

[0111] Furthermore, the defect features and full-focus image features are fused to obtain comprehensive features, including:

[0112] The encoder is used to process defect features and full-focus image features to obtain multi-scale features;

[0113] Constructing graph structures with multi-scale features using graph convolutional networks;

[0114] Fusion is performed based on graph structure to obtain comprehensive features.

[0115] Specifically, S3: Multi-scale feature fusion:

[0116] X obtained from S2 freq and the input full-focus image features Y all_focus ∈R B×C×H×W Multi-level fusion is performed to generate comprehensive features. The specific steps are as follows:

[0117] S31. Process X via encoder freq and Y all_focus Generate multi-scale features:

[0118] X=[X0,X1,X2,X3], Y=[Y1,Y2,Y3,Y4].

[0119] Among them, X i ,,Y i ∈R B×Ci×Hi×Wi These represent the residual features outside the focal plane after frequency enhancement and the features of the fully focused image at different scales.

[0120] S32. Graph Structure Construction:

[0121] For each scale i, X i and Y i Construct as a graph structure G i =(V i E i ):

[0122] Node V i : X i and Y i Feature points (pixels or regions) are used as nodes, and each node represents a feature vector. For example, for X... i ∈R B×Ci×Hi×Wi The feature vector X at each pixel position (h,w) i [:,:,h,w]∈R Ci As a node.

[0123] Side E i Edges are constructed based on the spatial neighborhood relationships or similarities between feature points. For example, if two nodes v m v n If the cosine similarity of the feature vectors is greater than the threshold θ, then add an edge (v). m ,vn) .

[0124] Generative graph structure representation G i Node eigenmotes Z i ∈R ∣Vi∣×(Ci+Ci) (by splicing X) i and Y i (Characteristics), adjacency matrix A i ∈R ∣Vi∣×∣Vi∣ .

[0125] S33. Fusion of Graph Convolutional Networks (GCNs):

[0126] For graph G i Applying GCN for feature updates:

[0127]

[0128] Among them, A^ i =D -1 / 2 i (A i +I)D -1 / 2 i For the normalized adjacency matrix, D i W is the degree matrix. gcn ∈R (Ci+Ci)×Cout Let F be the learnable weights, σ ​​be the ReLU activation function, and F be the output. i gcn ∈R ∣Vi∣×Cout F i gcn Reconstructing back to spatial dimensions yields F. i gcn ∈R B ×Cout×Hi×Wi C out =64.

[0129] S34. Enhance features through 3x3 convolution and attention mechanisms:

[0130]

[0131] Where Conv is a 3x3 convolution (input C) out (64-channel output). Dimensionality reduction and upsampling:

[0132]

[0133] Generate F i out ∈R B×64×64×64 .

[0134] S35. Aggregated Multi-Scale Features:

[0135]

[0136] Generate F fuse ∈R B×64×64×64 Further enhancements:

[0137] F′ fuse =Attention(Conv(F) fuse )),

[0138] F″ fuse =Conv(F′ fuse ,64,64),

[0139] Where Conv is a 3x3 convolution.

[0140] S36. Multi-stage integration.

[0141] Furthermore, based on graph structure, fusion is performed to obtain comprehensive features, including:

[0142] Update the graph structure by performing feature updates to obtain the first feature;

[0143] For the first feature, perform convolution and attention enhancement to obtain the second feature;

[0144] The second feature is reduced in dimensionality and upsampled to obtain the third feature;

[0145] Aggregate the second feature to obtain the aggregated feature;

[0146] Based on aggregated features and third features, comprehensive features are obtained.

[0147] Furthermore, based on aggregated features and third features, comprehensive features are obtained, including:

[0148] Shallow fusion of the third feature yields shallow features;

[0149] Deep fusion of the third feature to obtain deep features;

[0150] By fusing deep and shallow features, initial comprehensive features are obtained;

[0151] The initial comprehensive features and aggregated features are combined to obtain the comprehensive features.

[0152] Specifically, S36. Multi-stage fusion includes:

[0153] S361. Fusion of shallow features:

[0154]

[0155] Where Conv is a 3x3 convolution.

[0156] S362. Fusing deep features:

[0157]

[0158] S363. Merging shallow and deep layers:

[0159] F mid =Attention(Conv(Concat(F) left ,F right ))

[0160] S364. Final Fusion:

[0161]

[0162] Where Conv is a 3x3 convolution, generating F final ∈R B×64×64×64 .

[0163] Furthermore, based on comprehensive features, the defect prediction map is obtained including:

[0164] Perform convolution mapping on the comprehensive features to obtain single-channel feature maps;

[0165] The single-channel feature map is upsampled to the target resolution using bilinear interpolation to obtain a defect prediction map.

[0166] Specifically, the S4 defect prediction result diagram is generated:

[0167] F obtained from S3 final Perform convolution mapping:

[0168] F out =Conv(F final (K=3, padding=1)

[0169] Where Conv is a 3×3 convolution with 1 output channel, generating a single-channel feature map F. out ∈R B×1×64×64 Subsequently, the sampled data is upsampled to the target resolution using bilinear interpolation.

[0170] P defect =Interpolate(F out (256,256),mode=bilinear)

[0171] Generate the final defect prediction map P defect ∈R B×1×256×256 It is used for the location and identification of surface defects.

[0172] This embodiment also provides a surface defect detection system for metal products based on light field imaging, including: an image acquisition module, an image processing module, and a defect detection module;

[0173] The acquisition module is used to acquire light field images of the surface of metal products;

[0174] The image processing module is used to obtain focused stack image features and full-focus image features based on the light field image; obtain residual features outside the focal plane based on the focused stack image features and full-focus image features; obtain defect features based on the residual features outside the focal plane; and fuse the defect features and full-focus image features to obtain comprehensive features.

[0175] The defect detection module is used to obtain a defect prediction map based on comprehensive features. The defect prediction map is used for defect location and detection.

[0176] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for detecting surface defects in metal products.

[0177] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting surface defects in metal products based on light field imaging, characterized in that, include: Acquire light field images; Based on the light field image, obtain the focused stack image features and the full-focus image features; Based on the focused stack image features and the full-focus image features, residual features outside the focal plane are obtained; Based on the focused stack image features and the fully focused image features, the residual features outside the focal plane are obtained as follows: The residual features outside the focal plane are obtained by subtracting the focus stack in the light field image from the fully focused image of the light field image. Defect features are obtained based on the residual features outside the focal plane; Based on the residual features outside the focal plane, the defect features are obtained as follows: Perform a Fourier transform on the residual features outside the focal plane to obtain the frequency domain features; Wavelet transform and texture feature extraction are performed on the residual features outside the focal plane to obtain wavelet features and texture features; Based on the wavelet features and texture features, a spatial attention map is generated; Based on the spatial attention map and combined with the frequency domain features, a dynamic frequency mask is obtained; Based on the dynamic frequency mask, the defect features are obtained; Based on the dynamic frequency mask, obtaining the defect features includes: Based on the dynamic frequency mask, high-frequency features and low-frequency features are obtained; The high-frequency and low-frequency features are subjected to frequency-weighted convolution and edge enhancement fusion processing to obtain the defect features; The defect features and the full-focus image features are fused to obtain comprehensive features; Based on the comprehensive features, a defect prediction map is obtained, wherein the defect prediction map is used for defect location and detection.

2. The method for detecting surface defects in metal products based on light field imaging according to claim 1, characterized in that, The defect features and the full-focus image features are fused to obtain comprehensive features, including: The encoder is used to process the defect features and the full-focus image features to obtain multi-scale features; The graph structure of the multi-scale features is constructed using a graph convolutional network; The integrated features are obtained by fusing the graph structure.

3. The method for detecting surface defects in metal products based on light field imaging according to claim 2, characterized in that, The fusion based on the graph structure to obtain the comprehensive features includes: The graph structure is updated to obtain a first feature; The first feature is subjected to convolution and attention enhancement to obtain the second feature; The second feature is then subjected to dimensionality reduction and upsampling to obtain the third feature; Aggregate the second feature to obtain the aggregated feature; Based on the aggregated features and the third feature, the comprehensive features are obtained.

4. The method for detecting surface defects in metal products based on light field imaging according to claim 3, characterized in that, Based on the aggregated features and the third feature, the comprehensive features are obtained as follows: Shallow fusion of the third feature yields shallow features; Deep fusion of the third feature is performed to obtain deep features; The deep and shallow features are fused to obtain the initial comprehensive features; The initial integrated features and aggregated features are combined to obtain the integrated features.

5. The method for detecting surface defects in metal products based on light field imaging according to claim 1, characterized in that, Based on the aforementioned comprehensive features, obtaining the defect prediction map includes: The comprehensive features are convolutionally mapped to obtain single-channel feature maps; The single-channel feature map is upsampled to the target resolution using bilinear interpolation to obtain the defect prediction map.

6. A surface defect detection system for metal products based on light field imaging, used to implement the method as described in any one of claims 1-5, characterized in that, include: Image acquisition module, image processing module, and defect detection module; The acquisition module is used to acquire light field images of the surface of metal products; The image processing module is used to obtain focused stack image features and full-focus image features based on the light field image; and to obtain residual features outside the focal plane based on the focused stack image features and full-focus image features. Defect features are obtained based on the residual features outside the focal plane; The defect features and the full-focus image features are fused to obtain comprehensive features; Based on the focused stack image features and the fully focused image features, the residual features outside the focal plane are obtained as follows: The residual features outside the focal plane are obtained by subtracting the focus stack in the light field image from the fully focused image of the light field image. The defect detection module is used to obtain a defect prediction map based on the comprehensive features, wherein the defect prediction map is used for defect localization and detection.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images