Metal defect identification method trained by few training samples

Abstract

The invention discloses a metal defect identification method trained by a small number of training samples, and the method comprises the steps: completing the labeling of a data set through Label labeling software, and carrying out the data enhancement through random frequency domain filtering; a wavelet convolution M-WTConv module based on multi-level feature interaction extraction is added; adding a windowed cross attention DM-FAM module based on frequency domain analysis, dynamic frequency band selection and dynamic frequency perception; a semantic enhancement cross-modal fusion module SECF is added to improve the feature representation capability of a detection head in the YOLOv10; and gradually training the improved neural network in two stages to complete metal defect identification. According to the method, the target detection capability of metal surface defects under high-reflection and few-sample scenes is effectively enhanced, and higher robustness, recognition rate and accuracy are achieved for metal defect detection and recognition tasks under the condition of few samples.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and recognition, and specifically, relates to a method for identifying metal defects trained with a small number of training samples. Background Art

[0002] Currently, the technology for identifying metal defects is in a crucial stage of rapid development and plays a vital role, especially in industrial production. With the continuous upgrading of metal processing technologies, the product precision and complexity have been significantly improved. Traditional defect detection methods have gradually revealed problems such as low efficiency and high false detection rate, and it is difficult to meet the stringent requirements of modern industry for high precision and high efficiency. Therefore, the detection of metal surface defects is accelerating the transformation towards intelligence and automation. By introducing advanced technologies such as artificial intelligence and machine vision, intelligent detection systems can achieve more efficient and accurate defect identification, greatly reducing human errors and improving detection efficiency. This trend not only promotes the technological innovation in the metal processing field but also provides strong technical support for the industrial intelligent transformation and high-quality production, becoming one of the core driving forces for the future development of the metal processing industry.

[0003] With the increasing demand for intelligent defect detection in the metal processing industry, the application of deep learning technology in metal surface defect identification has become the mainstream trend. However, in actual industrial production, the cost of obtaining a large number of high-quality labeled samples is high and the cycle is long. Especially in some special metal materials or new processing technologies, the scarcity of defect samples further restricts the training effect of traditional deep learning models. This problem of insufficient samples leads to weak generalization ability of the model and it is difficult to adapt to the complex and changeable industrial environment. In addition, characteristics such as high reflectivity, complex surface texture, and diverse defect morphologies in metal processing scenarios further increase the difficulty of defect detection. Traditional deep learning methods usually rely on large-scale labeled data, but in actual production, due to the uneven distribution of defect samples and the extremely low occurrence frequency of some defect types, the problem of unbalanced training data is particularly prominent. Therefore, how to achieve efficient and accurate metal defect identification under the condition of a small number of training samples has become an important research direction. At the same time, how to overcome model overfitting, improve the robustness of defect detection, and ensure the defect recognition ability of the model under limited samples are still key challenges to be solved urgently.

[0004] In recent years, the small-sample image object detection technology based on deep learning has shown a certain development trend in the field of metal surface defect detection and has begun to be applied in practice. Regarding small-sample training, Xia et al. (XIA R Y, LI GQ, HUANG Z W, et al. Bi-path Combination YOLO for Real-time Few-shot Object Detection[J]. Pattern Recognition Letters, 2023, 165: 91-97.) proposed a real-time few-shot object detection method BC-YOLO based on transfer learning. This structure provides a new idea for small-sample detection. However, due to the particularity of metal defects, it still needs to be specifically improved. In terms of attention, Liang et al. (Liang, YuJie, et al. Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion[C]. Advances in Neural Information Processing Systems, 2024, 37: 63441-63465.) proposed a fusion method combining frequency-domain and spatial-domain features and designed an attention mechanism to optimize band selection. However, in the field of metal defect detection, the performance and modeling still need to be further optimized to meet the requirements of edge device operation. Tang Hesheng et al. (Tang Hesheng, Li Du, Liao Yangyang, et al. Structural Shape Optimization Design Based on Physics-driven Deep Learning[J]. Journal of Hunan University (Natural Sciences), 2024, 51(11): 33-42.) proposed the PIC2f-YOLO method for metal surface defects. Through the PIC2f module, AIFI attention module, and ADown downsampling module, the number of parameters of the algorithm is reduced and the feature extraction ability of the model is improved. However, it still has deficiencies in the ability to detect small-sample defects.

[0005] In summary, although the metal defect recognition technology based on deep learning with a small number of training samples has made certain developments, there are still many deficiencies waiting to be improved to better meet the requirements of high efficiency and high detection rate for metal surface defect detection and recognition under the condition of a small number of training samples. This will provide new possibilities for the metal processing industry to achieve efficient and low-cost intelligent defect detection. Summary of the Invention

[0006] The present invention aims to overcome the deficiencies in the above prior art and proposes a metal defect recognition method trained with a small number of training samples, so as to improve the detection, recognition efficiency, robustness and accuracy of tiny metals in the case of small-sample training in the industrial production field.

[0007] To achieve the above invention content, the present invention provides a metal defect recognition method trained with a small number of training samples, including the following steps:

[0008] (1) Collect image samples of various defective metals;

[0009] (2) Preprocess and annotate the images;

[0010] (3) Make a training set based on data augmentation and enhancement;

[0011] (4) Add a wavelet convolutional M-WTConv module based on multi-level feature interaction extraction;

[0012] (5) Add a windowed cross-attention DM-FAM module based on frequency domain analysis and dynamic frequency band selection and dynamic frequency perception;

[0013] (6) Add a semantic enhancement cross-modal fusion module SECF to improve the feature representation ability of the detection head in YOLOv10;

[0014] (7) Two-stage progressive training of the improved neural network to complete metal defect recognition;

[0015] The collection of image samples of various defective metals refers to collecting image samples of defective metals of six types: bubbles, scratches, oxidation spots, oil stains, cracks, and depressions, with 30 samples collected for each type. The preprocessing and annotation of the images refer to using the Labelme picture annotation tool to annotate the corresponding defects of the collected pictures. The making of the training set based on data augmentation and enhancement refers to using a physical rendering engine to generate highly reflective synthetic data and simulate different lighting conditions, and performing random frequency domain filtering on the input images to enhance the robustness of the model to frequency domain features. The two-stage progressive training strategy refers to dividing the model training into a pre-training stage and a fine-tuning training stage. The pre-training stage refers to using the normal samples and synthetic defect samples of the industrial quality inspection dataset MVTecAD to conduct large-scale training on the backbone network and the neck network. The fine-tuning training stage refers to using the support set samples collected for the recognition task and adopting the N-way K-shot training method to optimize the meta-learning parameters and the classification head to achieve the learning effect of training with a small number of training samples. The improved neural network refers to the deep neural network obtained by modifying the network structure in steps (4) to (6).

[0016] Furthermore, a metal defect recognition method trained with a small number of training samples provided by the present invention is characterized in that adding the wavelet convolutional M-WTConv module based on multi-level feature interaction extraction means adding an improved wavelet convolutional module in parallel to all convolutional modules of the YOLOv10 backbone network. The improved wavelet convolutional module refers to adding a multi-level feature interaction module on the basis of the original wavelet convolutional module. The multi-level feature interaction module is composed of a cascade of a wavelet decomposition layer, three component feature extraction layers, a multi-level interaction layer, a feature fusion layer, and an inverse transform layer. The component feature extraction layer refers to performing convolution addition on four different component feature maps output by the wavelet decomposition layer. The feature fusion layer refers to splicing the output feature maps of the multi-level feature interaction module and performing convolution to obtain the corresponding output feature maps.

[0017] Furthermore, a metal defect recognition method trained with a small number of training samples provided by the present invention is characterized in that the addition of the windowed cross-attention DM-FAM module based on frequency-domain analysis and dynamic band selection and dynamic frequency perception means adding an improved dynamic frequency-domain attention module before the upsampling layer of the YOLOv10 neck network. The improved dynamic frequency-domain attention module is a network composed of four parts in series: a grouped frequency-domain feature extraction module, an adaptive band selection module, a frequency-domain and spatial-domain fusion module, and an inverse transform and grouping and merging module. The grouped frequency-domain feature extraction module is a network composed of a grouped parallel module and a fast Fourier transform module. Among them, the grouped parallel module groups the number of channels of the image feature map according to the number of GPU cores of the device, and the fast Fourier transform module converts the input feature map from the spatial domain to the frequency domain and separates the amplitude spectrum and the phase spectrum. The adaptive band selection module is a network composed of a global average pooling layer, a multi-layer perceptron layer, and a dynamic frequency-domain feature mask layer in series. The dynamic frequency-domain feature mask layer is a mask model based on Gaussian distribution modeling. The inverse transform and grouping and merging module is to perform an inverse Fourier transform on the feature map output by the dynamic frequency-domain feature mask layer and merge all groups to obtain a new feature map. The frequency-domain-spatial-domain fusion module is a network composed of an energy extraction module, a window prediction WDNet module, a feature projection module, a cross-attention calculation module, and a dynamic window interaction and fusion module in series. Among them, the energy extraction module is to perform a fast Fourier transform on the input feature map to obtain an energy spectrum. The window prediction WDNet module is a network composed of a 3×3 convolutional layer, a ReLU activation function, a max pooling layer, a 3×3 convolutional layer, a ReLU activation function, an adaptive pooling layer, a fully connected layer, and a Sigmoid activation function in series. The feature projection module is to obtain dynamic weights by passing the feature map output by the WDNet module through an input global average pooling layer and an MLP layer, and then multiply the value output by the full connection of the feature map of the inverse transform and grouping and merging module to obtain the Query value in the cross-attention calculation module. At the same time, the value output by the full connection of the feature map of the inverse transform and grouping and merging module is used as the Key and Value values in the cross-attention calculation module. The cross-attention calculation module is to perform Softmax self-attention calculation. The dynamic window interaction and fusion module is to use a cyclic shift strategy to achieve cross-window interaction.

[0018] Furthermore, a method for identifying metal defects trained with a small number of training samples provided by the present invention is characterized in that adding a semantic enhancement cross-modal fusion module SECF to improve the feature representation ability of the detection head in YOLOv10 means adding a module for texturizing visual features on the basis of the existing detection head of YOLOv10, combining a cascade composed of a semantic enhancement module of a large language model LLM and a cross-modal feature fusion module. The module for texturizing visual features refers to a text description of region proposals generated through semantic conversion. The semantic enhancement module combining the large language model LLM refers to encoding the text description into a semantic vector and inputting it into a semantic analysis module for semantic enhancement to capture the global context information and semantic features of the target. The cross-modal feature fusion module refers to combining the fused features with the original input features by element-wise addition and finally outputting the enhanced feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an embodiment of the present invention;

[0020] Figure 2 is a structural diagram of an improved M-WTConv wavelet convolution module according to an embodiment of the present invention;

[0021] Figure 3 is a structural diagram of an improved dynamic frequency domain attention DM-FAM module according to an embodiment of the present invention;

[0022] Figure 4 is a structural diagram of WDNet according to an embodiment of the present invention;

[0023] Figure 5 is a structural diagram of the cross-modal fusion module SECF according to an embodiment of the present invention;

[0024] Figure 6 is a structural diagram of the fusion diagram of the existing YOLOv10 and the above-mentioned modules according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The following further elaborates on the technical route of the present invention through specific embodiments and drawings.

[0026] As Figures 1 to 6 shown, an embodiment of the method for identifying metal defects trained with a small number of training samples based on deep learning according to the present invention includes the following steps:

[0027] 1. Collect image samples of various defective metals

[0028] Collect image samples at different environments, angles, and distances as the support set for training, with 30 samples for each category. Obtain the open-source industrial quality inspection dataset MVTec AD as the base class training set, and divide it into a training set, a test set, and a validation set. Among them, the training set has 4000 images, the test set has 1000 images, and the validation set has 1000 images, that is, the ratio of the training set, the test set, and the validation set is 0.8:0.1:0.1.

[0029] 2. Preprocess and annotate the images

[0030] Annotate the collected images using the PPOCRLabel annotation tool, and detect and feedback through the existing YOLOv10 model. Manually annotate and correct the annotation samples with poor detection results.

[0031] 3. Production of the training set based on data augmentation and enhancement

[0032] Use a physical rendering engine to generate highly reflective synthetic data to simulate different lighting conditions. Perform random frequency domain filtering on the input images to enhance the model's robustness to frequency domain features.

[0033] 4. Add the wavelet convolution M-WTConv module based on multi-level feature interaction extraction

[0034] To overcome the problem of insufficient feature extraction by the original ordinary convolution in YOLOv10, this patent adds a wavelet convolution module M-WTConv based on multi-level feature interaction extraction to improve feature extraction. Specifically, parallel cascades of the M-WTConv module are added to the 1st and 3rd Conv modules in the backbone network of the original YOLOv10, and a feature addition operation is performed at the end, as Figure 2 and Figure 6 shown. The improved wavelet convolution increases the receptive field but has fewer parameters. Different frequency band components can be operated separately, so as to more effectively extract low-frequency and smooth features, and high-frequency detail features; layer-by-layer interaction realizes the deep integration of multi-scale and multi-direction features, which can not only capture the global shape of defects but also highlight local details, and has a good extraction effect on complex defects. Among them, the M-WTConv module consists of a wavelet decomposition layer, three component feature extraction layers, a multi-level interaction layer, a feature fusion layer, and an inverse transform layer in cascade.

[0035] Add a parallel cascade of the M-WTConv module to the first Conv module in the backbone network of the original YOLOv10. Specifically, as Figure 6 shown, the M-WTConv module described consists of a wavelet decomposition layer, three component feature extraction layers, a multi-level interaction layer, a feature fusion layer, and an inverse transform layer in cascade, as Figure 2 shown.

[0036] The input image uses a two-dimensional wavelet decomposition transform in the wavelet decomposition layer to generate a total of 4 components: LL, LH, HL, and HH. Among them, LL represents the low-frequency component and smooth features, LH represents the high-frequency component and horizontal detail features, HL represents the high-frequency component and vertical detail features, and HH represents the high-frequency component and diagonal detail features. Secondly, in the component feature extraction layer, the above four wavelet components are used for convolution operations to extract preliminary features, and independent convolution operations are performed on each component. The formula is as follows:

[0037]

[0038] After completing the independent convolution operations, the ReLU activation function is used for each component to enhance the non-linear representation ability.

[0039] Then, the output feature maps of the four components described above are input into the multi-level interaction layer. The multi-level interaction layer includes the update of high-frequency components and the update of low-frequency components. The update formula for high-frequency components is as follows:

[0040]

[0041]

[0042]

[0043] The update formula for low-frequency components is expressed as follows:

[0044]

[0045] Among them, represents the i-th feature component at time t, represents the i-th feature component at time t + 1, and i ∈ {LL, LH, HL, HH}.

[0046] After the above update operations, the high-frequency components fuse more low-frequency global information, enhancing the correlation between the defect edges and the background; the low-frequency components, through the feedback of high-frequency details, contain more local texture and edge information, better capturing the multi-scale and multi-directional characteristics of the defects. The above steps are repeated three times to gradually enhance the mutual representation ability of the low-frequency and high-frequency components, achieving the capture of defect features at different scales at different levels. Then, the updated components are input into the feature fusion layer and fused in a direct splicing form to achieve the purpose of information fusion without increasing the number of parameters, and further fused through the convolution layer. The formula is as follows:

[0047]

[0048] Among them, Conv represents the convolution operation, Concat represents the splicing operation, represents each feature component. Finally, FR Perform the inverse wavelet transform to obtain the output feature map.

[0049] 5. Add the windowed cross-attention DM-FAM module based on frequency-domain analysis, dynamic band selection, and dynamic frequency perception

[0050] To overcome the problems of the original YOLOv10's insufficient ability to extract small defects and insufficient ability to distinguish noise interference, this patent constructs a dynamic frequency-domain attention DM-FAM module based on frequency-domain analysis and dynamic band selection to improve feature extraction. Specifically, add the DM-FAM module before the two upsampling modules in the original YOLOv10 neck network, as Figure 3 shown. The dynamic frequency-domain attention based on band analysis and dynamic band selection uses frequency-domain analysis to enhance high-frequency defect signals, filters low-frequency noise through dynamic band selection, and enhances the feature expression ability through windowed cross-attention with dynamic frequency perception, balancing high-frequency details and global semantics, and effectively improving the feature extraction effect of various defects. Among them, the DM-FAM module consists of a grouped frequency-domain feature extraction module, a channel-aware band selection module, a grouping and merging and inverse transform module, and a windowed cross-attention module with dynamic frequency perception.

[0051] The input feature map is a spatial-domain feature map. Perform frequency-domain grouping on the input feature map and perform the fast Fourier transform to obtain the grouped frequency-domain representation. Grouped calculation can improve the operation speed, reduce unnecessary operations, and at the same time support parallel operations, improving the calculation efficiency. The grouped processing formula is as follows:

[0052]

[0053] In the above formula, G represents the total number of groups; g represents the index of the current group, g ∈ {1, 2,..., G}; C represents the total number of channels of the feature map, denotes the fast Fourier transform, and the formula is as follows:

[0054]

[0055] In the above formula, I represents the input feature map,

[0056] After obtaining the grouped transformed feature map, separate the amplitude spectrum A (g) (u, v, c) = |F (g) (u, v, c)|. The amplitude spectrum reflects the energy distribution of the image, and the high frequency corresponds to edge defects; the phase spectrum φ (g) (u, v, c) = ∠F (g) (u, v, c). The phase spectrum reflects the structural information of the image.

[0057] Generate an adaptive frequency band, which is realized by global average pooling (GAP) and a multi-layer perceptron (MLP), achieving channel-adaptive frequency band selection. While saving computational complexity, it retains key frequency band information and enhances the ability to extract defects. The formula is as follows:

[0058]

[0059] In the above formula, A (g) is the amplitude spectrum of the output feature map calculated by grouping, are the upper and lower bounds of the output frequency band.

[0060] According to the dynamically obtained upper and lower frequency band bounds above, perform Gaussian modeling on the frequency domain feature mask to better extract defect features. The formula is as follows:

[0061]

[0062] In the above formula, r m =(r l +r h ) / 2, Δr = r h -r l , β is the band relaxation factor, and Z is the normalization factor.

[0063] Apply the mask to the amplitude spectrum of each group. The formula is as follows:

[0064]

[0065] Perform inverse transformation on the spatial domain features, and we have:

[0066]

[0067] Merge the grouped features. The formula is as follows:

[0068]

[0069] Obtain the spectral energy in the spatial domain, which is obtained using the fast Fourier transform. The formula is as follows:

[0070] E freq = log(1 + |FFT2D(F spatial )|) (13)

[0071] Predict window information through the lightweight network WDNet. Specifically, the WDNet network mainly consists of a 3×3 convolutional layer, a ReLU activation function, a max pooling layer, a 3×3 convolutional layer, a ReLU activation function, an adaptive pooling layer, a fully connected layer, and a Sigmoid activation function, as Figure 4As shown, the fully connected layer transforms the number of parameters into [B, 3], and the Sigmoid activation function restricts its range to (0, 1), resulting in the formula:

[0072] w, h, ρ = WDNet(E freq ) (14)

[0073] In the above formula, w represents the window width in the frequency domain direction, h represents the window height in the spatial domain direction, and ρ represents the overlapping ratio between windows. The frequency domain axis and the spatial axis are non-uniformly divided according to the prediction parameters, that is, the window is dynamically divided according to the energy peak position and equally spaced division is maintained to retain local features.

[0074] Perform frequency-aware feature projection to generate the attention weights of the frequency domain features, and the formula is:

[0075]

[0076] In the above formula, GAP is global average pooling, and the MLP layer changes the number of channels of the network from C to 4C and then from 4C to C.

[0077] The query Query uses F filtered Weighted flattening with the α feature, and the calculation formula is:

[0078]

[0079] The key-value pair (Key, Value) is directly projected using a fully connected layer after window division, and the calculation formula is:

[0080]

[0081] At this time, the attention is calculated independently for each window, and the attention calculation formula is as follows:

[0082]

[0083] In the above formula, B is a learnable position bias matrix.

[0084] In order to enable information exchange between different windows, information fusion is performed, and the strategy of circular shift is used to achieve cross-window interaction. Specifically, the spatial domain features are Circular shift of pixels to re-divide the window, then re-calculate the attention of the new window, and finally restore the original position through reverse shift to achieve the effect of information interaction between different windows.

[0085] After obtaining the window attention, weighted fusion is performed on the feature outputs of each window attention, and the calculation formula is:

[0086]

[0087] In the above formula, λ w is a learnable weight parameter, w is the window index, and W is the number of windows. Finally, a residual connection is made between the output feature map and the original spatial domain feature map:

[0088] F result = F spatial + γ·F output (20)

[0089] In the above formula, γ is a learnable scaling factor, and F result is the final output feature map.

[0090] 6. Add a Semantic-Enhanced Cross-modal Fusion (SECF) module to improve the feature representation ability of the detection head in YOLOv10

[0091] In the metal surface detection task, especially in the few-shot learning scenario, the model often faces the problems of insufficient utilization of semantic information and difficulty in cross-modal feature alignment. In few-shot learning, the model needs to make full use of limited data information and at the same time enhance the semantic understanding ability of the target. To overcome these challenges, this patent constructs a Semantic-Enhanced Cross-modal Fusion (SECF) module, as Figure 5 shown, to improve the feature representation ability of the detection head in YOLOv10. The semantic-enhanced cross-modal fusion module refers to a cascaded structure composed of a visual feature texturing module, a semantic enhancement module combined with a large language model (LLM), and a cross-modal feature fusion module, which can effectively improve the feature expression ability and model generalization performance in the few-shot scenario.

[0092] Specifically, the SECF module first uses the visual feature textualization module to perform regional alignment on the visual feature map output by the detection head through RoIAlign, accurately extracts the target area features, uses the Transformer's self-attention mechanism to capture global context information, enhances the semantic expression of features, and combines the multimodal fusion capability of the BLIP model to efficiently align and fuse visual features with semantic information to generate a target text description; then, the generated text description is input into the semantic enhancement module combined with the large language model LLM. This module is based on the BERT architecture and captures the context information of the text and encodes the text description through masked language modeling MLM and next sentence prediction NSP tasks to generate deep semantic embedding and semantic enrichment. The semantic vector output by BERT is further processed through the fully connected layer FC-Feature to adapt to the needs of downstream tasks; finally, the cross-modal feature fusion module aligns the visual features through FC-Vision-Align to make them consistent with the semantic feature dimensions, and then uses Cross-Attention to capture the correlation information between the two through a two-way attention mechanism to achieve efficient interaction between visual features and semantic features. On this basis, the fused features are refined through FC-Feature-Output, and the fused features after normalization and feature refinement are combined with the initial input feature map by element-by-element addition to generate the final feature representation. The specific implementation steps of SECF are as follows:

[0093] (1) Visual feature textualization module

[0094] Regional feature extraction based on RoIAlign: Input the visual feature map output by the detection head Where C is the number of channels, H×W is the spatial dimension, given N region proposals, each region is defined by the bounding box coordinates (x1, y1, x2, y2), where (x1, y1) is the upper left corner coordinate and (x2, y2) is the lower right corner coordinate;

[0095] Use bilinear interpolation sampling to dynamically adjust the pooling position according to the offset. The sampling process refers to the bilinear interpolation formula of MaskR-CNN:

[0096] R(n,c,i,j)=(x,y)∈bin∑Fv(c,x+Δx,y+Δy)·max(0,1-∣x′-x∣)·max(0,1-∣y′-y∣) (21)

[0097] Then the lightweight visual encoding LVE is used to map the regional features into a visual description vector Use global average pooling GAP to compress the spatial dimension to 1×1:

[0098]

[0099] The depthwise separable convolution DWConv reduces the computational amount and extracts the inter-channel correlation. The formula is as follows:

[0100] V d = DWConv(R gap )·W e (23)

[0101] where is a learnable projection matrix.

[0102] The BLIP-based text generator converts the visual description vector V d into the text description T = {t1, t2,..., t N}. The text generation formula is:

[0103] t n = BLIP-Generator(V d (n) ) (24)

[0104] where t n is the text description of the nth region proposal.

[0105] (2) Semantic enhancement module combined with the large language model LLM

[0106] The input is the text description T = {t1, t2,..., t N} generated by the BLIP-based text generator. Based on the BERT-based semantic vector generation, the text description is encoded into a semantic vector through the BERT model For text encoding, there is:

[0107]

[0108] where the output dimension is 768.

[0109] For dimension alignment, project the BERT output into the space aligned with the visual features, which can be expressed as follows:

[0110]

[0111] where is a learnable matrix.

[0112] For cross-modal alignment optimization, perform contrastive learning and design the contrastive loss function L align :

[0113]

[0114] where The visual description vector proposed for the i-th region The visual description vector proposed for the i-th region, where τ is the temperature coefficient (by default, τ = 0.07).

[0115] (3) Cross-modal attention fusion module

[0116] The input is the visual description vector and the semantic vector

[0117] The dual-stream cross attention (DCA) includes visual-to-semantic (V2S) attention and semantic-to-visual (S2V) attention. The visual-to-semantic (V2S) attention can be expressed as:

[0118]

[0119] The semantic-to-visual (S2V) attention can be expressed as:

[0120]

[0121] where W q , W k , W v , W q ′, W k ′, are learnable parameters.

[0122] Feature concatenation: Concatenate the outputs of the visual-to-semantic V2S attention and the semantic-to-visual S2V attention to generate the fused feature which is expressed by the following formula:

[0123] F cat = Concat(Attention V2S , Attention S2V ) (30)

[0124] Based on the feature refinement of the fully connected layer, input the concatenated feature F cat into the fully connected layer FC-Feature-Output for feature refinement and dimension alignment:

[0125] F refined = FC-Feature-Output(F cat ) (31)

[0126] where FC-Feature-Output is the fully connected layer, and the weight matrix outputs the feature

[0127] The residual connection and layer normalization refine the feature F refined with the original visual feature F v to perform a residual connection and generate the final output feature through layer normalization LayerNorm

[0128] F f = LayerNorm(F v + F refined ) (32)

[0129] Throughout the process, the visual features output by the detection head are converted into text descriptions through visual feature textification, and the powerful semantic understanding ability of the LLM is used to generate high-quality semantic vectors, making up for the lack of semantic information in few-shot data. At the same time, through cross-modal feature fusion, visual features and semantic features are efficiently interacted to generate more discriminative fusion features, enabling the network to better learn target features in few-shot scenarios, thereby improving the detection accuracy and generalization ability.

[0130] 7. Two-stage progressive training to improve the neural network for metal defect recognition

[0131] The model training is divided into a pre-training stage and a fine-tuning training stage; in the pre-training stage, the backbone network and the neck network are trained on a large scale using normal samples and synthetic defect samples from the industrial quality inspection dataset MVTecAD. In the fine-tuning training stage, the support set samples collected for the recognition task are used to optimize the meta-learning parameters and the classification head using the N-way K-shot training method to achieve the learning effect of training with a small number of training samples.

[0132] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0133] (1) Based on the original YOLOv10 network, in all convolutional modules of the backbone network, the improved wavelet convolution M-WTConv module for multi-level feature interaction extraction, which is composed of a cascade of a wavelet decomposition layer, three component feature extraction layers, a multi-level interaction layer, a feature fusion layer, and an inverse transform layer, is added in parallel to enhance the mutual representation ability of low-frequency and high-frequency components and capture defect features of different scales at different levels. Then, the updated components are input into the feature fusion layer and fused in a direct splicing form to achieve the purpose of information fusion without increasing the number of parameters.

[0134] (2) Based on the original YOLOv10 network, the present invention adds a dynamic frequency-domain attention DM-FAM module based on frequency-domain analysis and dynamic band selection before the two upsampling layers in the neck network. The dynamic frequency-domain attention based on band analysis and dynamic band selection enhances high-frequency defect signals through frequency-domain analysis, filters low-frequency noise through dynamic band selection, and enhances the feature expression ability through windowed cross-attention with dynamic frequency perception, balancing high-frequency details and global semantics, and effectively improving the feature extraction effect of various defects. Among them, the DM-FAM module consists of a grouped frequency-domain feature extraction module, a channel-aware band selection module, an inverse transform grouping and merging module, and a windowed cross-attention module with dynamic frequency perception.

[0135] (3) Based on the existing YOLOv10 detection head, the present invention adds a module composed of three modules: visual feature texturization, semantic enhancement combined with a large language model LLM, and cross-modal attention fusion. Through RolAlign pooling, lightweight visual coding, semantic vector generation, a two-stream cross-attention mechanism, and residual connections, accurate modeling of the target spatial distribution, semantic context information, and cross-modal features is achieved, and feature fusion is performed on the results. The processed feature map is connected to the input feature map through a residual connection, realizing enhanced feature representation ability, improved semantic understanding ability, optimized computational efficiency, and enhanced few-shot adaptability. Through the above improvements, the SECF module significantly improves the performance of the YOLOv10 model in the few-shot metal defect detection task.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or substitutions can still be made to the specific implementation methods of the present invention, and any modifications or substitutions that do not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A method for identifying metal defects trained with a small number of training samples, characterized in that First, obtain higher-quality training samples and recognition data through enhanced data preprocessing methods. Then, construct a deep learning neural network that fuses YOLOv10 with a wavelet convolution module, dynamic frequency domain attention, and a semantic enhancement cross-modal fusion module SECF. Use two-stage progressive training to specifically enhance the detection and recognition speed and accuracy of metal surface defects, including the following steps: (1) Collect image samples of various defective metals; (2) Preprocess and annotate the images; (3) Produce a training set based on data augmentation and enhancement; (4) Add a wavelet convolution M-WTConv module based on multi-level feature interaction extraction; (5) Add a windowed cross-attention DM-FAM module based on frequency domain analysis, dynamic band selection, and dynamic frequency perception; (6) Add a semantic enhancement cross-modal fusion module SECF to improve the feature representation ability of the detection head in YOLOv10; (7) Two-stage progressive training is used to improve the neural network to complete metal defect recognition; The collection of image samples of various defective metals refers to collecting image samples of defective metals of six types: bubbles, scratches, oxidation spots, oil stains, cracks, and depressions, with 30 images collected for each type; The preprocessing and annotation of the images refer to using the Labelme image annotation tool to annotate the corresponding defects of the collected images; The production of a training set based on data augmentation and enhancement refers to performing random frequency domain filtering on the input images to enhance the robustness of the model to frequency domain features; The two-stage progressive training is used to improve the neural network to complete metal defect recognition, which means dividing the model training into a pre-training stage and a fine-tuning training stage; The pre-training stage refers to using the normal samples and synthetic defect samples of the industrial quality inspection dataset MVTecAD to conduct large-scale training on the improved neural network; The fine-tuning training stage refers to using the support set samples collected for the recognition task to optimize the meta-learning parameters and classification head using the N-way K-shot training method to achieve the learning effect of training with a small number of training samples; The improved neural network refers to the deep neural network obtained by modifying the network structure in steps (4) to (6).

2. The metal defect recognition method trained with a small number of training samples according to claim 1, characterized in that, The addition of a wavelet convolution M-WTConv module based on multi-level feature interaction extraction means adding an improved wavelet convolution module in parallel to all convolution modules in the YOLOv10 backbone network. The improved wavelet convolution module refers to adding a multi-level feature interaction module on the basis of the original wavelet convolution module. The multi-level feature interaction module is composed of a cascade of a wavelet decomposition layer, three component feature extraction layers, a multi-level interaction layer, a feature fusion layer, and an inverse transform layer. The component feature extraction layer refers to performing convolution addition on the four different component feature maps output by the wavelet decomposition layer; The feature fusion layer refers to splicing the output feature maps of the multi-level feature interaction module and performing convolution to obtain the corresponding output feature maps.

3. A method for identifying metal defects trained with a small number of training samples according to claim 1, characterized in that The addition of the windowed cross-attention DM-FAM module based on frequency-domain analysis, dynamic band selection, and dynamic frequency perception refers to adding an improved dynamic frequency-domain attention module before the upsampling layer of the YOLOv10 neck network. The improved dynamic frequency-domain attention module is a network composed of four parts in series: a grouped frequency-domain feature extraction module, an adaptive band selection module, an inverse transform and grouped merging module, and a frequency-domain and spatial-domain fusion module. The grouped frequency-domain feature extraction module is a network composed of a grouped parallel module and a fast Fourier transform module. Among them, the grouped parallel module groups the number of channels of the image feature map according to the number of GPU cores of the device, and the fast Fourier transform module converts the input feature map from the spatial domain to the frequency domain and separates the amplitude spectrum and the phase spectrum. The adaptive band selection module is a network composed of a global average pooling layer, a multi-layer perceptron layer, and a dynamic frequency-domain feature mask layer in series. The dynamic frequency-domain feature mask layer is a mask model based on Gaussian distribution modeling. The inverse transform and grouped merging module refers to performing an inverse Fourier transform on the feature map output by the dynamic frequency-domain feature mask layer and merging all groups to obtain a new feature map. The frequency-domain-spatial-domain fusion module is a network composed of an energy extraction module, a window prediction WDNet module, a feature projection module, a cross-attention calculation module, and a dynamic window interaction fusion module in series. Among them, the energy extraction module refers to performing a fast Fourier transform on the input feature map to obtain an energy spectrum. The window prediction WDNet module is a network composed of a 3×3 convolutional layer, a ReLU activation function, a max pooling layer, a 3×3 convolutional layer, a ReLU activation function, an adaptive pooling layer, a fully connected layer, and a Sigmoid activation function in series. The feature projection module refers to obtaining dynamic weights by passing the feature map output by the WDNet module through an input global average pooling layer and an MLP layer, and then multiplying the value output by the full connection of the feature map of the inverse transform and grouped merging module to obtain the Query value in the cross-attention calculation module. At the same time, the value output by the full connection of the feature map of the inverse transform and grouped merging module is used as the Key and Value values in the cross-attention calculation module. The cross-attention calculation module refers to performing Softmax self-attention calculation. The dynamic window interaction fusion module refers to using a cyclic shift strategy to achieve cross-window interaction.

4. A method for identifying metal defects trained with a small number of training samples according to claim 1, characterized in that, The addition of the Semantic Enhancement Cross-modal Fusion Module (SECF) to improve the feature representation ability of the detection head in YOLOv10 means adding a cascade consisting of a visual feature texturization module, a semantic enhancement module of a large language model (LLM), and a cross-modal feature fusion module on the basis of the existing detection head of YOLOv10. Among them, the visual feature texturization module converts visual features into text descriptions, and the generated text descriptions are then used as inputs and cooperate with the semantic enhancement module of the large language model (LLM) to generate enhanced semantic features. The enhanced semantic features are integrated with other modal information through the cross-modal feature fusion module to form a unified feature representation. The visual feature texturization module refers to the text description of the region proposals generated through semantic conversion. The semantic enhancement module of the large language model (LLM) refers to encoding the text description into a semantic vector and inputting it into the semantic analysis module for semantic enhancement, and capturing the global context information and semantic features of the target. The cross-modal feature fusion module refers to combining the fused features with the original input features by element-wise addition and finally outputting the enhanced feature representation.

Images