Industrial defect detection method based on feature fusion and progressive supervision strategy
By adopting the feature extraction and information fusion strategy of Swin Transformer network and progressive fine decoding blocks and focus modules in industrial defect detection, the problem of insufficient feature extraction and information fusion capabilities is solved, which significantly improves detection efficiency and accuracy, and is suitable for Various industrial defect detection scenarios.
Patent Information
- Application Number
- CN202410976577.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-20
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art has insufficient feature extraction and information fusion capabilities in industrial defect detection, which makes it difficult to effectively distinguish small defect areas in the background, reducing detection efficiency.
Using multi-scale feature extraction based on Swin Transformer network, and introducing progressive fine decoding blocks (PR decoding blocks) and focus modules (Focus modules) into the decoder blocks, improving features layer by layer and monitoring layered losses. The ability to distinguish between defective areas and background areas, reducing false positives and false negatives.
It significantly improves the performance of industrial defect detection, improves the accuracy and robustness of segmentation results, and is better than the existing technology in indicators such as Dice coefficient, recall rate, accuracy, F1 score and cross-border ratio, and is suitable for various industries. Defect detection scenario.
Smart Images

Figure CN118967592A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an industrial defect detection method based on feature fusion and progressive supervision strategy, and belongs to the field of computer vision. Background Art
[0002] Image segmentation technology plays a vital role in the industrial field, which helps to divide images into semantically meaningful regions. Its application enables industrial enterprises to achieve automation, improve product quality, reduce costs, and enhance competitiveness.
[0003] Image segmentation technology is widely used in industrial environments. For example, it is widely used in industrial robot vision systems to improve the robot's ability to accurately identify and locate targets. In addition, image segmentation technology has also made significant contributions to industrial automation and monitoring [6]. By analyzing segmented images of the factory's internal environment and equipment status, companies can achieve real-time monitoring and optimization of the production process, thereby promptly discovering and solving potential problems and improving production efficiency and resource utilization. Notably, one of its most valuable applications is defect detection in assembly line production. Using image segmentation technology, industrial companies can quickly and accurately identify defects, flaws, or other non-conformities on the product surface, thereby effectively monitoring and controlling product quality. This is crucial for improving product quality, reducing after-sales service costs, and maintaining brand reputation.
[0004] However, image segmentation technology in the industrial field faces many challenges and problems. Deep learning segmentation models in the industrial field face problems such as insufficient data annotation, class imbalance, and decreased segmentation accuracy caused by overfitting. In general, the segmentation results have a large number of false positives and false negatives, as well as widespread boundary errors.
[0005] To address these challenges, this paper proposes Progressive Refinement U-Net (PRU-Net). Based on the SwinU-Net architecture, we redesigned the decoder module and added three Progressive Refinement Decoding Blocks (PR Decoding Blocks). By propagating and supervising hierarchical losses and gradually refining the segmentation results, we aim to improve the quality of the segmentation results while reducing boundary errors. At the same time, in each PR decoding block, we introduced a focusing module structure to help the model distinguish defective areas from normal areas more effectively, thereby reducing the occurrence of false positives and false negatives. Summary of the invention
[0006] In order to address the deficiencies of the prior art, the purpose of the present invention is to provide an industrial defect detection method based on feature fusion and progressive supervision strategy, so as to solve the problem that its feature extraction and information fusion capabilities are not sufficient to effectively distinguish small defect areas in the background, thereby reducing the efficiency in industrial defect detection tasks.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: an industrial defect detection method based on feature fusion and progressive supervision strategy, comprising the following steps:
[0008] S1. Based on the Swin U-Net architecture, the Swin Transformer network is used as the encoder to extract multi-scale features of the input image. The Swin Transformer network uses a hierarchical structure and a self-attention mechanism of a sliding window to efficiently capture local and global information in the image.
[0009] S2. Each decoder block contains two Focus modules, which improve the ability to distinguish defect areas from background areas by refining features layer by layer and supervising at each decoding stage. The Focus module achieves accurate positioning of defect areas and reduces false detection and missed detection by fusing deep semantic features, shallow semantic features and segmentation prediction.
[0010] The S3, PR decoder block upsamples and refines features layer by layer, using the multi-scale feature information extracted in the encoder to enable the model to more accurately segment defective areas in the image. Each PR decoder block not only contains standard convolution operations, but also combines a Focus module to strengthen attention to defective areas and ensure that defective areas can be accurately located at different scales. Through multi-level feature fusion and refinement, the final segmentation result generated has high resolution and accuracy, effectively improving the performance of industrial defect detection.
[0011] S4. Training and testing were performed on the MVTecAnomaly Detection (MVTecAD) dataset. The results showed that the method of the present invention is superior to the existing technology in terms of Dice coefficient, recall rate, precision, F1 score and intersection over union (IoU), and is suitable for various industrial defect detection scenarios.
[0012] Furthermore, the aforementioned step S1 is specifically as follows:
[0013] First, based on image data in various industrial scenarios, the Swin Transformer network is used as the encoder for multi-scale feature extraction. The Swin Transformer block is different from the traditional multi-head self-attention (MSA) module. It is built on a moving window mechanism. Each Swin Transformer block consists of a layer normalization (LN) layer, a multi-head self-attention module, a residual connection, and a two-layer multi-layer perceptron (MLP) with GELU non-linear activation. In two consecutive Swin Transformer blocks, the window-based multi-head self-attention (W-MSA) module and the moving window-based multi-head self-attention (SW-MSA) module are applied successively. First, the input features are processed by layer normalization and the feature representation is calculated by the window multi-head self-attention module.
[0014]
[0015] Then, the representation is input into the multilayer perceptron to obtain z l :
[0016]
[0017] Next, the output feature z of the previous step is processed by layer normalization l , and a new feature representation is calculated by moving the window multi-head self-attention module
[0018]
[0019] Finally, the representation is processed again through a multilayer perceptron to obtain z l+1 :
[0020]
[0021] The calculation formula of the self-attention mechanism is as follows: (e')
[0022]
[0023] in, denote the query matrix, key matrix and value matrix respectively, M 2 represents the number of patches in the window, and d represents the dimension of the query or key. B is the bias matrix Through the above steps, the SwinTransformer block can effectively capture the local to global feature representation and use the moving window mechanism to enhance the continuity and context relevance of feature learning.
[0024] Furthermore, the aforementioned step S2 is specifically as follows:
[0025] First, the deep semantic feature F H and segmentation prediction F seg Perform double upsampling to ensure the resolution of the feature map is consistent, and obtain the upsampled feature F′ H and F′ seg In the first branch, the shallow semantic features F L and the upsampled segmentation prediction F′ seg Perform element-wise multiplication to obtain a feature map that emphasizes defect areas:
[0026] F L ⊙F′ seg
[0027] Then, the feature map is processed by the cross entropy block (CE Block) to calculate its cross entropy value, which is used to further enhance the features of the defect area, and the processed features are multiplied by the weight parameter α for adjustment. The formula is:
[0028] α×CE Block(F L ⊙F′ seg )
[0029] Finally, the upsampled deep semantic feature F′ is subtracted from the above features H , get the feature F1 to reduce the influence of the background area, the formula is:
[0030] F1=α×CE Block(F′ seg ⊙F L )-F′ H
[0031] In the second branch, the shallow semantic features F L And the upsampled reverse segmentation prediction 1-F′ seg Perform element-wise multiplication to obtain a feature map that emphasizes the background area:
[0032] F L ⊙(1-F′ seg )
[0033] Subsequently, the feature map is processed by the cross entropy block to calculate its cross entropy value, which is used to further enhance the features of the background area, and the processed features are multiplied by the weight parameter β for adjustment. The formula is:
[0034] β×CE Block(F L ⊙(1-F′ seg )
[0035] Finally, feature F2 is obtained, which is used to emphasize the background area in the final fusion. The formula is:
[0036] F2=β×CE Block((1-F′ seg )⊙F L )
[0037] Finally, the features of the first and second branches are subjected to element-wise XOR operation to obtain the output out of the Focus module, which is:
[0038]
[0039] Through this design, the Focus module can effectively fuse feature information at different levels, strengthen attention to defect areas, and reduce false detection and missed detection, thereby improving the performance of the overall model. The PR decoder block generates high-precision segmentation results by refining features layer by layer and combining the multi-scale features of the encoder, providing reliable technical support for industrial defect detection.
[0040] Furthermore, the aforementioned step S3 is specifically as follows:
[0041] First, the PR decoder block receives the segmentation result F from the previous decoder. seg , decoder features e D and the encoder features e at the corresponding level E In order to ensure the consistency of the resolution of the feature map, the decoder feature e D and encoder features e E Perform double upsampling to obtain e′ D and e′ E .
[0042] In the first branch, the segmentation result F seg , decoder features e D and encoder features e E Input to the Focus module FM1 to obtain feature z1:
[0043] z1=FM1(F seg ,CBlock(e E ),e D )
[0044] Next, z1 and the upsampled encoder are embedded into CBlock(e′ E ) to perform cascade operations:
[0045] z1←[z1;CBlock(e′ E )]
[0046] Then, the concatenated features are convolved twice to obtain the refined feature representation:
[0047] z1′←CBlock(CBlock(z1))
[0048] In the second branch, the segmentation result F seg , the upsampled encoder embedding e′ E and decoder embedding e′ D Input to the second Focus module FM2 to generate feature z2:
[0049] z2=FM2(F seg ,CBlock(e′ E ),CBlock(e′ D ))
[0050] Then, the refined features generated by the first branch are concatenated with the features generated by the second branch:
[0051] z=[z1′;z2]
[0052] Finally, through three convolution operations, the cascaded features are further extracted and fused to generate the final output out:
[0053] out=CBlock(CBlock(CBlock(z)))
[0054] In order to guide F layer by layer seg The generation process of each layer of the PR decoder block calculates F seg and the actual segmentation result y gt The loss between them is as follows:
[0055]
[0056] Through multi-level feature fusion and refinement, the final segmentation result generated has high resolution and accuracy, which effectively improves the performance of industrial defect detection.
[0057] Compared with the prior art, the technical solution adopted by the present invention has the following technical effects:
[0058] The present invention proposes an industrial defect detection method based on feature fusion and progressive supervision strategy to improve the performance of industrial defect detection. PR U-Net is based on the Swin U-Net architecture and is improved by integrating PR decoding blocks and focus modules (FM). The PR decoding block allows for gradual refinement of features and supervision at each decoding stage, while the focus module enhances the model's ability to distinguish between defective and non-defective areas through feature fusion and segmentation prediction. The design of these modules not only improves the ability of feature extraction and information fusion, but also effectively reduces the occurrence of false positives and false negatives, thereby improving the accuracy and robustness of segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 It is a flow chart of the method of the present invention.
[0060] Figure 2 Example of comparing segmentation results. DETAILED DESCRIPTION
[0061] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention:
[0062] like Figure 1 As shown, the present invention proposes an industrial defect detection method based on feature fusion and progressive supervision strategy. It includes the following steps:
[0063] S1. First, based on image data in various industrial scenarios, the Swin Transformer network is used as the encoder for multi-scale feature extraction. The Swin Transformer block is different from the traditional multi-head self-attention (MSA) module. It is built on a moving window mechanism. Each Swin Transformer block consists of a layer normalization (LN) layer, a multi-head self-attention module, a residual connection, and a two-layer multi-layer perceptron (MLP) with GELU non-linear activation. In two consecutive Swin Transformer blocks, the window-based multi-head self-attention (W-MSA) module and the moving window-based multi-head self-attention (SW-MSA) module are applied successively. First, the input features are processed by layer normalization and the feature representation is calculated by the window multi-head self-attention module.
[0064]
[0065] Then, the representation is input into the multilayer perceptron to obtain z l :
[0066]
[0067] Next, the output feature z of the previous step is processed by layer normalization l , and is calculated by a moving window multi-head self-attention module
[0068] Calculate the new feature representation
[0069]
[0070] Finally, the representation is processed again through a multilayer perceptron to obtain z l+1 :
[0071]
[0072] The calculation formula of the self-attention mechanism is as follows: (e')
[0073]
[0074] in, denote the query matrix, key matrix and value matrix respectively, M 2 Indicates the number of patches in the window, d
[0075] represents the dimension of the query or key. B is the bias matrix Through the above steps, the SwinTransformer block can effectively capture the local to global feature representation and use the moving window mechanism to enhance the continuity and context relevance of feature learning.
[0076] S2. First, the deep semantic feature F H and segmentation prediction F seg Perform double upsampling to ensure consistent resolution of feature maps.
[0077] F L ⊙F′ seg
[0078] Then, the feature map is processed by the cross entropy block (CE Block) to calculate its cross entropy value, which is used to further enhance the features of the defect area, and the processed features are multiplied by the weight parameter α for adjustment. The formula is:
[0079] α×CE Block(F L ⊙F′ seg )
[0080] Finally, the upsampled deep semantic feature F′ is subtracted from the above features H , get the feature F1 to reduce the influence of the background area, the formula is:
[0081] F1=α×CE Block(F′ seg ⊙F L )-F′ H
[0082] In the second branch, the shallow semantic features F L And the upsampled reverse segmentation prediction 1-F′ seg Perform element-wise multiplication to obtain a feature map that emphasizes the background area:
[0083] F L ⊙(1-F′ seg )
[0084] Subsequently, the feature map is processed by the cross entropy block to calculate its cross entropy value, which is used to further enhance the features of the background area, and the processed features are multiplied by the weight parameter β for adjustment. The formula is:
[0085] β×CE Block(F L ⊙(1-F′ seg )
[0086] Finally, feature F2 is obtained, which is used to emphasize the background area in the final fusion. The formula is:
[0087] F2=β×CE Block((1-F se ' g )⊙F L )
[0088] Finally, the features of the first and second branches are subjected to element-wise XOR operation to obtain the output out of the Focus module, which is:
[0089]
[0090] Through this design, the Focus module can effectively fuse feature information at different levels, strengthen attention to defect areas, and reduce false detection and missed detection, thereby improving the performance of the overall model. The PR decoder block generates high-precision segmentation results by refining features layer by layer and combining the multi-scale features of the encoder, providing reliable technical support for industrial defect detection.
[0091] S3. First, the PR decoder block receives the segmentation result F from the previous decoder. seg , decoder features e D and the encoder features e at the corresponding level E In order to ensure the consistency of the resolution of the feature map, the decoder feature e D and encoder features e E Perform double upsampling to obtain e′ D and e′ E .
[0092] In the first branch, the segmentation result F seg , decoder features e D and encoder features e E Input to the Focus module FM1 to obtain feature z1:
[0093] z1=FM1(F seg ,CBlock(e E ),e D )
[0094] Next, z1 and the upsampled encoder are embedded into CBlock(e′E ) to perform cascade operations:
[0095] z1←[z1;CBlock(e′ E )]
[0096] Then, the concatenated features are convolved twice to obtain the refined feature representation:
[0097] z1′←CBlock(CBlock(z1))
[0098] In the second branch, the segmentation result F seg , the upsampled encoder embedding e′ E and decoder embedding e′ D Input to the second Focus module FM2 to generate feature z2:
[0099] z2=FM2(F seg ,CBlock(e′ E ),CBlock(e′ D ))
[0100] Then, the refined features generated by the first branch are concatenated with the features generated by the second branch:
[0101] z=[z1′;z2]
[0102] Finally, through three convolution operations, the cascaded features are further extracted and fused to generate the final output out:
[0103] out=CBlock(CBlock(CBlock(z)))
[0104] In order to guide F layer by layer seg The generation process of each layer of the PR decoder block calculates F seg and the actual segmentation result y gt The loss between them is as follows:
[0105]
[0106] Through multi-level feature fusion and refinement, the final segmentation result generated has high resolution and accuracy, which effectively improves the performance of industrial defect detection.
[0107] S4. Training and testing were performed on the MVTecAnomaly Detection (MVTecAD) dataset. The results showed that the method of the present invention is superior to the existing technology in terms of Dice coefficient, recall rate, precision, F1 score and intersection over union (IoU), and is suitable for various industrial defect detection scenarios.
[0108] In the present invention, the MVTec Anomaly Detection (MVTecAD) dataset is used, which is a widely used dataset for detecting surface defects in industrial manufacturing. The MVTecAD dataset was created by MVTec Software GmbH and contains various types of high-resolution images that contain various surface anomalies such as cracks, dents, scratches, and spot defects. These images cover a variety of materials including plastics, paper, textiles, and metals. The MVTecAD dataset is intended to promote the research of industrial surface defect detection algorithms as a benchmark for evaluating the performance of algorithms in actual industrial scenarios. Each type of image is divided into a training set and a test set. The training set contains normal samples and abnormal samples, and the test set contains only abnormal samples.
[0109] This example selects four methods for comparison to evaluate the effectiveness of PR U-Net in industrial defect detection using computer vision. These methods are classic models and SOTA models for image segmentation in the field of computer vision, including ViT, TransUnet, Swin U-Net, and SAFM Res Unet (SOTA).
[0110] Dice, F1 score, and IoU are used as quantitative indicators to evaluate the effectiveness of various models. The quantitative evaluation results are shown in Table 1.
[0111] Table 1 Comparison results of different classic models
[0112]
[0113] The following conclusions can be drawn from Table 1:
[0114] PR U-Net significantly outperforms the existing benchmark model Swin U-Net and other classic segmentation models such as ViT and TransUnet in all evaluation indicators. In addition, PR U-Net performs well in accuracy, recall, precision, F1 score and IoU, especially in the segmentation ability of small defect areas, which has achieved significant improvement.
[0115] By comparing the SAFMResUnet model, we further verified the effectiveness of the PR decoding block and the focus module in improving model performance. The design of these modules not only improves the ability of feature extraction and information fusion, but also effectively reduces the occurrence of false positives and false negatives, thereby improving the accuracy and robustness of segmentation.
[0116] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.
Claims
1. An industrial defect detection method based on feature fusion and progressive supervision strategy, characterized in that: The steps include: S1. Based on the Swin U-Net architecture, the Swin Transformer network is used as the encoder to extract multi-scale features of the input image. The Swin Transformer network uses a hierarchical structure and a self-attention mechanism of a sliding window to efficiently capture local and global information in the image. S2. Each decoder block contains two Focus modules, which improve the ability to distinguish defect areas from background areas by refining features layer by layer and supervising at each decoding stage. The Focus module achieves accurate positioning of defect areas and reduces false detection and missed detection by fusing deep semantic features, shallow semantic features and segmentation prediction. S3, PR decoder block upsamples and refines features layer by layer, and uses the multi-scale feature information extracted from the encoder to enable the model to more accurately segment the defective areas in the image. Each PR decoder block not only contains standard convolution operations, but also combines the Focus module to strengthen the attention to the defective area and ensure that the defective area can be accurately located at different scales. Through multi-level feature fusion and refinement, the final segmentation result generated has high resolution and accuracy, which effectively improves the performance of industrial defect detection; S4. Training and testing were performed on the MVTec Anomaly Detection (MVTec AD) dataset. The results showed that the method of the present invention is superior to the existing technology in terms of Dice coefficient, recall rate, precision, F1 score and intersection over union (IoU), and is suitable for various industrial defect detection scenarios.
2. The industrial defect detection method based on feature fusion and progressive supervision strategy according to claim 1 is characterized in that Swin Transformer encoder, step S1 is specifically: Firstly, based on image data in various industrial scenarios, the Swin Transformer network is used as the encoder for multi-scale feature extraction. The Swin Transformer block is different from the traditional multi-head self-attention (MSA) module. It is built based on a moving window mechanism. Each Swin Transformer block consists of a layer normalization (LN) layer, a multi-head self-attention module, a residual connection, and a two-layer multi-layer perceptron (MLP) with GELU nonlinear activation. In two consecutive Swin Transformer blocks, the window-based multi-head self-attention (W-MSA) module and the moving window-based multi-head self-attention (SW-MSA) module are applied successively. Firstly, the input features are processed by layer normalization and the feature representation is calculated by the window multi-head self-attention module. Then, the representation is input into the multilayer perceptron to obtain z l : Next, the output feature z of the previous step is processed by layer normalization l , and a new feature representation is calculated by moving the window multi-head self-attention module Finally, the representation is processed again through a multilayer perceptron to obtain z l+1 : The calculation formula of the self-attention mechanism is as follows: (e') in, denote the query matrix, key matrix and value matrix respectively, M 2 represents the number of patches in the window, and d represents the dimension of the query or key. B is the bias matrix Through the above steps, the SwinTransformer block can effectively capture the local to global feature representation and use the moving window mechanism to enhance the continuity and context relevance of feature learning.
3. The industrial defect detection method based on feature fusion and progressive supervision strategy according to claim 1 is characterized in that The Focus module included in the decoder, step S2 is specifically: First, the deep semantic feature F H and segmentation prediction F seg Perform double upsampling to ensure the resolution of the feature map is consistent, and obtain the upsampled feature F′ H and F′ seg In the first branch, the shallow semantic features F L and the upsampled segmentation prediction F′ seg Perform element-wise multiplication to obtain a feature map that emphasizes defect areas: F L ⊙F′ seg Then, the feature map is processed by the cross entropy block (CE Block) to calculate its cross entropy value, which is used to further enhance the features of the defect area, and the processed features are multiplied by the weight parameter α for adjustment. The formula is: α×CE Block(F L ⊙F′ seg ) Finally, the upsampled deep semantic feature F′ is subtracted from the above features H , get the feature F1 to reduce the influence of the background area, the formula is: F1=α×CE Block(F′ seg ⊙F L )-F′ H In the second branch, the shallow semantic features F L And the up-sampled reverse segmentation prediction 1-F′ seg Perform element-wise multiplication to obtain a feature map that emphasizes the background area: F L ⊙(1-F′ seg ) Subsequently, the feature map is processed by the cross entropy block to calculate its cross entropy value, which is used to further enhance the features of the background area, and the processed features are multiplied by the weight parameter β for adjustment. The formula is: β×CE Block(F L ⊙(1-F′ seg ) Finally, feature F2 is obtained, which is used to emphasize the background area in the final fusion. The formula is: F2=β×CE Block((1-F′ seg )⊙F L ) Finally, the features of the first and second branches are subjected to element-wise XOR operation to obtain the output out of the Focus module, which is: The PR decoder block generates segmentation results by refining features layer by layer and combining the multi-scale features of the encoder.
4. The industrial defect detection method based on feature fusion and progressive supervision strategy according to claim 1 is characterized in that PR decoder module, step S3 is specifically: First, the PR decoder block receives the segmentation result F from the previous decoder. seg , decoder features e D and the encoder features e at the corresponding level E In order to ensure the consistency of the resolution of the feature map, the decoder feature e D and encoder features e E Perform double upsampling to obtain e′ D and e′ E , In the first branch, the segmentation result F seg , decoder features e D and encoder features e E Input to the Focus module FM1 to obtain feature z1: z1=FM1(F seg ,CBlock(e E ),e D ) Next, z1 and the upsampled encoder are embedded into CBlock(e′ E ) to perform cascade operations: z1←[z1;CBlock(e′ E )] Then, the concatenated features are convolved twice to obtain the refined feature representation: z′1←CBlock(CBlock(z1)) In the second branch, the segmentation result F seg , the upsampled encoder embedding e′ E and decoder embedding e′ D Input to the second Focus module FM2 to generate feature z2: z2=FM2(F seg ,CBlock(e′ E ),CBlock(e′ D )) Then, the refined features generated by the first branch are concatenated with the features generated by the second branch: z = [z′1; z2] Finally, through three convolution operations, the cascaded features are further extracted and fused to generate the final output out: out=CBlock(CBlock(CBlock(z))) In order to guide F layer by layer seg The generation process of each layer of the PR decoder block calculates F seg and the actual segmentation result y gt The loss between them is as follows: The model is trained using this loss function.
Citation Information
Cited By
Aluminum profile surface defect detection method based on DMSA-Swinin-Unet network model
CN120339262A
Low-voltage electric appliance defect positioning method and system based on dynamic weighted feature fusion
CN120953283A
