A multi-task appearance defect recognition method based on detection and segmentation feature interaction

Abstract

The application provides a multi-task appearance defect recognition method based on detection and segmentation feature interaction, and belongs to the technical fields of computer vision, machine learning and the like.Based on a detection and segmentation joint framework Mask RCNN, a deep feature fusion module with an attention mechanism and a bidirectional FPN structure is designed to fully fuse information of feature maps with different resolutions; a feature reconstruction module with context aggregation is designed to represent defect features of various scales; a feature interaction network composed of a position interaction module and a channel interaction module is designed to decouple task sharing feature maps into feature maps for specific tasks, fully interact channel information and position information between different tasks, and improve the feature representation capability of task feature maps.The method can greatly improve the appearance defect recognition performance by fully fusing information of the feature maps, fully utilizing the features of the segmentation branch, and combining the detection and segmentation feature interaction.

Description

Technical Field

[0001] This invention belongs to the technical fields of computer vision and machine learning, and specifically relates to a multi-task joint appearance defect recognition algorithm based on deep learning. Background Technology

[0002] Industrial appearance defect identification is a crucial task combining computer vision with actual industrial production. It enables the monitoring of various industrial environments during production and use, promptly identifying appearance defects, their types, and locations. This lays a solid foundation for subsequent advanced tasks such as repair and recycling. Currently, industrial appearance defect identification technology is widely used in industrial consumer goods, highway bridges, and material components, demonstrating broad application prospects and significant market value. Therefore, researching industrial appearance defect identification technology and its applications in actual industrial production is of great importance.

[0003] Generally, the requirements for defect identification can be divided into three levels: what is the defect, where is the defect, and how many defects are there. The task of determining "what is the defect" corresponds to classification in computer vision, and can be called "defect classification." Generally, it only requires identifying the type of defect an image belongs to. The task of identifying "where the defect is" is a further step, corresponding to detection, and can be called "defect detection." This requires not only identifying the category of the defect in the image but also locating its specific position. The next step, "how many defects are there," corresponds to segmentation, and can be called "defect segmentation." This requires determining whether each pixel in the image is a defect or, more specifically, what type of defect it belongs to. Although the goals and requirements of these three stages differ, they can sometimes be converted and combined. In this invention, they are collectively referred to as "defect identification."

[0004] Object detection algorithms in computer vision not only focus on the category of each object in an image but also on its location within the image. This aligns perfectly with the "defect detection" task in industrial appearance defect recognition. Therefore, many well-developed object detection algorithms have been rationally applied to the field of "defect detection," with the most widely used being the two-stage method represented by Faster R-CNN and the one-stage method represented by the YOLO (You Only Look Once) series. The first stage of the two-stage method generates candidate regions that may contain defects, and then classifies and regresses these candidate regions; while the one-stage method directly classifies and locates defects based on the features extracted by the network.

[0005] Defects in industrial production environments are typically complex, with significant variations in shape and size. Detection-based methods can only provide the outermost bounding box of the defect, making it difficult to accurately describe its actual location and number, which is detrimental to subsequent identification and repair. Therefore, image segmentation methods that can obtain pixel-level defect locations have become increasingly popular, with semantic segmentation methods being the most widely used in the industrial defect field. Defect segmentation treats the defect region as a region of interest, separating it from the normal background region. It can not only classify, locate, and segment various defects but also simultaneously obtain attributes such as the defect's area and shape. Most defect segmentation networks are improvements on mature semantic segmentation networks, such as FCN, UNet, the Deeplab series, and PSPNet.

[0006] Depending on the needs of defect recognition, multiple tasks are often combined, such as classification and detection or segmentation. However, the most widely used approach is multi-task learning that combines joint detection and segmentation. Mask R-CNN is a classic image instance segmentation method, and many works have applied it to the field of defect recognition, designing different strategies to improve performance for specific tasks. However, we have found that directly applying Mask R-CNN to industrial appearance defect recognition presents a series of problems.

[0007] Mask R-CNN employs a ResNet-FPN architecture, constructing a path for information to flow from deep layers to shallow layers, attempting to improve the detection capability of small objects by fusing deep and shallow features. However, shallow features, including edge shapes and other characteristics, are very useful for the accurate detection and segmentation of large objects. But in the current ResNet-FPN structure, shallow features, in order to flow to the deep features responsible for large object detection and segmentation, pass through dozens of network layers, losing a significant amount of information. The shape and edge information contained in the shallow features almost completely disappears after flowing to the deeper layers, leaving only strong semantic information in the deeper layers, resulting in poor detection and segmentation performance for large objects.

[0008] The segmentation branch of Mask R-CNN is an improvement based on FCN. Currently, the segmentation branch uniformly aligns all RoIs to a size of 14×14. However, in industrial appearance defect recognition, defect sizes are not fixed, and many defects are quite large. Therefore, a 14×14 region is insufficient to fully represent the defect features. Subsequent upsampling and interpolation to the original RoI size results in significant information loss. Furthermore, during testing, the RoI input for the segmentation branch is the detection region of the detection branch, leading to the inability to segment areas outside the detection bounding box.

[0009] Furthermore, Mask R-CNN suffers from competitive learning due to defect detection and defect segmentation sharing the same feature map. Specifically, defect detection requires coarser-grained features for region localization, while defect segmentation demands finer-grained features for pixel-by-pixel classification. Simultaneously performing defect detection and segmentation on the same feature map means simultaneously focusing on both coarse and fine-grained features, inevitably leading to learning confusion. When the feature map focuses on coarse-grained features, the performance of segmentation tasks requiring fine-grained features degrades; conversely, when it focuses on fine-grained features, the performance of detection tasks requiring coarse-grained features suffers. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention improves upon the Mask R-CNN joint detection and segmentation framework, proposing a multi-task appearance defect recognition method based on feature interaction between detection and segmentation. To address the inefficiency of feature fusion between feature maps of different resolutions in current frameworks, this invention designs a deep feature fusion module with an attention mechanism and a bidirectional FPN structure to fully fuse information from feature maps of different resolutions. To address the issue that small-sized RoIs cannot fully represent defect features, a feature reconstruction module with context aggregation is designed, capable of representing defect features at various scales. To address the competitive learning problem caused by shared feature maps in defect detection and segmentation, a feature interaction network composed of a position interaction module and a channel interaction module is designed. This network decouples task-shared feature maps into task-specific feature maps, enabling sufficient channel and position information interaction between feature maps of different tasks, improving the feature representation capability of task feature maps and providing more favorable feature maps for subsequent tasks. The method of this invention achieves a significant improvement in the performance of appearance defect recognition by fully fusing information from feature maps and making full use of features in the segmentation branches, combined with the interaction of detection and segmentation features. It not only far surpasses the benchmark network framework Mask RCNN, but its segmentation ability also has certain advantages compared with other cutting-edge segmentation methods commonly used in the field of defect segmentation.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A multi-task appearance defect recognition method based on the interaction of detection and segmentation features, characterized by the following steps:

[0013] S1. For a given industrial appearance defect image with a size of M×M×3, after feature extraction by the ResNet50 feature extraction network, four feature maps with different resolutions are output, from largest to smallest: M / 4×M / 4, M / 8×M / 8, M / 16×M / 16, and M / 32×M / 32, with 256 channels in each.

[0014] S2. Perform deep feature fusion on the obtained 4 feature maps. Specifically, first use a bidirectional FPN consisting of deep-to-shallow and shallow-to-deep paths to facilitate information flow in the feature maps. Then, after each layer of feature maps, add an attention mechanism consisting of spatial attention and channel attention to refine the feature maps, resulting in 4 task-shared feature maps F.

[0015] S3. Input the task-shared feature map F into the feature interaction network, which consists of a position interaction module and a channel interaction module set sequentially. The feature interaction network uses the position interaction module and the channel interaction module to decouple the shared feature map F and learn from each other into a feature map F for the detection task. det and feature maps F for segmentation tasks seg .

[0016] S4, Feature map F for the detection task det Perform the detection branch operation: convert the multi-scale feature map F det The data is fed into the RPN network for candidate box extraction, and then the candidate boxes are combined with the feature map F. det The corresponding layer is subjected to RoI Align operation to a fixed size, and finally the fixed-size RoIs are classified and regressed to obtain the detection results.

[0017] S5, Feature map F for segmentation task seg Perform the branching operation: divide the feature map F in ascending order of resolution. seg Feature reconstruction is performed; the feature reconstruction process is as follows: firstly, the context aggregation module is used to... Context aggregation is performed to obtain the data, and then... The feature map resolution is restored by progressive upsampling and using a symmetrical encoder-decoder structure with skip connections during the upsampling process. Finally, the feature map is upsampled to the original image size and classified pixel by pixel to obtain the segmentation result.

[0018] (1) The deep feature fusion module proposed in this invention can fully fuse the feature map output by the backbone, and output a feature map containing more beneficial information for subsequent tasks.

[0019] (2) The feature reconstruction module proposed in this invention enables the segmentation branch to reconstruct the low-resolution feature map to the original image size, thus avoiding information loss caused by segmentation on small-sized RoIs.

[0020] (3) The feature interaction network proposed in this invention can decouple the feature maps shared by tasks into specific feature maps for detection and segmentation, thus avoiding competitive learning.

[0021] (4) The method of the present invention significantly improves the performance of network defect identification. On the electronic device appearance defect dataset, the detection indicators such as AP and AR10 are improved by 1.21 and 1.20 points respectively compared with the benchmark network Mask RCNN, while the segmentation indicators such as mIoU, P, R and F1 are improved by 5.24, 7.45, 2.25 and 5.0 points respectively. Attached Figure Description

[0022] Figure 1 A schematic diagram of the MASK RCNN framework for joint detection and segmentation.

[0023] Figure 2 A schematic diagram of the structure of a multi-task appearance defect recognition algorithm.

[0024] Figure 3 .Depth feature fusion module structure diagram.

[0025] Figure 4 Feature reconstruction module structure diagram.

[0026] Figure 5 . Row and column mutual attention module structure diagram.

[0027] Figure 6 Channel interaction module structure diagram.

[0028] Figure 7 Comparison of model and Mask RCNN results in defect identification on electronic device appearance defect dataset. Detailed Implementation

[0029] The focus of this invention is to improve upon the problems existing in multi-task appearance defect identification and to propose a new multi-task appearance defect identification method. The overall framework of this invention is shown in the appendix. Figure 2 Based on the joint detection and segmentation framework MaskRCNN, the unidirectional FPN is replaced with a deep feature fusion module, the original segmentation branch is replaced with a feature reconstruction module, and a feature interaction network is inserted between the backbone network and the task branch network.

[0030] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] S1. For a given industrial appearance defect image with a size of 512×512×3, after feature extraction by the ResNet50 feature extraction network, four feature maps with different resolutions, C2, C3, C4 and C5, are output, which are 128×128, 64×64, 32×32 and 16×16 from largest to smallest, and the number of channels is 256 for each.

[0032] S2. Input the four feature maps C2, C3, C4, and C5 obtained into the deep feature fusion module for deep feature fusion. Specifically, firstly, a bidirectional FPN consisting of deep-to-shallow and shallow-to-deep paths is used to allow sufficient information flow through the feature maps to obtain feature maps M2, M3, M4, and M5. Then, an attention mechanism consisting of spatial attention and channel attention is applied after each layer of feature maps to refine the feature maps, outputting four task-shared feature maps F (M2′, M3′, M4′, and M5′).

[0033] S3. Input the task-shared feature map F into the feature interaction network, which consists of a position interaction module and a channel interaction module set sequentially. The feature interaction network uses the position interaction module and the channel interaction module to decouple the shared feature map and learn from each other to form a feature map F for the detection task. det and feature maps F for segmentation tasks seg .

[0034] S4, Feature map F for the detection task det Perform the detection branch operation: convert the multi-scale feature map F det The data is fed into the RPN network for candidate box extraction, and then the candidate boxes are combined with the feature map F. det The corresponding layer is subjected to RoI Align operation to a fixed size, and finally the fixed-size RoIs are classified and regressed to obtain the detection results.

[0035] S5, Feature map F for segmentation task seg Perform the branching operation: sort the feature maps in ascending order of resolution. Feature reconstruction is performed; the feature reconstruction process is as follows: firstly, the context aggregation module is used to... Context aggregation is performed to obtain feature maps. Then the feature map The feature map resolution is restored by progressive upsampling and using a symmetrical encoder-decoder structure with skip connections during the upsampling process. Finally, the feature map is upsampled to the original image size and classified pixel by pixel to obtain the segmentation result.

[0036] The following provides a detailed explanation of each of the relevant details.

[0037] (1) Deep Feature Fusion Module

[0038] The deep feature fusion module aims to fully fuse and refine the features extracted by the ResNet50 feature extraction network, providing more favorable feature inputs for subsequent defect detection and segmentation. The original FPN only has one path for information flow from deep to shallow layers, causing deep features to lose shape and position information contained in shallow features, which is detrimental to the detection and segmentation of large-sized defects. This invention introduces a bidirectional FPN structure to perform bidirectional information fusion of feature maps from deep to shallow and from shallow to deep. Considering that each layer of the feature after bidirectional FPN fusion contains information from multiple layers, which can easily lead to the weakening of information in that layer, an attention mechanism is added after feature fusion to refine it. The complete structure of the deep feature fusion module is shown in the attached figure. Figure 3 As shown.

[0039] Depend on Figure 3 As can be seen, the deep feature fusion module includes two main components: a bidirectional FPN structure and an attention mechanism consisting of channel attention and spatial attention. These two parts will be described in detail below.

[0040] 1. Bidirectional FPN structure

[0041] From the appendix Figure 3 As can be seen, the bidirectional FPN structure adds a shallow-to-deep path after the original deep-to-shallow path, achieving bidirectional information flow. Its specific workflow is as follows:

[0042] For an input image, after feature extraction by ResNet50, feature maps C2, C3, C4, and C5 are obtained, with their resolutions decreasing by a factor of two in each iteration. The original FPN's deep-to-shallow path starts from the deepest feature map P5, progressively upsampling it and adding it pixel-by-pixel to feature maps C4, C3, and C2 to obtain feature maps P4, P3, and P2. The added shallow-to-deep path starts from the highest resolution feature map P2, using a 3×3 convolution without changing the shape of the feature map to obtain M2; then, a 3×3 convolution with a stride of 2 is used to downsample M2 by a factor of 2, and it is added pixel-by-pixel to the feature map P3 using a 3×3 convolution without changing its shape to obtain M3; the operations for M4 and M5 are similar to those for M3.

[0043] 2. Attention Mechanism

[0044] In a bidirectional FPN structure, each feature map layer incorporates information passed from the layers above and below, which to some extent weakens the information in the current layer. Attention mechanisms can help learn "which" and "where" features are more helpful for the task. Therefore, an attention mechanism is introduced for each feature map output by the bidirectional FPN structure, whose structure is identical, as shown in the attached figure. Figure 3As shown, the attention mechanism consists of two components: Channel Attention (CA) is responsible for which channels the feature map should focus on, while Spatial Attention (SA) is responsible for which spatial locations the feature map should focus on.

[0045] (2) Feature Interaction Network

[0046] The feature interaction network operates on each layer (M2′, M3′, M4′, M5′) of the feature map F output by the deep feature fusion module. The operation process is no different except for the resolution size. Therefore, this embodiment selects M2′ as an example for detailed description.

[0047] For the shared feature map M2′ output by the deep feature fusion module, the feature interaction network first copies M2′ into two feature maps. Then, the two feature maps M2′ pass through the position interaction module and the channel interaction module in turn to learn feature information that is beneficial to their own task from each other. The position interaction module focuses on which positions of the other party's feature map are helpful to its own task, while the channel interaction module focuses on which channels of the other party's feature map are beneficial to its own task.

[0048] Four shared feature maps F are processed by a feature interaction network to output feature map F. det and F seg , will F det Used for defect detection tasks, while F seg Segmentation is performed after feature reconstruction. Adding a feature interaction network avoids competitive learning between defect detection and defect segmentation on the same feature map, and allows both tasks to learn information beneficial to themselves from each other, thus improving the performance of both defect detection and segmentation. The positional interaction module and channel interaction module that constitute the feature interaction network will be described in detail below.

[0049] 1. Location Interaction Module

[0050] This invention forms a position interaction module by stacking row and column mutual attention modules twice, with appended... Figure 5 This diagram illustrates the structure of the row-column mutual attention module for defect detection and segmentation features. The processing flow shown in the diagram is represented by feature map M2′. The workflow of the row-column mutual attention module consists of three steps: shared feature preprocessing, row-column position mutual learning, and task feature acquisition. The specific operations are as follows:

[0051] Shared feature preprocessing: For a task-shared feature map M2′, 1×1 convolutions are used to obtain feature maps F1 and F2 for different tasks, while maintaining their dimensions. Then, F1 and F2 are reconstructed to obtain four two-dimensional tensors S. 1H S1W S 2H and S 2W .

[0052] Taking F1 as an example: reconstruct the features of F1 along the column direction of the feature map, and combine its channels and row vectors to obtain a two-dimensional tensor S of size (CW)×H. 1H This is used for subsequent feature interactions in the column direction; simultaneously, F1 is used for feature reconstruction in the row direction of the feature map, and its channels and column vectors are combined to obtain a two-dimensional tensor S of size (CH)×W. 1W This is used for feature interactions in subsequent row directions. Performing the same operation on F2 yields the two-dimensional tensor S. 2H and S 2W .

[0053] Row and column position mutual learning: for four two-dimensional tensors S 1H S 1W S 2H and S 2W They are divided into two groups, S, which is related to the row vector. 1W S 2W and S related to column vectors 1H and S 2H The two sets of tensors generate mutual attention weight graphs in the same way; here, S, which is related to the column vector, is selected. 1H and S 2H A detailed explanation will be provided. Specifically, the two-dimensional tensor S of a task... 1H / 2H ( / represents the transpose of a two-dimensional tensor with another task) Perform matrix multiplication, then use the Softma x operation to normalize the row vectors, and obtain the column mutual attention weight graph H corresponding to the two tasks. 21 and H 12 All dimensions are H×H. For S 1W and S 2W Perform the same operation to obtain W. 21 and W 12 .

[0054] Task feature acquisition: based on the four weight graphs H already obtained 21 H 12 W 21 and W 12 And feature maps F1 and F2 are used to further obtain feature maps corresponding to the defect detection and defect segmentation tasks. Based on H... 21 W 21 F1 computation of defect detection task feature map According to H 12 W 12 F2 and F2 computation of defect segmentation task feature map The calculation methods are exactly the same; here we will use the calculation method. This will be presented as an example. The feature map of the F1 reconstruction (CW) × H size will be compared with the column cross-attention weight map H. 21 Matrix multiplication is performed to obtain column-enhanced feature maps. The F1 matrix is ​​then reconstructed into a (CH)×W feature map and a row mutual attention weight map W. 21 Matrix multiplication is performed to obtain row-enhanced feature maps. Both enhanced feature maps are then reconstructed into C×H×W feature maps, and element-wise summed to obtain row- and column-enhanced feature maps. Residual concatenation is performed between the row- and column-enhanced feature maps and the shared feature map M2′ to obtain the defect detection task feature map. Defect segmentation task feature map The acquisition method and Similarly, to prevent information redundancy, the enhanced feature map is multiplied by a learnable parameter γ before being added to M2′. 1 / 2 Dynamically adjust its proportion or The proportion of. Then and These will be used as feature maps for subsequent defect detection and defect segmentation tasks, respectively.

[0055] The shared feature map M2′ is processed through a row-column mutual attention module. A pixel on the feature map of Task 1 or 2 learns from pixels in the same row or column as that pixel on the feature map of Task 2 or 1. However, to learn information from all pixels on the other's feature map, the feature map needs to be input into the row-column mutual attention module again. The stacking of the two row-column mutual attention modules constitutes a complete positional interaction module, which divides a shared feature map into two task-specific unique feature maps and successfully enables information exchange between any pixels on the two task feature maps, further improving the performance of each task.

[0056] 2. Channel Interaction Module

[0057] The channel interaction module improves performance by mutually learning between two feature maps across channels, resulting in defect detection features that are more beneficial for defect detection tasks and segmentation features that are more beneficial for defect segmentation tasks. (Appendix) Figure 6 This diagram illustrates the structure of the channel interaction module for defect detection and defect segmentation features. Its workflow consists of three steps: shared feature preprocessing, channel mutual learning, and task feature acquisition, as detailed below:

[0058] Shared feature preprocessing: for the feature maps obtained from the location interaction module Feature maps F'1 and F'2 for different tasks are obtained by using 1×1 convolutions respectively, while keeping their size unchanged. Then, F'1 and F'2 are reshaped into two-dimensional tensors C1 and C2 of size C×(HW) for subsequent channel cross-learning.

[0059] Channel Mutual Learning: To jointly improve the performance of defect detection and defect segmentation, channel mutual learning constructs mutual attention weight graphs on channels to facilitate information exchange between tasks. Specifically, a task's two-dimensional tensor C... 1 / 2 ( / represents the transpose of a two-dimensional tensor with another task) Perform matrix multiplication, then use the Softmax operation to normalize the row vectors to obtain the mutual attention weight maps W1 and W2 for each task, both of size C×C.

[0060] Task Feature Acquisition: Based on feature maps F'1, F'2, and mutual attention weight maps W1 and W2, feature maps corresponding to defect detection and defect segmentation tasks can be easily obtained. First, F'1 and F'2 are reconstructed into a C×(HW) two-dimensional tensor. Then, they are multiplied by the mutual attention weight map corresponding to each task to obtain the enhanced feature map after channel mutual learning, which is a C×(HW) two-dimensional tensor. The enhanced feature map is then reconstructed into... C×H×W of the same size and Residual connections are performed. To prevent information redundancy, a learnable parameter λ is introduced into the reconstructed enhanced feature map. 1 / 2 It dynamically adjusts its contribution to the task feature map.

[0061] The obtained task feature map F det and F seg Using them as inputs for defect detection and defect segmentation tasks respectively avoids learning competition between the two tasks on the shared feature map.

[0062] (3) Feature Reconstruction Module

[0063] To address the problem that current segmentation branches struggle to fully represent defects, this invention designs a feature reconstruction module based on a symmetric encoder-decoder. In general, the design of this module aims to perform pixel-by-pixel classification on the feature map at the original image resolution during segmentation, avoiding the loss of significant defect information when segmenting small RoIs. During feature map reconstruction, due to the presence of numerous large-span and connected defects, pixels on the feature map are required to have a large receptive field. DeeplabV3 has proven that dilated convolution is an effective way to improve the receptive field; therefore, a context aggregation module is designed using dilated convolution to ensure that a single pixel in the deepest feature map has the receptive field of the entire image. Then, the lowest resolution feature map is progressively upsampled to restore information, with skip connections added during the upsampling process. The structure of the feature reconstruction module is shown in the attached figure. Figure 4 As shown.

[0064] For the feature map F output by the feature interaction network seg The feature reconstruction module starts from the deepest layer Reconstruction has begun. (Regarding...) Before upsampling, to enable pixels on the feature map to have a larger receptive field to identify large-span defects in connectivity, a context aggregation module was first designed using dilated convolution to process the feature map. Advanced semantic information fusion is performed. A dilated convolution with a dilation rate of 'd' is equivalent to inserting 'd-1' zeros between adjacent elements of the convolution kernel, increasing the receptive field of the pixel without increasing computational cost. The specific operations of the context aggregation module are as follows:

[0065] The context aggregation module includes three branches, with the topmost branch maintaining... Unchanged; the middle branches will After passing through a 3×3 convolutional layer with a stride dilation of 3; the following branch will... The feature maps from the first branch are concatenated channel-wise with the feature maps obtained from the dilated convolutional layer with a dilation rate of 3, and then passed through a 3×3 convolutional layer with a dilation rate of 5. The feature maps from the three branches are then concatenated channel-wise, and finally, a 1×1 convolution is used to restore the number of channels to the same level as before. The same result yields a feature map. All dilated convolutional layers have a stride of 1 and use padding to maintain the feature map size. A normalization layer and an activation layer are appended after each dilated convolution.

[0066] After the context aggregation module, each pixel in the feature map contains enough context information. Next, this 16×16 feature map... To perform stepwise feature reconstruction and restore the original image size, the specific steps are as follows: Apply a 2×2 transposed convolution with a stride of 2 to the feature map. Perform double upsampling, then utilize skip connections and feature maps Channel-wise concatenation is performed, and the concatenated feature maps are then used to recover the number of channels using 3×3 convolutions, resulting in a 32×32 resolution feature map. Subsequent higher-resolution feature map operations are similar, until the result is obtained. 128×128 resolution feature maps of the same layer Since the deep feature fusion module does not output feature maps with higher resolution, the 128×128 feature map is used instead. The image is upsampled four times to the original size of 512×512. The operation used is a 4×4 transposed convolution with a stride of 4.

[0067] (4) Train the model and demonstrate its effectiveness in defect identification through experiments.

[0068] In order to jointly optimize defect detection and defect segmentation, the appearance defect recognition method of the present invention needs to minimize the multi-task joint loss that includes detection and segmentation during training, which is the same as the training method of the original detection and segmentation joint framework MASK RCNN.

[0069] In the detection task, the loss is divided into RPN loss and RoI loss. The calculation methods for these two parts are the same, both including classification loss and regression loss, which are expressed by formula (1):

[0070]

[0071] In the above formula, L det To detect mission losses, L cls For classification loss, L reg For regression loss; N cls N is the batch size. reg p represents the number of candidate boxes or RoIs, where i represents the index of the candidate box or RoI, and p represents the index of the candidate box or RoI. i It predicts classification probability, for positive samples. When negative samples t i These are the parametric coordinates of the predicted bounding box. These are the parametric coordinates of the ground truth bounding box. L cls and L reg They are represented as follows:

[0072]

[0073]

[0074] Where R is the smooth L1 loss, as shown in formula (4):

[0075]

[0076] Segmentation tasks, being essentially pixel-by-pixel classification tasks, often employ multi-class cross-entropy loss. However, in the defect recognition of this invention, since only defects and background are distinguished, binary cross-entropy loss is used, as shown in equation (5):

[0077]

[0078] In the above formula, j represents a pixel, N represents the total number of pixels to be segmented, and y j p(y) represents the true label of the pixel, 0 or 1. i ) represents the network's predicted label for pixel j, with a value between 0 and 1. When the predicted value p(y) j ) and real label y j Approaching, for example, y j =1 and the predicted value p(y) j When the value is close to 1, the loss is close to 0; conversely, the loss is greater when the value is close to 1.

[0079] The multi-task joint loss of the industrial appearance defect recognition algorithm in this invention can be expressed as the defect detection loss L. det and defect segmentation loss L mask The linear summation is shown in equation (6). It should be noted that L... det The detection loss includes both RPN and RoI components.

[0080] L total =L det +L mask (6)

[0081] Using electronic device appearance defect datasets and Highway-crack datasets as validation objects, the identification results of this invention and existing methods are as follows:

[0082]

[0083] Table 1. Comparison of the method of the present invention with other segmentation methods.

[0084]

[0085]

[0086] Table 2 shows the effectiveness of the design of each module in this invention.

[0087] In Table 1, the optimal metrics are shown in bold, with upward arrows indicating that higher metrics are better. The left side of Table 1 displays the defect segmentation performance metrics of our proposed method and cutting-edge defect segmentation methods on an electronic device appearance defect dataset. It can be seen that, after the enhancement of the feature interaction network, our method achieves the highest mIoU, P, R, and F1 scores in defect segmentation. The accuracy metric shows the greatest improvement compared to the method in Chapter 3, thus surpassing the original highest accuracy UNet network's 79.02 to reach 79.72. Compared to the baseline network Mask R-CNN, all metrics show significant improvements, with mIoU, P, R, and F1 increasing by 5.24, 7.45, 2.25, and 5.00 points respectively, fully demonstrating the powerful performance of the proposed method in defect segmentation.

[0088] Table 1 (right side) shows the performance metrics of our proposed method and cutting-edge defect segmentation methods on the Highway-crack dataset. The comparison reveals that the feature interaction network proposed in this invention also demonstrates good performance on the Highway-crack dataset. Except for a slight decrease in precision (P), mIoU, R, and F1 scores are all improved, at 0.14, 0.38, and 0.12, respectively. Compared to the baseline network Mask R-CNN, our proposed method shows significant improvements, with substantial increases in all metrics except for recall (R). Mask R-CNN achieves high recall at the expense of precision (P), resulting in very low precision, which is undesirable for defect segmentation. In summary, our proposed method further improves the performance metrics of defect segmentation, achieving better defect recognition results.

[0089] Table 2 shows the ablation results of the electronic device appearance defect dataset module. The optimal index is highlighted in black, and upward arrows indicate that higher indices are better. The deep feature fusion module is denoted as "fusion," the feature reconstruction module as "reconstruction," and the feature interaction network as "interaction." First, two additional aspect ratios were added to the preset anchors of Mask R-CNN. This improvement enhanced both defect detection and segmentation performance, resulting in a higher-performance benchmark. The deep feature fusion module improved both defect detection and segmentation, with an AP index improvement of 0.22 for defect detection. This indicates that the deep feature fusion module effectively fused the features output by the feature extraction network, making the output features more conducive to defect detection and segmentation. The feature reconstruction module significantly improved network performance, especially segmentation performance, with an mIoU improvement of 2.74 points and a P improvement of 5.42 points. This fully demonstrates that the feature reconstruction network effectively solves the problem of insufficient feature information caused by basic segmentation on small-sized RoIs, and the output high-resolution feature map effectively reconstructs the feature representation of the defect region. Combining the deep feature fusion module with the feature reconstruction module further improved the overall network performance, especially in defect segmentation, where multiple metrics reached their highest levels. Compared to using feature reconstruction alone, several metrics showed further improvements, indicating that the feature maps output by the deep feature fusion module retained more feature information, playing a significant role in feature reconstruction. Compared to the baseline, the AP metric for detection improved by 0.2 points, and the mIoU, P, R, and F1 metrics for segmentation improved by 3.59, 5.26, 0.89, and 3.24 points, respectively. Adding the feature interaction network further increased the AR10 for defect detection by 0.2, and improved the mIoU, P, R, and F1 metrics for defect segmentation by 0.93, 1.81, 0.39, and 1.12, respectively, demonstrating the rationality and effectiveness of the feature interaction network.

[0090] Figure 7 This figure compares the defect recognition results of the method of this invention with Mask R-CNN on an electronic device appearance defect dataset. As can be seen from the figure, when performing defect detection, Mask R-CNN tends to detect multiple discontinuous defects as a single large defect (row 1-3), while the present invention can focus on finer-grained features during defect detection, thus avoiding this problem. In defect segmentation, Mask R-CNN suffers from over-segmentation of defects, under-segmentation of small defects, and insufficient coherence in segmenting large-span defects, while the present invention shows significant improvement, achieving near-realistic segmentation results for various defects.

Claims

1. A multi-task appearance defect recognition method based on the interaction of detection and segmentation features, characterized in that, Includes the following steps: S1. For a given industrial appearance defect image, its size is M×M×3. After the feature extraction network ResNet50 extracts features, it outputs 4 feature maps with different resolutions, from largest to smallest: M / 4×M / 4, M / 8×M / 8, M / 16×M / 16, and M / 32×M / 32, with 256 channels in each. S2. Deep feature fusion is performed on the four feature maps obtained. Specifically, a bidirectional FPN consisting of deep-to-shallow and shallow-to-deep paths is used to facilitate information flow in the feature maps. Then, an attention mechanism consisting of spatial attention and channel attention is added after each layer of feature maps to refine the feature maps, resulting in four task-shared feature maps F. S3. Input the task-shared feature map F into the feature interaction network, which is composed of a position interaction module and a channel interaction module set in sequence. The feature interaction network uses the position interaction module and the channel interaction module to decouple the shared feature map F and learn from each other into feature maps for the detection task and feature maps for the segmentation task. S4. Perform detection branch operation on the feature map of the detection task: feed the multi-scale feature map into the RPN network to extract candidate boxes, then combine the candidate boxes with the corresponding layer of the feature map to perform RoI Align operation to a fixed size, and finally classify and regress the fixed-size RoIs to obtain the detection results. S5. Perform segmentation branching operations on the feature maps of the segmentation task: Reconstruct the features of the feature maps in order of increasing resolution; The feature reconstruction process is as follows: First, use the context aggregation module to perform context aggregation to obtain the feature maps, then gradually upsample the feature maps. During the upsampling process, use a symmetrical encoder-decoder structure with skip connections to restore the feature map resolution; Finally, upsample the feature maps to the original image size and perform pixel-by-pixel classification to obtain the segmentation result.

Images