Industrial Defect Instance Segmentation Method

Abstract

This invention discloses a method for segmenting industrial defect instances. The method includes the following steps: during training, the shallow network parameters of the feature extraction network are not fixed to ensure that the pre-trained network obtained on the open-source natural instance dataset can better fit the industrial defect instances, thus effectively extracting features from them; a cascaded defect region proposal module is used to avoid the extensive parameter adjustments required when migrating from natural instance segmentation to industrial defect instance segmentation, and a mechanism of multiple candidate box optimization is adopted to improve algorithm accuracy; an adaptive segmentation quality assessment probe is proposed, which generates an adaptive segmentation threshold based on the segmentation results, binarizes the segmentation results, and scores the segmentation results, solving the problem of excessively high instance segmentation scores. These designs enable this invention to effectively solve the problem of industrial defect instance segmentation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and more specifically, relates to a method for segmenting industrial defect instances. Background Technology

[0002] In recent years, industrial defect detection, as a crucial part of industrial production and manufacturing, has become one of the important research directions in the field of industrial vision. Industrial defect instance segmentation is a specific sub-category of industrial defect detection, aiming to segment potential defective regions in the sample to be inspected at the instance level, including bounding box analysis, category scoring, and segmentation. This technology can assist engineers in identifying potential defective products, classifying defects among defective products, and processing these defective products according to their severity, which can greatly improve industrial production efficiency and has rich practical application value.

[0003] Existing mainstream instance segmentation methods can be broadly classified into two categories: single-stage methods and two-stage methods. Single-stage methods treat the instance segmentation task as a whole or decompose it into parallel branches, offering high real-time performance but limited accuracy. Two-stage methods decompose the instance segmentation task into two sequential subtasks, achieving high accuracy but lower real-time performance. Currently, all instance segmentation algorithms are designed for natural instance segmentation tasks; there are no instance segmentation algorithms designed for industrial defect detection tasks. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for segmenting industrial defect instances. To achieve the above objectives, this invention provides the following technical solution:

[0005] An industrial defect instance segmentation method includes the following steps:

[0006] Step S1, train the industrial defect instance segmentation network (1), use images with instance-level annotations as training data to input into the training industrial defect instance segmentation network model, and use the loss function to constrain the training process;

[0007] Step S2, segmentation stage, input the test image (2) containing the industrial defect region into the industrial defect instance segmentation network (1) to obtain the industrial defect instance segmentation result. Step S1 can be subdivided into 3 sub-steps: S1-1, industrial image feature extraction; S1-2, industrial defect localization based on the cascaded defect region proposal module; S1-3, industrial defect instance segmentation based on adaptive segmentation quality assessment.

[0008] Preferably, the training process of the industrial defect instance segmentation network (1) incorporates a series of targeted designs for industrial defect instances. This network can perform secondary corrections on the candidate box positions and accurately locate potential defect areas. This process can be specifically divided into three steps:

[0009] Step S1-1, Industrial Image Feature Extraction: This step utilizes a backbone network based on a convolutional neural network to extract the fusion features of the input image, specifically including:

[0010] Step S1-1(1): Use ResNet50 pre-trained on ImageNet as the feature extraction network to extract features from the input image. The size of the image feature maps output from the first to the fourth convolutional blocks are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, respectively, and the number of channels are C1, C2, C3, C4, respectively. C1, C2, C3, and C4 are preset values. ImageNet is a natural instance dataset. Considering the significant difference between natural instances and industrial defect instances, unlike the approach of natural instance segmentation tasks, the shallow network parameters of ResNet50 are not fixed during training.

[0011] Step S1-1(2): The 1 / 32 resolution feature map output from the fourth convolutional block extracted by the backbone network is dimensionality reduced by a convolutional layer. The fused feature F1 is obtained, and then max pooling is used to reduce the resolution to half of the original to obtain the fused feature F2.

[0012] Step S1-1(3): The 1 / 16 resolution feature map output from the third convolutional block extracted by the backbone network is dimensionality reduced by a convolutional layer. F1 is upsampled to its nearest neighbor and stacked with the previous sample before being fused using a 1×1 convolution. The fusion feature F3 is obtained;

[0013] Step S1-1(4): The 1 / 8 resolution feature map output from the third convolutional block extracted by the backbone network is dimensionality reduced by a convolutional layer. The F1 sample is upsampled from the nearest neighbor sample and stacked with the previous sample, then fused using a 1×1 convolution. The fusion feature F4 is obtained;

[0014] Step S1-1(5): The 1 / 4 resolution feature map output from the third convolutional block extracted by the backbone network is smoothed through a convolutional layer, and then the F1 is upsampled to the nearest neighbor and stacked with the former, and then fused using a 1×1 convolution. The fusion feature F5 is obtained;

[0015] Step S1-2: Industrial defect localization based on the cascaded defect region proposal module. Preferably, the fused features F1, F2, F3, F4, and F5 are input into the cascaded defect region proposal network to locate potential defect regions, specifically including:

[0016] Step S1-2(1): Use a 3×3 dilated convolution kernel to encode the fusion feature F1 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L1. Correct the preset candidate boxes in the fusion feature F1 according to the regression value L1. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R1 and the class score S1. Correct the preset candidate boxes according to the second regression value R1. Select the top K candidate boxes with the highest foreground scores based on the class score S1. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A1 corresponding to the fusion feature F1, where K is a preset value. Use ROI Align to extract the ROI feature G1 in the fusion feature F1 based on the candidate box A1.

[0017] Step S1-2(2): Use a 3×3 dilated convolution kernel to encode the fusion feature F2 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L2. Correct the preset candidate boxes in the fusion feature F2 according to the regression value L2. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R2 and the class score S2. Correct the preset candidate boxes according to the second regression value R2. Select the top K candidate boxes with the highest foreground scores based on the class score S2. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A2 corresponding to the fusion feature F2, where K is a preset value. Use ROI Align to extract the ROI feature G2 in the fusion feature F2 based on the candidate box A2.

[0018] Step S1-2(3): Use a 3×3 dilated convolution kernel to encode the fusion feature F3 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L3. Correct the preset candidate boxes in the fusion feature F3 according to the regression value L3. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R3 and the class score S3. Correct the preset candidate boxes according to the second regression value R3. Select the top K candidate boxes with the highest foreground scores based on the class score S3. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A3 corresponding to the fusion feature F3, where K is a preset value. Use ROI Align to extract the ROI feature G3 in the fusion feature F3 based on the candidate box A3.

[0019] Step S1-2(4): Use a 3×3 dilated convolution kernel to encode the fusion feature F4 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L4. Correct the preset candidate boxes in the fusion feature F4 according to the regression value L4. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R4 and the class score S4. Correct the preset candidate boxes according to the second regression value R4. Select the top K candidate boxes with the highest foreground scores based on the class score S4. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A4 corresponding to the fusion feature F4, where K is a preset value. Use ROI Align to extract the ROI feature G4 in the fusion feature F4 based on the candidate box A4.

[0020] Step S1-2(5): Use a 3×3 dilated convolution kernel to encode the fusion feature F5 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L5. Correct the preset candidate boxes in the fusion feature F5 according to the regression value L5. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform secondary regression and class score on potential defects to obtain the secondary regression value R5 and the class score S5. Correct the preset candidate boxes according to the secondary regression value R5. Select the top K candidate boxes with the highest foreground scores based on the class score S5. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A5 corresponding to the fusion feature F5, where K is a preset value. Use ROI Align to extract the ROI feature G5 in the fusion feature F1 based on the candidate box A5.

[0021] Steps S1-3: Industrial defect instance segmentation based on adaptive segmentation quality assessment. The previously obtained ROI features G1, G2, G3, G4, and G5 are input into the industrial defect instance segmentation module to generate instance segmentation results. The industrial defect instance segmentation module consists of a detection probe, a segmentation probe, and an adaptive segmentation quality assessment probe: the detection probe generates candidate box regression values ​​and classification scores; the segmentation probe generates segmentation results; the adaptive segmentation quality assessment probe calculates a dynamic threshold based on the segmentation probe output to binarize the segmentation results, outputs a segmentation quality score for the binarized segmentation results, and multiplies this score by the classification score to obtain the final instance segmentation score. Specifically:

[0022] Step S1-3(1): Input the ROI feature G1 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L1 and the class score prediction P1.

[0023] Step S1-3(2): Input the ROI feature G2 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the position of the activation features, followed by two parallel fully connected layers to obtain the candidate box regression value L2 and the class score prediction P2.

[0024] Step Y1-3(3): Input ROI feature G3 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L3 and the class score prediction P3.

[0025] Step Y1-3(4): Input ROI feature G4 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L4 and the class score prediction P4.

[0026] Step S1-3(5): Input ROI feature G5 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L5 and the class score prediction P5.

[0027] Step S1-3(6): Input the ROI feature G1 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T1 is generated by a 3×3 convolutional kernel. T1 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M1. The adaptive threshold is used to binarize T1. The binarized segmentation result is then downsampled to 1 / 4 resolution using max pooling and superimposed with the ROI feature G1 along the channel dimension. This result is then downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E1 is obtained by connecting two fully connected layers.

[0028] Step S1-3(7): Input ROI feature G2 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T2 is generated by a 3×3 convolutional kernel. T2 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M2. The adaptive threshold is used to binarize T2, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G2 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E2 is obtained by connecting two fully connected layers.

[0029] Step S1-3(8): Input ROI feature G3 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T3 is generated by a 3×3 convolutional kernel. T3 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M3. The adaptive threshold is used to binarize T3, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G3 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R.14×14×257 →R 7×7×256 The segmentation quality score E3 is obtained by connecting two fully connected layers.

[0030] Step S1-4(9): Input ROI feature G4 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T4 is generated by a 3×3 convolutional kernel. T4 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M4. The adaptive threshold is used to binarize T4, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G4 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E4 is obtained by adding two fully connected layers.

[0031] Step S1-3(10): Input ROI feature G5 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T5 is generated by a 3×3 convolutional kernel. T5 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M5. The adaptive threshold is used to binarize T5, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G1 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E5 is obtained by following two fully connected layers.

[0032] In step S1-3(11), the segmentation quality scores F1, E2, E3, E4, F5, candidate box regression values ​​L1, L2, L3, L4, L5, category score predictions P1, P2, P3, P4, P5, adaptive segmentation thresholds M1, M2, M3, M4, M5, and segmentation results T1, T2, T3, T4, T5 are respectively added to the corresponding ground truth labels to calculate the loss and obtain the total loss for updating the model parameters.

[0033] Preferably, in the segmentation stage, the defect sample to be detected is input into the industrial defect instance segmentation network (1), and the industrial defect instance segmentation module outputs the instance segmentation result including the candidate box regression value L, the instance segmentation score Z obtained by multiplying the category score P and the segmentation quality score E, and the segmentation result T.

[0034] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0035] (1) This invention proposes an industrial defect instance segmentation method. The core framework of the method consists of a feature extraction network, a feature pyramid network, a cascaded defect region proposal module, and an industrial defect instance segmentation module. This invention is designed for industrial defect instance data, specifically including: not fixing the shallow network parameters of the feature extraction network during training to ensure that the pre-trained network obtained on the open-source natural instance dataset can better fit the industrial defect instances and effectively extract features from them; using a cascaded defect region proposal module to avoid the large number of parameter adjustments required when migrating from the natural instance segmentation task to the industrial defect instance segmentation task, and adopting a mechanism of multiple candidate box optimization to improve the algorithm accuracy; and proposing an adaptive segmentation quality evaluation probe, which can generate an adaptive segmentation threshold based on the segmentation quality to binarize the segmentation results and score the segmentation results, thus solving the problem of overly high instance segmentation scores.

[0036] (2) Since the current mainstream instance segmentation methods are all focused on natural instance segmentation tasks and there are no algorithms designed for industrial defect instance segmentation tasks, this invention proposes the first algorithm to solve the industrial defect instance segmentation task. Attached Figure Description

[0037] Figure 1 This is an overall flowchart of the industrial defect instance segmentation method provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0039] This invention provides a method for segmenting industrial defect instances, such as... Figure 1 As shown, it includes the following steps:

[0040] Step S1, train the industrial defect instance segmentation network (1). Use images with instance-level annotations as training data to input into the model, and use a loss function to constrain the training process.

[0041] Step S2, Segmentation Stage. The test image containing the industrial defect region is input into the industrial defect instance segmentation network (1) to obtain the industrial defect instance segmentation result.

[0042] The invention mainly consists of three parts: 1) a feature extraction network training strategy; 2) the use of a cascaded defect region proposal module; and 3) an adaptive segmentation quality assessment probe. The steps in this invention will be described in detail below.

[0043] (1) Changes in the training strategy for feature extraction networks

[0044] The feature extraction network strategy steps in this embodiment of the invention are as follows: The feature extraction network uses ResNet50 pre-trained on ImageNet. During training, the feature network reads the pre-trained parameters but does not fix the parameters of any layer, and updates the model parameters using an end-to-end training method.

[0045] (2) Using the cascaded defect area suggestion module

[0046] The cascaded defect region suggestion module in this embodiment of the invention is used to locate potential defect regions in features, as follows:

[0047] Step S1-2(1): Use a 3×3 dilated convolution kernel to encode the fusion feature F1 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L1. Correct the preset candidate boxes in the fusion feature F1 according to the regression value L1. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R1 and the class score S1. Correct the preset candidate boxes according to the second regression value R1. Select the top K candidate boxes with the highest foreground scores based on the class score S1. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A1 corresponding to the fusion feature F1, where K is a preset value. Use ROI Align to extract the ROI feature G1 in the fusion feature F1 based on the candidate box A1.

[0048] Step S1-2(2): Use a 3×3 dilated convolution kernel to encode the fusion feature F2 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L2. Correct the preset candidate boxes in the fusion feature F2 according to the regression value L2. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R2 and the class score S2. Correct the preset candidate boxes according to the second regression value R2. Select the top K candidate boxes with the highest foreground scores based on the class score S2. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A2 corresponding to the fusion feature F2, where K is a preset value. Use ROI Align to extract the ROI feature G2 in the fusion feature F2 based on the candidate box A2.

[0049] Step S1-2(3): Use a 3×3 dilated convolution kernel to encode the fusion feature F3 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L3. Correct the preset candidate boxes in the fusion feature F3 according to the regression value L3. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R3 and the class score S3. Correct the preset candidate boxes according to the second regression value R3. Select the top K candidate boxes with the highest foreground scores based on the class score S3. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A3 corresponding to the fusion feature F3, where K is a preset value. Use ROI Align to extract the ROI feature G3 in the fusion feature F3 based on the candidate box A3.

[0050] Step S1-2(4): Use a 3×3 dilated convolution kernel to encode the fusion feature F4 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L4. Correct the preset candidate boxes in the fusion feature F4 according to the regression value L4. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform a second regression and class score on the potential defects to obtain the second regression value R4 and the class score S4. Correct the preset candidate boxes according to the second regression value R4. Select the top K candidate boxes with the highest foreground scores based on the class score S4. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A4 corresponding to the fusion feature F4, where K is a preset value. Use ROI Align to extract the ROI feature G4 in the fusion feature F4 based on the candidate box A4.

[0051] Step S1-2(5): Use a 3×3 dilated convolution kernel to encode the fusion feature F5 and activate it with a ReLU layer. Then, use a regressor to perform a candidate box regression to obtain the regression value L5. Correct the preset candidate boxes in the fusion feature F5 according to the regression value L5. Use adaptive convolution to encode the features. Then, use a classifier and a regressor to perform secondary regression and class score on potential defects to obtain the secondary regression value R5 and the class score S5. Correct the preset candidate boxes according to the secondary regression value R5. Select the top K candidate boxes with the highest foreground scores based on the class score S5. Use non-maximum suppression to remove redundant candidate boxes to obtain the candidate box A5 corresponding to the fusion feature F5, where K is a preset value. Use ROI Align to extract the ROI feature G5 in the fusion feature F1 based on the candidate box A5.

[0052] (3) An adaptive segmentation quality assessment probe was proposed.

[0053] This invention proposes an adaptive segmentation quality assessment probe. This module can generate an adaptive segmentation threshold based on the segmentation quality, binarize the segmentation results, and score the segmentation results, thus solving the problem of excessively high scores for instance segmentation. The method is as follows:

[0054] Step S1-3(1): Input the ROI feature G1 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L1 and the class score prediction P1.

[0055] Step S1-3(2): Input the ROI feature G2 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the position of the activation features, followed by two parallel fully connected layers to obtain the candidate box regression value L2 and the class score prediction P2.

[0056] Step S1-3(3): Input ROI feature G3 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L3 and the class score prediction P3.

[0057] Steps S1-3(4) involve inputting the ROI feature G4 into the detection probe and using a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L4 and the class score prediction P4.

[0058] Step S1-3(5): Input ROI feature G5 into the detection probe and use a pooling layer to downsample the feature to a fixed size. Downsampling and dimension upsampling of features are performed using a 7×7 convolution kernel. 7 ×7×256 →R 1×1×1024 The features are then encoded using a 1×1 convolutional kernel; two fully connected layers are used to encode the activation features at their positions, followed by two parallel fully connected layers to obtain the candidate box regression value L5 and the class score prediction P5.

[0059] Step S1-3(6): Input the ROI feature G1 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T1 is generated by a 3×3 convolutional kernel. T1 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M1. The adaptive threshold is used to binarize T1. The binarized segmentation result is then downsampled to 1 / 4 resolution using max pooling and superimposed with the ROI feature G1 along the channel dimension. This result is then downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E1 is obtained by connecting two fully connected layers.

[0060] Step S1-3(7): Input ROI feature G2 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features.14×14×256 →R 28×18×256 The segmentation result T2 is generated by a 3×3 convolutional kernel. T2 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M2. The adaptive threshold is used to binarize T2, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G2 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E2 is obtained by connecting two fully connected layers.

[0061] Step Y1-3(8) involves inputting the ROI feature G3 into the segmentation probe and using a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T3 is generated by a 3×3 convolutional kernel. T3 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M3. The adaptive threshold is used to binarize T3, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G3 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E3 is obtained by connecting two fully connected layers.

[0062] Step S1-3(9): Input ROI feature G4 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. The downsampled features are encoded using four 3×3 convolutional kernels, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T4 is generated by a 3×3 convolutional kernel. T4 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M4. The adaptive threshold is used to binarize T4, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G4 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score E4 is obtained by adding two fully connected layers.

[0063] Step S1-3(10): Input ROI feature G5 into the segmentation probe and use a pooling layer to downsample the feature to a fixed size. Ling uses four 3×3 convolutional kernels to encode the downsampled features, followed by a 2×2 deconvolutional layer to upsample the encoded features. 14×14×256 →R 28×18×256 The segmentation result T5 is generated by a 3×3 convolutional kernel. T5 is then input into an adaptive segmentation quality assessment probe, where global average pooling and a fully connected layer are applied, followed by a sigmoid operation to obtain an adaptive segmentation threshold M5. The adaptive threshold is used to binarize T5, and the binarized segmentation result is downsampled to 1 / 4 resolution using max pooling. This is then superimposed on the ROI feature G1 along the channel dimension and downsampled again by four 3×3 convolutional kernels to reduce the dimensionality of R. 14×14×257 →R 7×7×256 The segmentation quality score F5 is obtained by connecting two fully connected layers.

[0064] In this step, the segmentation result T, segmentation quality factor E, candidate box regression value L, and class score P interact and work together to improve the final segmentation accuracy. Specifically, during the training phase, the segmentation quality scores F1, E2, E3, E4, F5, candidate box regression values ​​L1, L2, L3, L4, L5, predicted class scores P1, P2, P3, P4, P5, adaptive segmentation thresholds M1, M2, M3, M4, M5, and segmentation results T1, T2, T3, T4, T5 are each added to the corresponding ground truth labels to calculate the loss and obtain the total loss used to update the model parameters. During the testing phase, the industrial defect instance segmentation module outputs the instance segmentation result, which includes the instance segmentation score Z obtained by multiplying the candidate box regression value L, the class score P, and the segmentation quality score E, as well as the segmentation result T.

[0065] This invention proposes a method for segmenting industrial defect instances. Designed specifically for industrial defect instance data, this invention includes: not fixing the shallow network parameters of the feature extraction network during training to ensure that the pre-trained network obtained on open-source natural instance datasets can better fit industrial defect instances, enabling effective feature extraction; using a cascaded defect region proposal module to avoid the extensive parameter adjustments required when migrating from natural instance segmentation to industrial defect instance segmentation, and employing a mechanism of multiple candidate box optimizations to improve algorithm accuracy; and proposing an adaptive segmentation quality assessment probe, which generates an adaptive segmentation threshold based on segmentation quality, binarizes the segmentation results, and scores the segmentation results, addressing the problem of excessively high instance segmentation scores. These designs enable this invention to effectively solve the problem of industrial defect instance segmentation.

[0066] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An industrial defect instance segmentation method, characterized in that, The method comprises the following steps: Step S1, training an industrial defect instance segmentation network (1), using an image with instance-level annotation as training data input into the training industrial defect instance segmentation network model, and using a loss function to constrain the training process; Wherein step S1 specifically comprises: step S1-1, industrial image feature extraction, using a convolutional neural network-based backbone network to extract the fusion features of the input image; Step S1-2, industrial defect positioning based on a cascaded defect region proposal module, inputting the fusion features F1, F2, F3, F4 and F5 into the cascaded defect region proposal network to locate the potential defect region; Step S1-3, industrial defect instance segmentation based on adaptive segmentation quality evaluation, inputting the obtained ROI features G1, G2, G3, G4 and G5 into the industrial defect instance segmentation module to generate instance segmentation results; the industrial defect instance segmentation module is composed of a detection probe, a segmentation probe and an adaptive segmentation quality evaluation probe: the detection probe is used to generate candidate box regression value and classification score; the segmentation probe is used to generate segmentation results; the adaptive segmentation quality evaluation probe calculates a dynamic threshold according to the output of the segmentation probe to binarize the segmentation results, outputs a segmentation quality score for the binarized segmentation results, and multiplies the score with the classification score to obtain the final instance segmentation score; the detection probe is specifically used for: Step S1-3(1), input ROI feature G1 into the detection probe, use the pooling layer, aiming to down-sample the feature to a fixed size , down-sample the down-sampled feature using a 7x7 convolution kernel and increase the dimension , followed by a 1x1 convolution kernel to encode the feature; use two fully connected layers to positionally encode the activated feature, followed by two parallel fully connected layers to obtain the candidate box regression value L1 and the class score prediction P1; Step S1-3(2), input ROI feature G2 into the detection probe, use a pooling layer aiming to down-sample the feature to a fixed size , down-sample the down-sampled feature using a 7x7 convolution kernel and increase the dimension , followed by a 1x1 convolution kernel to encode the feature; use two fully connected layers to positionally encode the activated feature, followed by two parallel fully connected layers to obtain the candidate box regression value L2 and the class score prediction P2; Step S1-3 (3), input ROI feature G3 into the detection probe, use a pooling layer, aiming to down-sample the feature to a fixed size , down-sample the down-sampled feature using a 7x7 convolution kernel and increase the dimension , followed by a 1x1 convolution kernel to encode the feature; use two fully connected layers to positionally encode the activated feature, followed by two parallel fully connected layers to obtain the candidate box regression value L3 and the class score prediction P3; Step S1-3 (4), input ROI feature G4 into the detection probe, use the pooling layer, aiming to down-sample the feature to a fixed size , down-sample the down-sampled feature using a 7x7 convolution kernel and increase the dimension , followed by a 1x1 convolution kernel to encode the feature; use two fully connected layers to positionally encode the activated feature, followed by two parallel fully connected layers to obtain the candidate box regression value L4 and the class score prediction P4; Step S1-3 (5), input ROI feature G5 into the detection probe, use a pooling layer aiming to down-sample the feature to a fixed size , down-sample the down-sampled feature using a 7x7 convolution kernel and increase the dimension , followed by a 1x1 convolution kernel to encode the feature; use two fully connected layers to positionally encode the activated feature, followed by two parallel fully connected layers to obtain the candidate box regression value L5 and the class score prediction P5; Step S2, segmentation stage, inputting a test image (2) containing an industrial defect region into the industrial defect instance segmentation network (1) to obtain an industrial defect instance segmentation result.

2. The industrial defect instance segmentation method of claim 1, wherein, The step S1-1 specifically comprises: Step S1-1(1), using the ResNet50 pre-trained on ImageNet as the feature extraction network to extract features of the input image, the image feature maps output from the first to the fourth convolutional blocks have sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image respectively, and the channel numbers are 64, 128, 256 and 512 respectively. , are preset values respectively. Step S1-1 (2), the feature map of 1 / 32 resolution output by the fourth convolution block of the backbone network extraction is dimensionally reduced through a convolution layer to obtain the fusion feature , and then the resolution is reduced to 1 / 2 of the original resolution using maximum pooling to obtain the fusion feature ; Step S1-1(3), the 1 / 16 resolution feature map output by the third convolutional block of the backbone network extraction is dimensionally reduced through a convolutional layer The F1 is nearest neighbor up-sampled and fused with the former by using a 1x1 convolution after stacking The fused feature F3 is obtained Step S1-1 (4), the feature map of 1 / 8 resolution output by the third convolutional block of the backbone network extraction is dimensionally reduced through a convolutional layer The F1 is fused by nearest neighbor up-sampling and 1x1 convolution after stacking The fused feature F4 is obtained Step S1-1 (5), the 1 / 4 resolution feature map output by the third convolutional block of the backbone network extraction is smoothed through a convolutional layer, the nearest neighbor up-sampling is performed on F1, and then the former is stacked and fused through 1x1 convolution The fused feature F5 is obtained.

3. The industrial defect instance segmentation method of claim 1, wherein, The step S1-2 specifically comprises: Step S1-2(1), using a 3×3 hollow convolution kernel to encode the fusion features F1 and using a ReLu layer to activate, then using a regressor to perform a candidate box regression to obtain a regression value L1; modifying the preset candidate box in the fusion features F1 according to the regression value L1, using adaptive convolution to encode the features, then connecting a classifier and a regressor to perform secondary regression and category score on the potential defect, to obtain a secondary regression value R1 and a category score S1; modifying the preset candidate box according to the secondary regression value R1, selecting the top K candidate boxes with the highest foreground score based on the category score S1, using non-maximum suppression to remove redundant candidate boxes, to obtain the candidate boxes A1 corresponding to the fusion features F1, K is a preset value; using ROI Align to extract the ROI features G1 in the fusion features F1 based on the candidate boxes A1; Step S1-2(2), feature encoding is performed on the fusion feature F2 using a 3*3 hollow convolution kernel and the ReLu layer is activated, and then a regression value L2 is obtained by performing one candidate frame regression using a regressor; the preset candidate frame in the fusion feature F2 is modified according to the regression value L2, feature encoding is performed using adaptive convolution, and then a secondary regression and a category score are performed on the potential defect by the classifier and the regressor, to obtain a secondary regression value R2 and a category score S2; the preset candidate frame is modified according to the secondary regression value R2, the top K candidate frames with the highest foreground scores are selected based on the category score S2, redundant candidate frames are removed using non-maximum suppression, to obtain the candidate frame A2 corresponding to the fusion feature F2, and K is a preset value; the ROI feature G2 in the fusion feature F2 is extracted based on the candidate frame A2 using the ROI Align; Step S1-2(3), feature encoding is performed on the fusion feature F3 using a 3*3 hollow convolution kernel and the ReLu layer is activated, and then a regression value L3 is obtained by performing one candidate frame regression using a regressor; the preset candidate frame in the fusion feature F3 is modified according to the regression value L3, feature encoding is performed using adaptive convolution, and then a secondary regression and a category score are performed on the potential defect by the classifier and the regressor, to obtain a secondary regression value R3 and a category score S3; the preset candidate frame is modified according to the secondary regression value R3, the top K candidate frames with the highest foreground scores are selected based on the category score S3, redundant candidate frames are removed using non-maximum suppression, to obtain the candidate frame A3 corresponding to the fusion feature F3, and K is a preset value; the ROI feature G3 in the fusion feature F3 is extracted based on the candidate frame A3 using the ROI Align; Step S1-2(4), feature encoding is performed on the fusion feature F4 using a 3*3 hollow convolution kernel and the ReLu layer is activated, and then a regression value L4 is obtained by performing one candidate frame regression using a regressor; the preset candidate frame in the fusion feature F4 is modified according to the regression value L4, feature encoding is performed using adaptive convolution, and then a secondary regression and a category score are performed on the potential defect by the classifier and the regressor, to obtain a secondary regression value R4 and a category score S4; the preset candidate frame is modified according to the secondary regression value R4, the top K candidate frames with the highest foreground scores are selected based on the category score S4, redundant candidate frames are removed using non-maximum suppression, to obtain the candidate frame A4 corresponding to the fusion feature F4, and K is a preset value; the ROI feature G4 in the fusion feature F4 is extracted based on the candidate frame A4 using the ROI Align; Step S1-2 (5), the fusion feature F5 is encoded using a 3*3 hollow convolution kernel and activated by a ReLu layer, and then a regressor is used for one candidate box regression to obtain a regression value L5; the preset candidate box in the fusion feature F5 is modified according to the regression value L5, and then encoded using an adaptive convolution, followed by a classifier and a regressor for secondary regression and category score of the potential defect, to obtain a secondary regression value R5 and a category score S5; the preset candidate box is modified according to the secondary regression value R5, the top K candidate boxes with the highest foreground scores are selected based on the category score S5, redundant candidate boxes are removed using non-maximum suppression, to obtain candidate boxes A5 corresponding to the fusion feature F5, and K is a preset value; the ROI feature G5 in the fusion feature F1 is extracted based on the candidate boxes A5 using ROI Align.

4. The industrial defect instance segmentation method of claim 1, wherein, The segmentation probe is specifically used for: Step S1-3 (6), input ROI feature G1 into the segmentation probe, use a pooling layer aiming to down-sample the feature to a fixed size , encode the down-sampled feature using 4 3x3 convolution kernels, followed by a 2x2 de-convolution layer to up-sample the encoded feature , followed by a 3x3 convolution kernel to generate the segmentation result T1; Step S1-3 (7), input ROI feature G2 into the segmentation probe, use a pooling layer aiming to down-sample the feature to a fixed size , encode the down-sampled feature using 4 3x3 convolution kernels, followed by a 2x2 de-convolution layer to up-sample the encoded feature , followed by a 3x3 convolution kernel to generate the segmentation result T2; Step S1-3 (8), input ROI feature G3 into the segmentation probe, use a pooling layer aiming to down-sample the feature to a fixed size , encode the down-sampled feature using 4 3x3 convolution kernels, followed by a 2x2 de-convolution layer to up-sample the encoded feature , followed by a 3x3 convolution kernel to generate the segmentation result T3; Step S1-4 (9), input ROI feature G4 into the segmentation probe, use a pooling layer aiming to down-sample the feature to a fixed size , encode the down-sampled feature using 4 3x3 convolution kernels, followed by a 2x2 de-convolution layer to up-sample the encoded feature , followed by a 3x3 convolution kernel to generate the segmentation result T4; Step S1-3 (10), input ROI feature G5 into the segmentation probe, use a pooling layer aiming to down-sample the feature to a fixed size , encode the down-sampled feature using 4 3x3 convolution kernels, followed by a 2x2 de-convolution layer to up-sample the encoded feature , followed by a 3x3 convolution kernel to generate the segmentation result T5.

5. The industrial defect instance segmentation method of claim 1, wherein, The adaptive segmentation quality evaluation probe is specifically used for: Step S1-3 (6), the segmentation result T1 is input into the adaptive segmentation quality evaluation probe, global average pooling and a full connection layer are used, and then a Sigmoid operation is used to obtain an adaptive segmentation threshold M1; The binarization is performed on the segmentation result T1 using an adaptive segmentation threshold, and the binarized segmentation result is down-sampled to 1 / 4 resolution through a max-pooling operation, and then is superimposed with the ROI feature G1 in the channel dimension, and then is down-sampled and reduced in dimension through four 3x3 convolution kernels , followed by two fully connected layers to obtain a segmentation quality score E1. Step S1-3 (7), the segmentation result T2 is input into the adaptive segmentation quality evaluation probe, global average pooling and a full connection layer are used, and then a Sigmoid operation is used to obtain an adaptive segmentation threshold M2; The binarization result is down-sampled to 1 / 4 resolution by maximum pooling operation and superimposed with the ROI feature G2 in the channel dimension, and then down-sampled and reduced in dimension by 4 3x3 convolution kernels , followed by two fully connected layers to obtain the segmentation quality score E2; Step S1-3 (8), the segmentation result T3 is input into the adaptive segmentation quality evaluation probe, global average pooling and a full connection layer are used, and then a Sigmoid operation is used to obtain an adaptive segmentation threshold M3; The binarization is performed on the segmentation result T3 using an adaptive segmentation threshold, and the binarized segmentation result is down-sampled to 1 / 4 resolution through a max-pooling operation, and then is superimposed with the ROI feature G3 in the channel dimension, and then is down-sampled and reduced in dimension through four 3x3 convolution kernels , followed by two fully connected layers to obtain a segmentation quality score E3; Step S1-3 (9), the segmentation result T4 is input into the adaptive segmentation quality evaluation probe, global average pooling and a full connection layer are used, and then a Sigmoid operation is used to obtain an adaptive segmentation threshold M4; The binarization is performed on the segmentation result T4 using an adaptive segmentation threshold, and the binarized segmentation result is down-sampled to 1 / 4 resolution through a max-pooling operation, and then is superimposed with the ROI feature G4 in the channel dimension, and then is down-sampled and reduced in dimension through four 3x3 convolution kernels , followed by two fully connected layers to obtain a segmentation quality score E4; Step S1-3 (10), the segmentation result T5 is input into the adaptive segmentation quality evaluation probe, global average pooling and a full connection layer are used, and then a Sigmoid operation is used to obtain an adaptive segmentation threshold M5; The binarization is performed on the segmentation result T5 using an adaptive segmentation threshold, and the binarized segmentation result is down-sampled to 1 / 4 resolution by a max-pooling operation, and then is superimposed with the ROI feature G1 in the channel dimension, and then is down-sampled and reduced in dimension by 4 3x3 convolution kernels , followed by two fully connected layers to obtain the segmentation quality score E5.

6. The industrial defect instance segmentation method of claim 5, wherein, After obtaining the five segmentation quality scores, the segmentation quality scores E1, E2, E3, E4, and E5, the candidate box regression values L1, L2, L3, L4, and L5, the category score predictions P1, P2, P3, P4, and P5, the adaptive segmentation thresholds M1, M2, M3, M4, and M5, and the segmentation results T1, T2, T3, T4, and T5 are added to the loss calculated by the corresponding true value labels to obtain a total loss for updating the model parameters.

7. The industrial defect instance segmentation method of claim 1, wherein, The step S2 specifically comprises: in the segmentation stage, inputting a to-be-detected defect sample into the industrial defect instance segmentation network (1), and outputting, by the industrial defect instance segmentation module, an instance segmentation result including an instance segmentation score Z obtained by multiplying a candidate box regression value L, a category score P, and a segmentation quality score E, and a segmentation result T.

Images