Small sample based dual-branch profile augmented industrial product defect detection method

Abstract

The disclosed small sample based double-branch contour enhanced industrial product defect detection method comprises the following steps: 1. Constructing a positive sample image dataset and a defect sample image dataset of industrial products; 2. Constructing a background removal module to remove the background of a reference image and a to-be-registered image; 3. Constructing an image rotation coarse registration module to rotate a first to-be-registered image to align with a first reference image; 4. Constructing a double-branch feature level fine registration module to solve the mean and variance of a feature map c as a template feature of the industrial product; 5. Solving the feature map d and the abnormal score matrix of the same industrial product defect sample image; and 6. Establishing an adaptive defect threshold segmentation network to obtain a defect segmentation image. The present application realizes a better balance between detection accuracy and model generality in a small sample scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision, and relates to a double-branch contour enhancement type industrial product defect detection method based on small samples. BACKGROUND

[0002] In the field of intelligent manufacturing and industrial automation, defect detection is a key link to ensure product quality, and machine vision technology based on deep learning is the mainstream key technology in the current defect detection field. However, in the actual industrial scene, there are many challenges: 1. Defect sample scarcity: Due to strict quality control, there are few defective products, resulting in a number of defect samples for training far lower than normal samples. This imbalance makes it difficult for the model to accurately identify a small number of defects; 2. Defect diversity and unpredictability: Industrial products may have various defect types from minor scratches to major structural damage. In addition, new products or improved products may also bring new defect types, making it almost impossible to collect a dataset that fully covers all potential defect types; 3. Improve the generalization ability of the model. Existing fully supervised models need to be trained with sufficient labeled samples for each defect type. For new sample types not observed during training, the model's performance drops significantly when tested directly on them, making it unable to adapt to the modern industrial demand for rapid production and frequent product updates. SUMMARY

[0003] The purpose of the present application is to provide a double-branch contour enhancement type industrial product defect detection method based on small samples, which solves the problem of low model detection precision and weak generalization ability caused by the small number of samples and the difficulty of feature alignment in the prior art.

[0004] The technical solution adopted by the present application is a double-branch contour enhancement type industrial product defect detection method based on small samples, comprising the following steps:

[0005] S1: Constructing an industrial product positive sample image dataset and a defect sample image dataset;

[0006] S2: Constructing a background removal module, inputting two different positive sample images of the same industrial product, one as a reference image and the other as a to-be-registered image, removing the background of the reference image and the to-be-registered image using the Otsu binarization method and the Canny edge detection operator respectively, to obtain a first reference image and a first to-be-registered image;

[0007] S3: Constructing an image rotation coarse registration module, rotating the first to-be-registered image to align with the first reference image to obtain a second to-be-registered image;

[0008] S4: Construct a double-branch feature-level fine registration module, and after training with positive sample image data set, input the first reference image and the second to-be-registered image for fine alignment, obtain the feature map a of the first reference image and the feature map b of the second to-be-registered image, perform channel dimension splicing on the up-sampling of the feature map a and the feature map b to obtain a feature map c, and solve the mean and variance of the feature map c as the template feature of the industrial product;

[0009] S5: Solve the feature map d of the same industrial product defect sample image, calculate the Mahalanobis distance between the feature map d and the template feature as the anomaly score matrix of the defect sample image;

[0010] S6: Establish an adaptive defect threshold segmentation network, solve the best global threshold for segmentation, and segment the anomaly score matrix by using the global threshold for segmentation to obtain a defect segmented image.

[0011] The characteristics of the present application are:

[0012] The small sample-based double-branch contour-enhanced industrial product defect detection method in S2 uses Otsu binarization method and Canny edge detection operator to remove background from the reference image and the to-be-registered image, specifically:

[0013] S2.1: The Otsu binarization method is used to extract the target object in the reference image and the to-be-registered image;

[0014] S2.2: The edge information of the target object is extracted by using the Canny edge detection operator, and the edge information is used for constraint processing of the target object: starting from the four peripheral edge pixels of the reference image and the to-be-registered image, traversing each pixel inward, and setting all the traversed pixels as background before encountering an edge pixel.

[0015] The small sample-based double-branch contour-enhanced industrial product defect detection method in S3 rotates the first to-be-registered image to align with the first reference image, specifically: the center axis angle alpha of the circumscribed rectangle of the maximum connected region of the first reference image is calculated, the center axis angle beta of the circumscribed rectangle of the maximum connected region of the first to-be-registered image is calculated, and the rotation matrix is constructed by comparing the angle difference between the center axis angle alpha and the center axis angle beta to rotate the first to-be-registered image to align with the first reference image.

[0016] The small sample-based double-branch contour-enhanced industrial product defect detection method in S4 is a double-branch feature-level fine registration module, specifically: a double-subnet structure is constructed on the basis of ResNet-18 network, the parameters of the two subnets are shared, a double-branch contour-enhanced spatial transformation network submodule is added after the first layer convolution block of each subnetwork, a spatial transformation network STN submodule is added after the second layer and third layer convolution blocks, and a negative cosine loss function is used as the loss function;

[0017] The mean and variance of the feature map c in S4 are solved by using a multivariate Gaussian distribution estimator.

[0018] The double-branch contour-enhanced industrial product defect detection method based on a small sample comprises a double-branch contour-enhanced spatial transformation network submodule, which is composed of two branches: a first branch is a main branch, the input is an original image fused with contour information, which is composed of two convolutional layers with a kernel size of 3*3, and a maximum pooling layer is arranged after each convolutional layer; a second branch is an auxiliary branch, the input is an image containing only contour information extracted from the original image by using a Canny edge detection operator, which is composed of convolutional layers with a kernel size of 3*3, and a maximum pooling layer is arranged after each convolutional layer; the main branch and the auxiliary branch process the two inputs respectively, spatial features are extracted through a convolutional neural network, and then the spatial features are fused through a full connection layer to predict transformation parameters and construct a transformation matrix to perform alignment transformation on the image.

[0019] In S5, the feature map d of the same industrial product defect sample image is solved, specifically: the reference image and a defect sample image of the same industrial product are respectively processed through the background removal module, the image rotation coarse registration module and the double-branch feature level fine registration module, and then the channel dimension is spliced after up-sampling to obtain the feature map d of the defect sample image.

[0020] In S6, the best segmentation global threshold is solved, specifically: the abnormal score matrix and the corresponding defect real mask image are input into the adaptive defect threshold segmentation network, the defect threshold is automatically determined by using the neural network, the abnormal score matrix is subtracted from the defect threshold, and the soft binarization processing is performed by using the activation function ReLU and normalization, the MSE loss between the predicted soft binarization defect segmentation image and the defect real mask image is minimized, and the best segmentation global threshold of the abnormal score matrix is obtained.

[0021] The beneficial effects of the present application are:

[0022] 1. The background removal and image level coarse alignment based on edge information constraint avoid the interference of sample background area change on feature extraction and reduce the difficulty of feature matching;

[0023] 2. The double-branch contour-enhanced spatial transformation network submodule constructed enhances the spatial characteristic perception ability of the model and effectively reduces the detection error caused by the non-complete alignment between images;

[0024] 3. The adaptive defect threshold segmentation network established reduces the dependence on labeled defect samples and improves the adaptability to new samples. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of the defect detection method of the present application;

[0026] Figure 2 is a defect sample image of the tablet in embodiment 1 of the present application;

[0027] Figure 3 is a reference image of the tablet in embodiment 1 of the present application;

[0028] Figure 4 is a defect segmentation image of the tablet in embodiment 1 of the present application;

[0029] Figure 5 is a defect sample image of the cable product in embodiment 2 of the present application;

[0030] Figure 6 is a reference image of the cable product in embodiment 2 of the present application;

[0031] Figure 7 is a defect segmentation image of the cable product in embodiment 2 of the present application;

[0032] Figure 8 is a defect sample image of the metal nut in embodiment 3 of the present application;

[0033] Figure 9 is a reference image of the metal nut in embodiment 3 of the present application;

[0034] Figure 10 is a defect segmentation image of the metal nut in embodiment 3 of the present application. DETAILED DESCRIPTION

[0035] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0036] The present application provides a small sample-based double-branch contour-enhanced industrial product defect detection method, as shown in Figure 1 , comprising the following steps:

[0037] S1: Collecting multiple types of defect-free and defective industrial product images, constructing a positive sample image dataset and a defect sample image dataset of industrial products.

[0038] S2: Constructing a background removal module, inputting two different positive sample images of the same industrial product, one as a reference image and the other as a to-be-registered image, using Otsu binarization method and Canny edge detection operator to remove the background of the reference image and the to-be-registered image respectively, obtaining a first reference image and a first to-be-registered image;

[0039] The Otsu binarization method and the Canny edge detection operator are used to remove the background of the reference image and the to-be-registered image respectively, specifically:

[0040] S2.1: extracting target objects in the reference image and the image to be registered by using the Otsu binarization method;

[0041] S2.2: extracting edge information of the target objects by using the Canny edge detection operator, and performing constraint processing on the target objects by using the edge information: starting from the peripheral edge pixels of the reference image and the image to be registered, traversing each pixel inward, setting all the traversed pixels as background before encountering the edge pixels, and after the traversal, the background of the reference image and the image to be registered is removed.

[0042] S3: constructing an image rotation coarse registration module, rotating the first image to be registered to align with the first reference image to obtain a second image to be registered;

[0043] The first image to be registered is rotated to align with the first reference image, specifically: the center axis angle α of the circumscribed rectangle of the largest connected region of the first reference image is obtained, the center axis angle β of the circumscribed rectangle of the largest connected region of the first image to be registered is obtained, and the rotation matrix is constructed by comparing the angle difference between the center axis angle α and the center axis angle β to rotate the first image to be registered to align with the first reference image, thereby realizing coarse alignment of the first image to be registered and the first reference image at the image level.

[0044] S4: constructing a double-branch feature-level fine registration module, and after training with positive sample image data set, inputting the first reference image and the second image to be registered for fine alignment to obtain the feature map a of the first reference image and the feature map b of the second image to be registered, and after channel dimension splicing of the up-sampling of the feature map a and the feature map b, the feature map c is obtained, and the mean and variance of the feature map c are solved by using the multivariate Gaussian distribution estimator as the template feature of the industrial product;

[0045] The double-branch feature-level fine registration module is specifically constructed as a double-subnet structure based on the ResNet-18 network, the parameters of the two subnets are shared, each subnetwork adds a double-branch contour enhancement spatial transformation network submodule after the first layer convolution block, adds a spatial transformation network STN submodule after the second layer and the third layer convolution block, and the loss function adopts a negative cosine loss function;

[0046] The double-branch contour enhancement spatial transformation network sub-module is composed of two branches: a first branch is a main branch, the input is an original image fused with contour information, composed of two convolution layers with a kernel size of 3*3, and a maximum pooling layer is arranged after each convolution layer; a second branch is an auxiliary branch, the input is an image containing only contour information extracted from the original image by using a Canny edge detection operator, composed of convolution layers with a kernel size of 3*3, and a maximum pooling layer is arranged after each convolution layer; the main branch and the auxiliary branch process the two inputs respectively, spatial features are extracted through the convolutional neural network, and then the spatial features are fused through a fully connected layer to predict a transformation parameter to construct a transformation matrix, and the image is aligned and transformed.

[0047] S5: solving the feature map d of the same industrial product defect sample image, calculating the Mahalanobis distance between the feature map d and the template feature as the anomaly score matrix of the defect sample image;

[0048] Solving the feature map d of the same industrial product defect sample image, specifically: the reference image and a defect sample image of the same industrial product are respectively processed through the background removal module, the image rotation coarse registration module and the double-branch feature level fine registration module, and then the channel dimension is spliced after up-sampling to obtain the feature map d of the defect sample image.

[0049] S6: establishing an adaptive defect threshold segmentation network, solving the best segmentation global threshold, and segmenting the anomaly score matrix by using the segmentation global threshold to obtain a defect segmentation image;

[0050] Solving the best segmentation global threshold, specifically: inputting the anomaly score matrix and the corresponding defect real mask image into the adaptive defect threshold segmentation network, automatically determining the defect threshold by using the neural network, subtracting the defect threshold from the anomaly score matrix, and performing soft binarization processing by using the activation function ReLU and normalization, minimizing the MSE loss between the predicted soft binarization defect segmentation image and the defect real mask image, and obtaining the best segmentation global threshold of the anomaly score matrix.

[0051] The double-branch contour enhancement industrial product defect detection method based on small samples provided by the application enhances the attention of the network to the relative relationship between samples, reduces the dependence on absolute label information and the demand for the number of labeled samples, meets the rapid iteration and diversification demand of industrial production, can quickly adapt to the characteristics of new products, efficiently identifies various unknown defects that may occur in the production process, ensures the double improvement of production efficiency and product quality, and realizes a better balance between detection accuracy and model universality in a small sample scene.

[0052] Example 1

[0053] The method provided by the application is used for defect detection of tablets, including the following steps:

[0054] S1: Constructing a positive sample image dataset and a defect sample image dataset of the tablet;

[0055] S2: Constructing a background removal module, inputting two different positive sample images of the tablet, one as a reference image (as shown in Figure 3 ), and the other as a to-be-registered image, removing the background of the reference image and the to-be-registered image by using Otsu binarization method and Canny edge detection operator respectively to obtain a first reference image and a first to-be-registered image;

[0056] S3: Constructing an image rotation coarse registration module, rotating the first to-be-registered image to align with the first reference image to obtain a second to-be-registered image;

[0057] S4: Constructing a double-branch feature-level fine registration module, and after training with the positive sample image dataset, inputting the first reference image and the second to-be-registered image for fine alignment to obtain a feature map a of the first reference image and a feature map b of the second to-be-registered image, performing channel dimension splicing on the feature map a and the feature map b after upsampling to obtain a feature map c, and solving the mean and variance of the feature map c as the template feature of the tablet;

[0058] S5: Inputting a reference image (as shown in Figure 3 ) of the tablet and a defect sample image (as shown in Figure 2 ) into the background removal module, the image rotation coarse registration module and the double-branch feature-level fine registration module respectively, and then performing channel dimension splicing after upsampling to obtain a feature map d of the defect sample image, calculating the Mahalanobis distance between the feature map d and the template feature as an anomaly score matrix of the defect sample image;

[0059] S6: Establishing an adaptive defect threshold segmentation network, solving the best global threshold for segmentation, and segmenting the anomaly score matrix by using the global threshold for segmentation to obtain a defect segmentation image of the tablet, as shown in Figure 4 .

[0060] Example 2

[0061] The method provided by the application is used for defect detection of a cable product, including the following steps:

[0062] S1: Constructing a positive sample image dataset and a defect sample image dataset of the tablet;

[0063] S2: Constructing a background removal module, inputting two different positive sample images of the tablet, one as a reference image (as shown in Figure 6 ), and the other as a to-be-registered image, removing the background of the reference image and the to-be-registered image by using Otsu binarization method and Canny edge detection operator respectively to obtain a first reference image and a first to-be-registered image;

[0064] S3: constructing an image rotation coarse registration module, rotating the first to-be-registered image to align with the first reference image to obtain a second to-be-registered image;

[0065] S4: constructing a double-branch feature-level fine registration module, and after training with positive sample image data set, inputting the first reference image and the second to-be-registered image for fine alignment to obtain a feature map a of the first reference image and a feature map b of the second to-be-registered image, performing channel dimension splicing on the feature map a and the feature map b after upsampling to obtain a feature map c, and solving the mean and variance of the feature map c as a template feature of the cable product;

[0066] S5: as shown in FIG. 5, taking a reference image (as shown in FIG. 5) of the cable product and a defect sample image (as shown in FIG. 5) as an example, respectively processing the two images through the background removal module, the image rotation coarse registration module and the double-branch feature-level fine registration module, and then performing channel dimension splicing after upsampling to obtain a feature map d of the defect sample image, calculating the Mahalanobis distance between the feature map d and the template feature as an anomaly score matrix of the defect sample image; Figure 6 Figure 5 S5: as shown in FIG. 5, taking a reference image (as shown in FIG. 5) of the cable product and a defect sample image (as shown in FIG. 5) as an example, respectively processing the two images through the background removal module, the image rotation coarse registration module and the double-branch feature-level fine registration module, and then performing channel dimension splicing after upsampling to obtain a feature map d of the defect sample image, calculating the Mahalanobis distance between the feature map d and the template feature as an anomaly score matrix of the defect sample image;

[0067] S6: establishing an adaptive defect threshold segmentation network, solving the best global threshold for segmentation, and segmenting the anomaly score matrix by using the global threshold for segmentation to obtain a defect segmentation image of the cable product, as shown in FIG. 6. Figure 7

[0068] Embodiment 3

[0069] The method provided by the application is used for defect detection of a metal nut, and includes the following steps:

[0070] S1: constructing a positive sample image data set and a defect sample image data set of the metal nut;

[0071] S2: constructing a background removal module, inputting two different positive sample images of the metal nut, one as a reference image (as shown in FIG. 7) and the other as a to-be-registered image, and respectively removing the backgrounds of the reference image and the to-be-registered image by using Otsu binarization method and Canny edge detection operator to obtain a first reference image and a first to-be-registered image; Figure 9

[0072] S3: constructing an image rotation coarse registration module, rotating the first to-be-registered image to align with the first reference image to obtain a second to-be-registered image;

[0073] ​​​S4: Construct a double-branch feature-level fine registration module, and after training with the positive sample image dataset, input the first reference image and the second to-be-registered image for fine alignment to obtain the feature map a of the first reference image and the feature map b of the second to-be-registered image, perform channel dimension splicing on the up-sampling of the feature map a and the feature map b to obtain a feature map c, and solve the mean and variance of the feature map c as the template feature of the metal nut;

[0074] S5: The reference image (as shown in FIG. 4) of the metal nut and a defect sample image (as shown in FIG. 5) are respectively processed by the background removal module, the image rotation coarse registration module and the double-branch feature-level fine registration module, and then after up-sampling and channel dimension splicing, a feature map d of the defect sample image is obtained, and the Mahalanobis distance between the feature map d and the template feature is calculated as the anomaly score matrix of the defect sample image. Figure 9 Figure 8 S5: The reference image (as shown in FIG. 4) of the metal nut and a defect sample image (as shown in FIG. 5) are respectively processed by the background removal module, the image rotation coarse registration module and the double-branch feature-level fine registration module, and then after up-sampling and channel dimension splicing, a feature map d of the defect sample image is obtained, and the Mahalanobis distance between the feature map d and the template feature is calculated as the anomaly score matrix of the defect sample image.

[0075] S6: Establish an adaptive defect threshold segmentation network, solve the best global threshold for segmentation, and segment the anomaly score matrix with the global threshold for segmentation to obtain the defect segmentation image of the metal nut, as shown in FIG. 6. Figure 10

[0076] As can be seen from Examples 1-3, the defect area of the tablet, the cable product and the metal nut can be accurately detected by the method of the present application, and the method of the present application can adapt to defect detection of different types of industrial products, and has wide product defect detection adaptability.​​

Claims

1. A dual-branch contour enhancement method for industrial product defect detection based on small samples, characterized by: The following steps are involved: S1: Construct a dataset of positive sample images and defect sample images of industrial products; S2: Construct a background removal module, input two different positive sample images of the same industrial product, one as the reference image and the other as the image to be registered, use the Otsu binarization method and the Canny edge detection operator to remove the background of the reference image and the image to be registered respectively, and obtain the first reference image and the first image to be registered; S3: constructing an image rotation coarse registration module to rotate the first image to be registered until it is aligned with the first reference image to obtain a second image to be registered; S4: Construct a dual-branch feature-level precise registration module and, after training it with a positive sample image dataset, input the first reference image and the second image to be registered for precise alignment. This module obtains feature map a of the first reference image and feature map b of the second image to be registered. The feature maps a and b are upsampled and concatenated in the channel dimension to obtain feature map c. The mean and variance of feature map c are calculated and used as the template features of the industrial product. S5: Solve the feature map d of the defect sample image of the same industrial product, calculate the Mahalanobis distance between the feature map d and the template feature, and use it as the abnormality score matrix of the defect sample image; S6: Establish an adaptive defect threshold segmentation network, solve the optimal global segmentation threshold, use the global segmentation threshold to segment the anomaly score matrix, and obtain the defect segmentation image.

2. The small sample-based dual-branch contour enhancement method for industrial product defect detection according to claim 1 is characterized in that: In S2, the Otsu binarization method and the Canny edge detection operator are used to remove the background of the reference image and the image to be registered, respectively. Specifically: S2.1: Extract the target objects in the reference image and the image to be registered using the Otsu binarization method; S2.2: Use the Canny edge detection operator to extract the edge information of the target object, and use the edge information to constrain the target object: starting from the edge pixels around the reference image and the image to be registered, traverse each pixel inward. Before encountering an edge pixel, set the traversed pixels as the background.

3. The small sample-based dual-branch contour enhancement method for industrial product defect detection according to claim 1, characterized in that: In S3, the first image to be registered is rotated to be aligned with the first reference image, specifically: the central axis angle α of the circumscribed rectangle of the maximum connected area of ​​the first reference image is calculated, and the central axis angle β of the circumscribed rectangle of the maximum connected area of ​​the first image to be registered is calculated. By comparing the angle difference between the central axis angle α and the central axis angle β, a rotation matrix is ​​constructed to rotate the first image to be registered to be aligned with the first reference image.

4. The small sample-based dual-branch contour enhancement method for industrial product defect detection according to claim 1, characterized in that: The dual-branch feature-level fine registration module in S4 is specifically constructed by building a dual-subnet structure based on the ResNet-18 network. The two subnets share parameters. Each subnet adds a dual-branch contour enhancement spatial transformer network submodule after the first convolution block, and adds a spatial transformer network STN submodule after the second and third convolution blocks. The loss function adopts the negative cosine loss function; The mean and variance of the characteristic graph c are solved in S4 by using a multivariate Gaussian distribution estimator.

5. The small sample-based dual-branch contour enhancement method for industrial product defect detection according to claim 4, characterized in that: The dual-branch contour enhancement spatial transformation network submodule consists of two branches: the first branch is the main branch, and its input is the original image fused with contour information. It is composed of two convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the second branch is the auxiliary branch, and its input is an image containing only contour information extracted from the original image using the Canny edge detection operator. It is composed of convolutions with a kernel size of 3×3, and each convolution layer is followed by a maximum pooling layer; the main branch and the auxiliary branch process the two inputs respectively, extract spatial features through a convolutional neural network, and then fuse these spatial features through a fully connected layer to predict transformation parameters and construct a transformation matrix to perform image alignment transformation.

6. The small sample-based dual-branch contour enhancement method for industrial product defect detection according to claim 1, characterized in that: The feature map d of the defective sample image of the same industrial product is solved in S5. Specifically, the reference image and a defective sample image of the same industrial product are processed by a background removal module, an image rotation coarse registration module, and a dual-branch feature-level fine registration module, respectively, and then upsampled and spliced ​​in the channel dimension to obtain the feature map d of the defective sample image.

7. The small sample-based dual-branch contour enhancement method for industrial product defect detection according to claim 1, characterized in that: The optimal global segmentation threshold is solved in S6, specifically by inputting the anomaly score matrix and the corresponding defect true mask map into the adaptive defect threshold segmentation network, automatically determining the defect threshold using the neural network, subtracting the defect threshold from the anomaly score matrix, and performing soft binarization processing using the activation function ReLU and normalization, minimizing the MSE loss between the predicted soft binary defect segmentation map and the defect true mask map, and obtaining the optimal global segmentation threshold for the anomaly score matrix.

Images