A defect detection method and model based on deep implicit network

Abstract

The application discloses a defect detection method and model based on a deep implicit network, and the method comprises the following steps: S1, model construction: a defect detection model is built by using a multi-scale mixed window self-attention module and a deep implicit network, wherein the model comprises a feature coding branch and an image restoration branch based on the deep implicit network; S2, model training: taking an image with a resolution determined by a specific task instead of a fixed resolution as input, an image feature map, a defect coding map and an image with the same resolution or a high resolution recovered by the deep implicit network are obtained, and an image restoration loss is taken as a main supervision signal; and S3, inference stage: the defect coding map is obtained, and a defect category and a position are determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image signal processing, and particularly relates to a defect detection method and model based on a deep implicit network. BACKGROUND

[0002] Surface defect detection technology has developed from traditional image processing-based algorithms to current deep learning-based algorithms, and has been widely applied in industrial production and manufacturing fields. With the continuous development of deep learning, target detection algorithms based on convolutional neural networks (CNN) have been continuously applied to surface defect detection scenarios. Current research on workpiece defect detection often emphasizes detection accuracy, ignoring computational complexity. Since detection efficiency is highly required in actual production processes, and computing resources are easily limited by operating costs, research on workpiece defect detection needs to consider the comprehensive trade-off between detection accuracy and detection speed in terms of performance under limited resources.

[0003] Traditional research on object surface defect detection is mainly based on image processing algorithms, and uses optical detection technology to achieve fast and accurate detection of target detection objects with stable imaging, fixed patterns, and single scenes. Image processing-based detection algorithms are highly dependent on the imaging quality and stability of the imaging mode of the collected images, and are sensitive to the selection of external light sources, the type of camera, and the parameters of the lens. Clear and stable defect images of the target detection object need to be obtained, and image processing algorithms are used to locate, segment, and filter the target defects to achieve overall visual defect detection. This requires the imaging system to be stable when working and needs to be pre-calibrated and image-corrected. The target image to be detected is obtained by the imaging system, and the image data is processed by the computer processing algorithm. Compared with traditional detection methods, target detection technology based on deep learning can better handle integrated multi-target defect detection tasks with complex relationships, thereby overcoming the application limitations of traditional image processing algorithms in complex and open industrial scenes.

[0004] Deep learning methods for defect detection are divided into fully supervised learning model and unsupervised learning model according to different data labels. At present, most of the surface defect detection based on deep learning is based on supervised representation learning method. The essence of representation learning is to regard the defect detection problem as a classification task in computer vision, including coarse-grained image label classification or region classification, and the most fine-grained pixel classification. Since the goal is completely consistent with the computer vision task, the defect detection method based on representation learning can be regarded as an application of the related classic network in the industrial field. In real industrial production, the huge difference in shape, size, texture, color, background, layout and imaging light of the detection object makes the defect classification in complex environment a difficult task. Due to the powerful feature extraction ability of CNN, the classification network based on CNN has become the most commonly used mode in surface defect classification. However, the surface defect detection technology based on deep learning method also has the problems of high requirement for original sampling data and low detection accuracy in specific scene, and the present application is proposed to overcome these problems. SUMMARY

[0005] The main purpose of the present application is to provide a defect detection method and model based on deep implicit network, so as to solve the technical problems of high data requirement and poor detection accuracy of the existing supervised learning detection method based on deep learning.

[0006] In order to achieve the above purpose, one aspect of the present application proposes the following technical scheme:

[0007] A defect detection method based on deep implicit network, comprising the following steps: S1, model construction: constructing a defect detection model, the model comprising a feature encoding branch and an image restoration branch based on deep implicit network; wherein the feature encoding branch is used to extract features of different levels and different scales in an original image of a to-be-detected object to obtain an image feature map and a defect encoding map; the image restoration branch is used to decode and obtain RGB values of an arbitrary real number point coordinate in an imaging plane with the image feature map, the defect encoding map and pixel position encoding as inputs; S2, training phase: taking an image with a non-fixed resolution determined by a specific task as input, obtaining an image feature map, a defect encoding map and an image with the same resolution or a high resolution restored by deep implicit network; the model takes image restoration loss as the main supervision signal; for a defect-free image, the obtained defect encoding map is constrained to be all 0 output; for an image with defects, firstly, the similarity between the defect encoding of the defect occurrence point and the defect encoding of any point on the image under the condition of no defects is minimized, and secondly, based on the triplet loss, the inter-class distance of the defect encoding of the defect occurrence point is maximized, and the intra-class encoding distance is minimized; S3, inference phase: firstly, traverse the training data set with the defect encoding map as the output to calculate the encoding centers of each class; subsequently, in the inference phase, for any input image, the corresponding image region class is determined according to the distance between the defect encoding map and each class encoding center.

[0008] Further, the feature encoding branch is constructed based on a multi-scale mixed window self-attention module, which uses a multi-scale method and an attention mechanism within the window to make the image feature map contain image details and local and global descriptions of different levels and scales, and at the same time, make the model adapt to different defect types from pixel-level burrs to overall color deviation.

[0009] Further, the feature encoding branch comprises a U-shaped network constructed by a multi-scale mixed window self-attention module, the U-shaped network taking an initial feature map obtained by reducing the image resolution and expanding the channel number as input, in the first half descending branch of the U-shaped network, reducing the sampling input by half in a convolutional manner after every k multi-scale mixed window self-attention module, and expanding the channel number by 2 times at the same time; at the same time, the U-shaped network retains low-level features and fuses multi-level features through a skip connection, the skip connection being a splicing of the output of each down-sampling stage of the first half descending branch of the U-shaped network and the feature map of the corresponding size of the second half ascending branch; wherein 1≤k≤3.

[0010] Further, each multi-scale mixed window self-attention module is formed by connecting a plurality of window self-attention modules with different window sizes in parallel, and is spliced into a feature map output in a fixed order when output.

[0011] Further, the image restoration branch is a deep implicit network, which takes pixel coordinates, feature vectors corresponding to coordinate positions and defect vectors as inputs to restore images pixel by pixel.

[0012] Further, when the specified output resolution is HxW, the imaging plane with a size of [-1, 1; -1, 1] is also uniformly divided into HxW grids, and the feature vectors and the defect vectors of the corresponding points on the image feature map and the defect encoding map are obtained by bilinear interpolation of the coordinate of the grid center point.

[0013] Further, the deep implicit network is a multi-layer perceptron, wherein the output of each layer except the output layer is activated by a sin function, and the output layer adopts a ReLU function activation.

[0014] Another aspect of the present application provides a defect detection model based on a deep implicit network, comprising a feature encoding branch and an image restoration branch connected to the output end of the feature encoding branch; wherein the feature encoding branch is used to extract features of different levels and different scales in the original image of the object to be detected to obtain an image feature map and a defect encoding map; the image restoration branch is used to input the image feature map, the defect encoding map and pixel position encoding to decode and obtain the RGB value of any real point coordinate in the imaging plane.

[0015] The beneficial effects of the present application relative to the prior art include: for the workpiece to be detected, the appearance description may affect the defect description, and the defect description also affects the induction of the defect-free appearance description, so the present application decouples the appearance description of the workpiece to be detected and the defect description of each type while ensuring the information loss in the decoupling process. Therefore, the present application reconstructs the defect detection task into an encoding and restoration process of image information, uses the distribution difference of the defect encoding obtained by the encoder to measure the defect type, and uses a deep implicit function network to restore the image according to the encoding; in terms of network structure, the method uses an implicit network containing a periodic activation function and position encoding to replace the common decoder network with convolution as the main architecture, and uses a feature encoder network structure based on a multi-level window attention mechanism to obtain feature encoding; and further constrains the distribution of different types of feature encoding in addition to the image restoration loss, so that the encoding distance between defect categories is greater than the feature encoding distance within the defect category. The present application improves the precision of defect detection, reduces data dependence, is suitable for detection scenarios with high data requirements, can adapt to different size defect types from pixel-level burrs to overall color deviation, and solves the information loss problem in the encoding-decoding process. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the defect detection method based on the deep implicit network of the embodiment of the present application.

[0017] Figure 2 FIG. 1 is a network architecture diagram of a defect detection model based on a deep implicit network according to an embodiment of the present application.

[0018] Figure 3 FIG. 2 is an exemplary network structure diagram of a multi-scale mixed window self-attention module (M-WSAB) in a feature encoding branch of a defect detection model according to an embodiment of the present application.

[0019] Figure 4 FIG. 3 is a network structure diagram of a window self-attention module (WSAB) in the M-WSAB according to an embodiment of the present application.

[0020] Figure 5 FIG. 4 is a network structure diagram of a feed-forward neural network (FFN) in the WSAB according to an embodiment of the present application.

[0021] Figure 6 FIG. 5 is a reasoning process schematic diagram of a defect detection method based on a deep implicit network according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0023] The surface defect detection technology based on the deep learning method also has the problems of high requirement for the amount of original sampling data and low detection accuracy in specific scenes. In order to overcome this problem, the present application provides a defect detection method based on a deep implicit network and a defect detection model based on a deep implicit network. The core idea of the defect detection method and model based on the deep implicit network provided by the present application includes two aspects: first, decoupling defect encoding and observation feature. Taking an image of a defective workpiece as an example, the observation feature explicitly encodes the image of the workpiece in the current shooting state under the assumption that there is no defect, and the defect encoding encodes the image observation difference caused by the defect. Second, in order to reduce data dependence, a deep implicit network and an image restoration task are introduced for auxiliary training.

[0024] Please refer to Figure 1 The defect detection method based on the deep implicit network provided by the present application based on the above core idea mainly includes three process steps: first, model construction: a defect detection model is constructed by using a multi-scale mixed window self-attention module and a deep implicit network, the model including a feature encoding branch and an image restoration branch based on a deep implicit network; second, model training: the defect detection model is trained by using an image restoration loss and a feature map loss; third, reasoning stage: the trained defect detection model is used to obtain a defect encoding image and determine the defect category and position.

[0025] The specific composition of each process step of the method and the model will be further described in detail below.

[0026] The first step, model construction: construct a defect detection model. Referring to Figure 2 , the model includes a feature encoding branch and an image restoration branch based on a deep implicit network. Wherein, the feature encoding branch is used to extract features of different levels and different scales in the original image of the object to be detected, so as to obtain an image feature map and a defect encoding map; the image restoration branch is used to decode the RGB value of any real point coordinate in the imaging plane by taking the image feature map, the defect encoding map and the pixel position encoding as input. That is, the model uses an implicit network containing a periodic activation function and a position encoding to replace the common decoder network with convolution as the main architecture, and uses a feature encoder network structure based on a multi-level window attention mechanism to obtain feature encoding.

[0027] Wherein, the feature encoding branch is constructed based on a mixed-scale window-based self-attention module (M-WSAB), which uses a multi-scale method and an attention mechanism within the window to make the image feature map contain image details and local and global descriptions of different levels and scales, and at the same time make the model adapt to different defect types of different sizes from pixel-level burrs to overall color deviation.

[0028] Referring back to Figure 2 , in some embodiments, the feature encoding branch includes a U-shaped network (U-Net) constructed by a mixed-scale window-based self-attention module (M-WSAB), which first reduces the image resolution and expands the channel number through several convolution modules (not shown in the figure) to obtain an initial feature map, and then the M-WSAB of the U-shaped network takes the initial feature map as input, and in the first half of the descending branch of the U-shaped network, it is sampled by convolution to half of the input after every k M-WSAB, while expanding the channel number to twice; at the same time, the U-shaped network retains low-level features and fuses multi-level features through a skip connection, as shown in Figure 2 , the skip connection is the splicing of the output of each down-sampling stage of the first half of the descending branch of the U-shaped network and the feature map of the corresponding size of the second half of the ascending branch; wherein, 1≤k≤3.

[0029] In some embodiments, each M-WSAB is composed of several window-based self-attention modules (WSAB) with different window sizes in parallel, and is spliced into a feature map output in a fixed order when output. As shown in Figure 3As shown, a multi-scale hybrid window self-attention module formed by WSABs with window sizes of 4x4, 8x8 and 16x16 in parallel is illustrated, and each branch with different window size contains two stacked window self-attention modules WSAB. It should be understood that, Figure 3 The number of WSABs in parallel and the window scale in each M-WSAB are not limited to Figure 3 As shown, other numbers and other window scales of WSABs can also be used.

[0030] The structure of the window self-attention module WSAB can refer to Figure 4 After the feature vector input WSAB, it sequentially passes through the input regularization layer Layer Norm1, the multi-head attention layer W-MSA, the output regularization layer Layer Norm2 and the output mapping layer FFN (feedforward neural network), and the skip connection is added between the input of Layer Norm1 and the output of W-MSA, and between the input of Layer Norm2 and the output of FFN. The structure of FFN is as shown in Figure 5 .

[0031] In the input stage of WSAB, the initial feature map is divided into several blocks according to the window scale, and each block of features is flattened into a feature vector with a size of [BxB, C] after position encoding and input into WSAB; B represents the window size, which can take a value of 4≤B≤128; C is the number of feature channels, which can take a value of 8≤C≤128. In the output stage of WSAB, the feature vector is converted back to an image block after deformation operation and spliced into a feature map.

[0032] The multi-head attention layer can be represented as follows:

[0033]

[0034]

[0035] For head h, Q h ,K h ,V h represent the outputs of the three branches of Q:query, K:key and V:value in the attention mechanism, respectively, and W h Q ,W h K ,W h V are the corresponding network weights; D represents the window size, C represents the number of channels, and t is the number of divided blocks. In the formula, matrix multiplication is represented by . represents the tth image block input to the attention head h; represents the corresponding output; K hT K h Y t represents the output of the t-th image block spliced by all heads, and Y represents the overall output of the multi-head attention layer.

[0036] The output mapping layer is a three-layer convolutional neural network, and the feature map output by the multi-head self-attention layer is spliced by feature map blocks. Here, a three-layer convolutional neural network is used for feature compression and local information smoothing to reduce the number of feature channels and avoid feature differences on the boundary caused by splicing.

[0037] The M-WSAB fuses the feature maps from each window size branch in a splicing manner. To further fuse features from different sizes, a 1x1 convolution is performed after splicing for feature compression and fusion.

[0038] In the embodiment of the present application, the image restoration branch is a deep implicit network, which takes pixel coordinates, feature vectors and defect vectors corresponding to coordinate positions as input to restore the image pixel by pixel. When the specified output resolution is HxW, the imaging plane with a size of [-1, 1; -1, 1] is also uniformly divided into HxW grids, and the feature vectors and defect vectors of the corresponding points on the image feature map and defect encoding map are obtained by bilinear interpolation of the coordinates of the grid center points. In some specific embodiments, the deep implicit network is a multilayer perceptron, in which the output of each layer except the output layer is activated by a sin function, and the output layer adopts a ReLU function activation.

[0039] Second step, model training: taking an image with a resolution determined by a specific task as input, obtaining an image feature map, a defect encoding map, and an image with the same resolution or high resolution restored by a deep implicit network; the model takes image restoration loss as the main supervision signal; for defect-free images, the obtained defect encoding map is constrained to be all 0 output; for images with defects, first minimize the similarity between the defect encoding of the defect occurrence point and the defect encoding of any point on the image without defects, and second, based on the triplet loss, maximize the inter-class distance of the defect encoding of the defect occurrence point and minimize the intra-class encoding distance. The embodiment of the present application further constrains the distribution of different category feature encodings in addition to the image restoration loss, so that the encoding distance between defect categories is greater than the feature encoding distance within the defect category.

[0040] In the embodiments of the present application, two actual requirements are focused on when designing the network model and its training process: first, the model can decouple the feature map of the image and the defect map. The feature map encodes the shape, color and other information of the target workpiece under the condition of no defects, as well as a small amount of disturbance caused by environmental changes and workpiece position changes. The defect map only contains the difference in feature dimensions caused by defects. When the defect map is all 0, the image of the workpiece without defects should be obtained. Second, as little labeled data as possible is used to complete the training.

[0041] Therefore, the training process includes image restoration loss and defect contrast loss. Specifically, in each iteration process, N images are randomly sampled as input, and the mean square error between the true value image and the restored image is calculated as the image restoration loss.

[0042] In addition, the defect segmentation true value is used as an index to minimize the variance of each category defect vector and maximize the difference between the means of different category defect vectors. If the defect encoding graph is represented as D i , the defect segmentation true value is G i , and the total number of defect categories is K, then the defect vector set X k corresponding to the kth category of defects is:

[0043] X k = {D i (u, v) | G i (u, v) = k, 0 ≤ u ≤ H, 0 ≤ v ≤ W, 1 ≤ i ≤ N}

[0044] When E k represents the mean and D k represents the variance, the hyperparameter is γ, u and v represent the coordinates, and i represents the serial number of the graph, the loss is defined as:

[0045]

[0046] In one embodiment, E k and D k are both high-dimensional vectors; j represents the serial number of the defect category, and E j represents the mean.

[0047] In another embodiment, D k is the cosine similarity between each high-dimensional vector within the category and the average vector E k , and at this time corresponds to where α is a hyperparameter.

[0048] Finally, since the corresponding application scenario in this example has fixed workpiece position and fixed light source, the variance of the image features corresponding to the N images is used to constrain the difference between the image features corresponding to different images.

[0049] Third step, reasoning stage: reference Figure 6 , first with defect encoding graph as output traversal training data set, calculate each category code center; subsequently, in the reasoning stage, for any input image, according to the distance of its defect encoding graph and each category code center judge corresponding image area category. Wherein, distance (or similarity) calculation mode and the loss loss2 of second step training stage keep consistent.

[0050] The method and model of the embodiment of the application can complete the defect detection and classification of parts in actual engineering scenes in cooperation with cameras and lenses.

[0051] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent alternatives or obvious variations can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A defect detection method based on deep implicit network, characterized in that, The method comprises the following steps: S1, model construction: constructing a defect detection model, which comprises a feature encoding branch and an image restoration branch based on a deep implicit network; wherein the feature encoding branch is used to extract features of different levels and different scales in an original image of a to-be-detected object to obtain an image feature map and a defect encoding map; the image restoration branch is used to decode and obtain RGB values of an arbitrary real point coordinate in an imaging plane by taking the image feature map, the defect encoding map and a pixel position code as inputs; The feature encoding branch is constructed based on a multi-scale mixed window self-attention module, which uses a multi-scale method and an attention mechanism within the window to make the image feature map contain image details and local and global descriptions of different levels and scales, and make the model adapt to different defect types of different sizes from pixel-level burrs to overall color deviation; The feature encoding branch comprises a U-shaped network constructed by the multi-scale mixed window self-attention module, which takes an initial feature map obtained by reducing the image resolution and expanding the channel number as input, reduces the input by half in a convolution manner at every k multi-scale mixed window self-attention module in the first half of the descending branch of the U-shaped network, and expands the channel number by two times; meanwhile, the U-shaped network retains low-level features and fuses multi-level features by using a skip connection, which is the output of each down-sampling stage of the first half of the descending branch of the U-shaped network and the feature map of the corresponding size of the second half of the ascending branch spliced; S2, training phase: taking an image with a non-fixed resolution determined by a specific task as input, obtaining an image feature map, a defect encoding map and an image with the same resolution or a high resolution restored by a deep implicit network; the model takes an image restoration loss as the main supervision signal; for a defect-free image, the obtained defect encoding map is constrained to be all 0 output; for an image with defects, firstly, the similarity between the defect encoding of the defect occurrence point and the defect encoding of any point on the image under the condition of no defects is minimized, and secondly, based on a three-tuple loss, the inter-class distance of the defect encoding of the defect occurrence point is maximized, and the intra-class encoding distance is minimized; S3, inference phase: firstly, traverse the training data set by taking the defect encoding map as output to calculate the encoding centers of each category; then, for any input image, the corresponding image region category is determined according to the distance between the defect encoding map and the encoding centers of each category. 2.The deep implicit network based defect detection method of claim 1, wherein: 1≤k≤3。 3.The deep implicit network based defect detection method of claim 2, wherein: Each multi-scale mixed window self-attention module is composed of a plurality of window self-attention modules with different window sizes connected in parallel, and the output is spliced into a feature map output in a fixed order. 4.The deep implicit network based defect detection method of claim 1, wherein: The image restoration branch is a deep implicit network, which takes pixel coordinates, feature vectors and defect vectors corresponding to the coordinates as inputs to restore the image pixel by pixel. 5.The deep implicit network based defect detection method of claim 4, wherein: When the specified output resolution is HxW, the imaging plane with a size of [-1, 1; -1, 1] is also uniformly divided into HxW grids, and the feature vectors and defect vectors of the corresponding points on the image feature map and the defect encoding map are obtained by bilinear interpolation of the grid center coordinates. 6.The deep implicit network based defect detection method of claim 1, wherein: The deep implicit network is a multi-layer perceptron, wherein the output of each layer except the output layer is activated by a sin function, and the output layer is activated by a ReLU function.

7. A defect detection model based on deep implicit network, characterized in that: The feature encoding branch is configured to extract features of different levels and different scales in the original image of the object to be detected to obtain an image feature map and a defect encoding map. The feature encoding branch includes a U-shaped network constructed by a multi-scale hybrid window self-attention module.

8. The deep implicit network based defect detection model of claim 7, wherein: 1≤k≤3。 9.The defect detection model based on deep implicit network of claim 7, wherein: The image restoration branch is a deep implicit network, which takes pixel coordinates, feature vectors corresponding to coordinate positions, and defect vectors as inputs to restore an image pixel by pixel. When the specified output resolution is HxW, the imaging plane with a size of [-1, 1; -1, 1] is also uniformly divided into HxW grids, and the feature vectors and the defect vectors of the corresponding points on the image feature map and the defect encoding map are obtained by bilinear interpolation of the coordinate of the center point of the grid.

Images