A defect detection method and device based on difference enhanced feature reconstruction
Through the differentially enhanced feature reconstruction method, a parallel feature compression reconstruction network is built using multi-level features and K nearest neighbor search, which solves the problem of poor defect detection in the existing technology, and realizes accurate and efficient detection of multiple defects.
Patent Information
- Application Number
- CN202411332147.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-09-24
AI Technical Summary
The existing machine vision defect detection algorithm is poorly robust and difficult to apply to multiple defect types, and cannot effectively detect defects of different sizes and types.
Using a differential enhancement feature reconstruction method, a normal feature memory pool is constructed by obtaining multi-level features, a K-nearest neighbor search is used to obtain differential features, and a multi-layer parallel feature compression reconstruction network is built to realize the detection of defective parts.
This method can accurately detect defects of different sizes and types, improves robustness, enhances the ability to discriminate defects, and improves the quality of feature reconstruction compared with traditional single-level autoencoder networks.
Smart Images
Figure CN118865047B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of industrial defect detection, and in particular relates to a defect detection method and device based on difference enhanced feature reconstruction. Background Art
[0002] Defect detection is a key link in industrial production and is related to product quality control and production efficiency. Traditional manual visual defect inspection methods use tools such as magnifying glasses to directly observe the product surface to identify defects such as cracks, scratches, and deformations. They rely on the experience and skills of the operator and are easily affected by subjective factors. They have problems such as high cost, low efficiency, and poor consistency, and cannot adapt to large-scale production. In order to solve the defects of manual visual defect inspection methods, machine vision-assisted defect detection methods have emerged. However, existing machine vision detection algorithms usually conduct defect analysis through edge detection, template matching, histogram analysis and other technologies. These methods have problems such as poor robustness and difficulty in applying to multiple defect types. Summary of the invention
[0003] The purpose of the present invention is to propose a defect detection method and device based on difference enhanced feature reconstruction, which can accurately detect defects of different sizes and types and has strong robustness.
[0004] The present invention is achieved through the following technical solutions:
[0005] A defect detection method based on difference enhanced feature reconstruction comprises the following steps:
[0006] Step S1, respectively obtaining multi-level features of normal samples and test samples, wherein the multi-level features have detailed texture information and contextual semantic information;
[0007] Step S2, construct a normal feature memory pool, store the multi-level features corresponding to the normal sample in the normal feature memory pool, use K-nearest neighbor retrieval to obtain the K-nearest neighbor index of the test sample, query the corresponding normal sample features in the memory pool according to the K-nearest neighbor index to obtain the reference feature, and then obtain the multi-level difference features according to the multi-level features of the test sample and the reference feature, and splice the multi-level difference features with the multi-level features of the test sample to obtain the multi-level splicing features;
[0008] Step S3, build a multi-layer parallel feature compression and reconstruction network formed by an encoder and a decoder. The multi-layer parallel feature reconstruction network includes a feature fusion layer to fuse multi-level splicing features, and construct reconstruction loss functions of each layer respectively. The total loss function is obtained according to the reconstruction loss functions of each layer to realize the detection of defective parts.
[0009] Furthermore, in step S1, according to the formula Get normal samples I i No. l Characteristics of the hierarchy ,in, I i Indicates i Normal samples are defect-free images of products. Represents the feature extractor l Layer output, l= 1,2,3, the feature extractor uses the WideResNet50 network pre-trained on the ImageNet dataset.
[0010] Furthermore, in step S1, a global average pooling layer is used to process the features of each level of the normal sample and the test sample respectively to obtain the normal sample features and the test sample features.
[0011] Furthermore, the step S2 specifically includes the following steps:
[0012] Step S21: Constructing a normal feature memory pool , store the normal sample features in the memory pool M l Among them, N is the normal sample size;
[0013] Step S22: According to the formula Get the test sample I q of k K nearest neighbor index KNN ( I q ,k ),in, , k Represents the test sample I q The number of neighbors of Represents the test sample after global average pooling I q The third level features, Represents a normal sample processed by global average pooling I i The third-level feature, argsort(.) means sorting and extracting indexes;
[0014] Step S23: Based on the K nearest neighbor index obtained in step S22, query the memory pool corresponding to each K nearest neighbor index. l Normal sample characteristics of the layer , and take the average of these normal sample features as the reference feature , according to the formula Get multi-level difference features, among which, γ represents the parameter that adjusts the difference characteristics, Z q,l Represents the test sample I q No. l Characteristics of the hierarchy;
[0015] Step S24: According to the formula The multi-level features of the test sample and the multi-level difference features are concatenated to obtain multi-level concatenated features as the input of the parallel feature compression and reconstruction network.
[0016] Furthermore, in step S3, the multi-layer parallel feature compression reconstruction network includes an encoder, a decoder and a feature fusion layer, and the feature fusion layer fuses multi-level splicing features in the compressed latent space output by the encoder.
[0017] Furthermore, in step S3, the encoder is composed of a cascaded feature compression module, the feature compression module is composed of a first convolutional layer, a first batch of normalization layers and a first nonlinear activation layer, the number of output channels of the first convolutional layer is half of its input channel number, and the decoder is composed of a cascaded feature reconstruction module, the feature reconstruction module is composed of a second convolutional layer, a second batch of normalization layers and a second nonlinear activation layer, and the number of output channels of the second convolutional layer is twice its input channel number.
[0018] Furthermore, in step S3, the feature fusion layer includes a forward feature fusion path and a reverse feature fusion path. The forward feature fusion path fuses the information of high-level features in the multi-level splicing features into the low-level features, and the reverse feature fusion path fuses the information of low-level features in the multi-level splicing features into the high-level features.
[0019] Further, in step S3, l The layer reconstruction loss function is expressed as , the total loss function is expressed as L=L 1 +L 2 +L 3 ,in, h l , w l Respectively represent l The height and width of the hierarchical features, Represents splicing features The center is located at coordinates ( m , n ) feature block, Represents reconstruction features The center is located at coordinates (m , n ) feature block, m ∈{1,2,…, h l}, n ∈{1,2,…, w l}.
[0020] Furthermore, the normal samples and the test samples are both from the MVTec dataset.
[0021] The present invention is also achieved through the following technical solutions:
[0022] A defect detection device based on difference enhancement feature reconstruction includes a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions in the memory to implement the following modules:
[0023] Feature acquisition module: obtains multi-level features of normal samples and test samples respectively. The multi-level features have low-level features with rich detailed texture information and high-level features with rich contextual semantic information;
[0024] Difference feature acquisition module: construct a normal feature memory pool, store the multi-level features corresponding to the normal sample in the normal feature memory pool, use K nearest neighbor retrieval to obtain the reference features of the test sample, and then obtain the multi-level difference features based on the multi-level features of the test sample and the reference features, and splice the multi-level difference features with the multi-level features of the test sample to obtain the spliced features;
[0025] Reconstruction network construction module: Build a multi-layer parallel feature compression reconstruction network formed by an encoder and a decoder. The multi-layer parallel feature reconstruction network contains a feature fusion layer to fuse multi-level splicing features, and construct each layer's reconstruction loss function separately. According to the reconstruction loss function of each layer, the total loss function is obtained to realize the detection of defective parts.
[0026] The present invention has the following beneficial effects:
[0027] 1. The present invention first obtains multi-level features of normal samples and test samples to obtain rich detail textures and rich contextual semantic information, then constructs a normal feature memory pool, and uses K-nearest neighbor retrieval to obtain the K-nearest neighbor index of the test sample, and then obtains multi-level difference features and splicing features. Finally, a parallel feature compression and reconstruction network formed by an encoder and a decoder is built, and the splicing feature is used as the input of the network. The total loss function obtained by reconstructing the loss function of each layer is used to realize the detection of defective parts. The multi-level features can achieve good characterization capabilities for defects of different sizes and types. The difference features are entered through the feature memory pool structure, and the anomalies are described from the perspective of the difference from the normal sample features, thereby enhancing the ability to distinguish defects. The parallel feature compression and reconstruction network, compared with the traditional single-layer autoencoder network, compresses and reconstructs features of different levels in parallel, and fuses multi-level features in the latent space, thereby improving the quality of single-level feature reconstruction and enhancing robustness, providing a new method for solving surface defect detection based on feature reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The present invention is further described in detail below with reference to the accompanying drawings.
[0029] Figure 1 Flow chart of the method of the present invention.
[0030] Figure 2 It is a principle block diagram of the device of the present invention. DETAILED DESCRIPTION
[0031] like Figure 1 and Figure 2 As shown, a defect detection method based on difference enhanced feature reconstruction includes the following steps:
[0032] Step S1, respectively obtaining multi-level features of normal samples and test samples, wherein the multi-level features have detailed texture information and contextual semantic information;
[0033] Specifically, according to the formula Get normal samples I i No. l Characteristics of the hierarchy , according to the formula Get test samples I q No. l The characteristics of the level, among which, I i Indicates i Normal samples are defect-free images of products. Represents the feature extractor lLayer output, l= 1, 2, 3, the feature extractor uses the WideResNet50 network pre-trained on the ImageNet dataset. In order to expand the receptive field of the feature and aggregate neighbor features, a global average pooling layer (AvgPool) with a window size of 3 and a step size of 1 is used to process the features of each level of the normal sample and the test sample respectively to obtain the normal sample features and the test sample features.
[0034] Among the multi-level features, low-level features (such as the first-level features) contain detailed texture information, and high-level features (such as the second and third-level features) contain contextual semantic information.
[0035] Both normal samples and test samples come from the MVTec dataset, which contains 5,354 images in 15 categories, covering various types of samples such as textures and objects. The dataset contains 3,629 normal samples without defects and 1,799 test samples with various defect types.
[0036] Step S2, construct a normal feature memory pool, store the multi-level features corresponding to the normal sample in the normal feature memory pool, use K-nearest neighbor retrieval to obtain the K-nearest neighbor index of the test sample, query the corresponding normal sample features in the memory pool according to the K-nearest neighbor index to obtain the reference feature, and then obtain the multi-level difference features according to the multi-level features of the test sample and the reference feature, and splice the multi-level difference features with the multi-level features of the test sample to obtain the multi-level splicing features;
[0037] The specific steps include:
[0038] Step S21: Constructing a normal feature memory pool , store the normal sample features in the memory pool M l Among them, N is the normal sample size;
[0039] Step S22: According to the formula Get the test sample I q of k K nearest neighbor index KNN ( I q ,k ),in, , k Represents the test sample I q The number of neighbors of Represents the test sample after global average pooling I q The third level features, Represents a normal sample processed by global average pooling I i The third level feature, argsort(.) means sorting and extracting the index, [1 :k ] means to take the first k The index of
[0040] For objects to be detected that are rotated, K-nearest neighbor retrieval uses high-level features that have been processed by average pooling. Since high-level features contain a description of the overall content of the image, the neighbor features obtained by K-nearest neighbor retrieval have similar rotation angles to the query features, which improves the quality of the reference features for calculating the difference features.
[0041] Step S23: Based on the K nearest neighbor index obtained in step S22, query the memory pool corresponding to each K nearest neighbor index. l Normal sample characteristics of the layer , and take the average of these normal sample features as the reference feature , according to the formula Get multi-level difference features, that is, the difference features between the test sample features and the normal sample features at each level, where: γ represents the parameter that adjusts the difference characteristics, Z q,l Represents the test sample I q No. l Characteristics of the hierarchy;
[0042] Step S24: According to the formula The multi-level features of the test sample and the multi-level difference features are concatenated to obtain multi-level concatenated features as the input of the parallel feature compression and reconstruction network.
[0043] Due to the local perception characteristics of the convolutional neural network, the receptive field of each feature block is limited, and it is processed through an average pooling layer with a window of 3 and a step size of 1 to expand its receptive field. At the same time, the average pooling layer realizes low-pass filtering of the input features, removes some high-frequency information, and is conducive to promoting the network's learning of normal sample features.
[0044] Step S3, building a multi-layer parallel feature compression reconstruction network formed by an encoder and a decoder, wherein the multi-layer parallel feature reconstruction network includes a feature fusion layer to fuse multi-level splicing features, respectively construct a reconstruction loss function of each layer, and obtain a total loss function according to the reconstruction loss function of each layer to realize the detection of defective parts;
[0045] Specifically, the multi-layer parallel feature compression and reconstruction network includes an encoder, a decoder and a feature fusion layer. The feature fusion layer fuses multi-level splicing features in the compressed latent space output by the encoder to improve the quality of feature reconstruction. In order to reduce the number of parameters of the parallel feature compression and reconstruction network and improve the inference speed, two cascaded 1×1 convolutions are used in the input layer of the encoder and the output layer of the decoder to compress and reconstruct the feature dimension, and other convolutional layers use 3×3 convolutions. Among them, the encoder is composed of a cascaded feature compression module, which is composed of a first convolutional layer, a first batch of normalization layers and a first nonlinear activation layer. The number of output channels of the first convolutional layer is half of its input channel number, thereby realizing feature compression. The decoder is composed of a cascaded feature reconstruction module, which is composed of a second convolutional layer, a second batch of normalization layers and a second nonlinear activation layer. The number of output channels of the second convolutional layer is twice its input channel number, thereby realizing feature reconstruction. The feature fusion layer includes a forward feature fusion path and a reverse feature fusion path. The forward feature fusion path fuses the information of high-level features in the multi-level splicing features into the low-level features, and the reverse feature fusion path fuses the information of low-level features in the multi-level splicing features into the high-level features. The forward feature fusion path fuses high-level information into low-level features. Since the dimensions of features at different levels are different, bilinear interpolation is first used to interpolate the horizontal and vertical dimensions of the features, and then a 1×1 convolutional layer is used to transform the channel dimension of the features, and finally the features are fused by addition; for the reverse feature fusion path, the low-level information is fused into the high-level feature layer, and the same operation as the forward feature fusion path is used for feature fusion.
[0046] No. l The layer reconstruction loss function is expressed as , adding the reconstruction losses of the three layers in parallel to obtain the total loss function expressed as L=L 1 +L 2 +L 3 ,in, h l , w l Respectively represent l The height and width of the hierarchical features, Represents splicing features The center is located at coordinates ( m , n ) feature block, Represents reconstruction features The center is located at coordinates ( m , n ) feature block, m ∈{1,2,…,h l}, n ∈{1,2,…, w l}.
[0047] Only normal samples are used for training. The cosine similarity between the reconstructed features of normal samples and the input features is high, that is, the model can reconstruct the features of normal samples, while the cosine similarity between the reconstructed features of abnormal samples and the input features is low, that is, the reconstruction error of abnormal samples is large, so defects can be detected based on the error of feature reconstruction.
[0048] Correspondingly, the defect detection device based on difference enhancement feature reconstruction includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules:
[0049] Feature acquisition module: obtains multi-level features of normal samples and test samples respectively. The multi-level features have low-level features with rich detailed texture information and high-level features with rich contextual semantic information;
[0050] Difference feature acquisition module: construct a normal feature memory pool, store the multi-level features corresponding to the normal sample in the normal feature memory pool, use K nearest neighbor retrieval to obtain the reference features of the test sample, and then obtain the multi-level difference features based on the multi-level features of the test sample and the reference features, and splice the multi-level difference features with the multi-level features of the test sample to obtain the spliced features;
[0051] Reconstruction network construction module: Build a multi-layer parallel feature compression reconstruction network formed by an encoder and a decoder. The multi-layer parallel feature reconstruction network contains a feature fusion layer to fuse multi-level splicing features, and construct each layer's reconstruction loss function separately. According to the reconstruction loss function of each layer, the total loss function is obtained to realize the detection of defective parts.
[0052] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.
Claims
1. A defect detection method based on difference enhanced feature reconstruction, characterized in that: The steps include: Step S1, respectively obtaining multi-level features of normal samples and test samples, wherein the multi-level features have detailed texture information and contextual semantic information; Step S2, constructing a normal feature memory pool, storing the multi-level features corresponding to the normal sample in the normal feature memory pool, using K-nearest neighbor retrieval to obtain the K-nearest neighbor index of the test sample, querying the corresponding normal sample features in the normal feature memory pool according to the K-nearest neighbor index to obtain a reference feature, and then obtaining a multi-level difference feature according to the multi-level features of the test sample and the reference feature, and splicing the multi-level difference feature with the multi-level features of the test sample to obtain a multi-level splicing feature; Step S3, building a multi-layer parallel feature compression and reconstruction network, the multi-layer parallel feature reconstruction network includes an encoder, a decoder and a feature fusion layer, the encoder is composed of cascaded feature compression modules, the decoder is composed of cascaded feature reconstruction modules, the feature fusion layer is used to fuse multi-level splicing features, and based on the multi-layer parallel feature compression and reconstruction network, each layer reconstruction loss function is constructed respectively, and a total loss function is obtained according to the reconstruction loss function of each layer to realize the detection of defective parts; The step S2 specifically includes the following steps: Step S21: Constructing a normal feature memory pool , store the normal sample features in the normal feature memory pool M l Among them, N is the normal sample size; For normal samples I i No. l The characteristics of the level, l= 1,2,3; Step S22: According to the formula Get the test sample I q of k K nearest neighbor index KNN ( I q ,k ),in, , k Represents the test sample I q The number of neighbors, indicating the test sample processed by global average pooling I q The third level features, Represents a normal sample processed by global average pooling I i The third-level feature, argsort(.) means sorting and extracting the index, [1: k ] means to take the first k The index of Step S23: Based on the K nearest neighbor index obtained in step S22, query the first l Normal sample characteristics of the layer , and take the average of these normal sample features as the reference feature , according to the formula Get multi-level difference features, among which, γ represents the parameter that adjusts the difference feature, Z q,l Represents the test sample I q No. l Characteristics of the hierarchy; Step S24: According to the formula The multi-level features of the test sample are spliced with the multi-level difference features to obtain the multi-level splicing features , as the input of the multi-layer parallel feature compression and reconstruction network.
2. The defect detection method based on difference enhanced feature reconstruction according to claim 1, characterized in that: In step S1, according to the formula Get normal samples I i No. l Characteristics of the hierarchy ,in, I i Indicates i Normal samples are defect-free images of products. Represents the feature extractor l Layer output, l= 1,2,3, the feature extractor uses the WideResNet50 network pre-trained on the ImageNet dataset.
3. The defect detection method based on difference enhanced feature reconstruction according to claim 2, characterized in that: In the step S1, a global average pooling layer is used to process the features of each level of the normal sample and the test sample respectively to obtain the normal sample features and the test sample features.
4. The defect detection method based on difference enhanced feature reconstruction according to claim 3 is characterized in that: In step S3: the multi-layer parallel feature compression reconstruction network includes an encoder, a decoder and a feature fusion layer, and the feature fusion layer fuses multi-level splicing features in the compressed latent space output by the encoder.
5. The defect detection method based on difference enhanced feature reconstruction according to claim 4, characterized in that: In step S3, the encoder is composed of a cascaded feature compression module, which is composed of a first convolutional layer, a first batch of normalization layers and a first nonlinear activation layer, and the number of output channels of the first convolutional layer is half of the number of input channels thereof; the decoder is composed of a cascaded feature reconstruction module, which is composed of a second convolutional layer, a second batch of normalization layers and a second nonlinear activation layer, and the number of output channels of the second convolutional layer is twice the number of input channels thereof.
6. The defect detection method based on difference enhanced feature reconstruction according to claim 5, characterized in that: In step S3, the feature fusion layer includes a forward feature fusion path and a reverse feature fusion path. The forward feature fusion path fuses the information of high-level features in the multi-level splicing features into the low-level features, and the reverse feature fusion path fuses the information of low-level features in the multi-level splicing features into the high-level features.
7. The defect detection method based on difference enhanced feature reconstruction according to claim 6, characterized in that: In step S3, l The layer reconstruction loss function is expressed as , the total loss function is expressed as L=L 1 +L 2 +L 3 ,in, h l , w l Respectively represent l The height and width of the hierarchical features, Represents splicing features The center is located at coordinates ( m , n ) feature block, Represents reconstruction features The center is located at coordinates ( m , n ) feature block, m ∈{1,2,…, h l }, n ∈{1,2,…, w l }.
8. A defect detection method based on difference enhanced feature reconstruction according to any one of claims 1 to 7, characterized in that: The normal samples and test samples are both from the MVTec dataset.
9. A defect detection device based on difference enhancement feature reconstruction, characterized in that: It includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules: Feature acquisition module: obtains multi-level features of normal samples and test samples respectively. The multi-level features have low-level features with rich detailed texture information and high-level features with rich contextual semantic information; Difference feature acquisition module: construct a normal feature memory pool, store the multi-level features corresponding to the normal sample in the normal feature memory pool, use K nearest neighbor retrieval to obtain the reference features of the test sample, and then obtain the multi-level difference features based on the multi-level features of the test sample and the reference features, and splice the multi-level difference features with the multi-level features of the test sample to obtain the spliced features; Reconstruction network construction module: build a multi-layer parallel feature compression reconstruction network, the multi-layer parallel feature reconstruction network includes an encoder, a decoder and a feature fusion layer, the encoder is composed of cascaded feature compression modules, the decoder is composed of cascaded feature reconstruction modules, the feature fusion layer is used to fuse multi-level splicing features, and based on the multi-layer parallel feature compression reconstruction network, the reconstruction loss function of each layer is constructed respectively, and the total loss function is obtained according to the reconstruction loss function of each layer to realize the detection of defective parts; The difference feature acquisition module is implemented by the following steps: Constructing a normal feature memory pool , store the normal sample features in the normal feature memory pool M l Among them, N is the normal sample size; For normal samples I i No. l The characteristics of the level, l= 1,2,3; According to the formula Get the test sample I q of k K nearest neighbor index KNN ( I q ,k ),in, , k Represents the test sample I q The number of neighbors of Represents the test sample after global average pooling I q The third level features, Represents a normal sample processed by global average pooling I i The third-level feature, argsort(.) means sorting and extracting the index, [1: k ] means to take the first k The index of Based on the K nearest neighbor index obtained in step S22, the first l Normal sample characteristics of the layer , and take the average of these normal sample features as the reference feature , according to the formula Get multi-level difference features, among which, γ represents the parameter that adjusts the difference feature, Z q,l Represents the test sample I q No. l Characteristics of the hierarchy; According to the formula The multi-level features of the test sample are spliced with the multi-level difference features to obtain the multi-level splicing features , as the input of the multi-layer parallel feature compression and reconstruction network.