Defect detection method and system based on semantic alignment

Abstract

The present invention relates to the field of image processing technology, and discloses a defect detection method and system based on semantic alignment, which aims to solve the problem of poor accuracy in existing defect detection methods. The scheme mainly includes: visually collecting sample product pictures at different angles and positions, and constructing a multi-angle sample data set based on the collected sample product pictures; constructing a semantically aligned deep learning network, and training to obtain a semantic alignment network; then selecting the first sample data corresponding to the normal sample product from the multi-angle sample data set, and extracting a normal first semantic feature; obtaining the picture to be tested of the sample product to be tested, and based on the semantic alignment network, obtaining the second sample data corresponding to the sample product to be tested and its corresponding second semantic feature, the second sample data approximating the first sample data; finally judging whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature. The present invention improves the accuracy of defect detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a defect detection method and system based on semantic alignment. Background Art

[0002] In the industrial field, most of the products produced are normal and fault-free, but unfortunately, products manufactured in certain scenarios also contain some defects, such as dents, wrong labels or parts duplication, etc. Therefore, it is very important to preserve the yield and consistency of product quality.

[0003] Traditional methods are all done manually, which often requires a lot of manpower, and different people's test results are inconsistent, and there are problems such as missed detection and false detection. Therefore, it is imperative to use computer vision solutions to replace manual detection.

[0004] At present, most visual solutions are based on the idea of ​​image comparison, which compares normal product pictures with unknown product pictures. If there is a difference, it means that the unknown product is abnormal. However, this often requires the normal picture to have highly obvious features. At the same time, the unknown product picture and the normal product picture are taken at different angles and positions, resulting in inaccurate comparison results. Summary of the invention

[0005] The present invention aims to solve the problem of poor accuracy in existing image-based defect detection methods and proposes a defect detection method and system based on semantic alignment.

[0006] The technical solution adopted by the present invention to solve the above technical problems is:

[0007] On the one hand, a defect detection method based on semantic alignment is provided, comprising the following steps:

[0008] Step 1: Visually collect sample product images at different angles and positions, and construct a multi-angle sample data set based on the collected sample product images;

[0009] Step 2: construct a semantically aligned deep learning network, and train the semantically aligned network based on the multi-angle sample data set;

[0010] Step 3: Selecting first sample data corresponding to a normal sample product from the multi-angle sample data set, and extracting a normal first semantic feature based on the first sample data;

[0011] Step 4: Obtain a picture of the sample product to be tested, and obtain second sample data corresponding to the sample product to be tested and its corresponding second semantic features according to the picture to be tested and the first sample data and based on the semantic alignment network, wherein the second sample data is close to the first sample data;

[0012] Step 5: Determine whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature.

[0013] As a further optimization, the sample product images at different angles include at least: sample product images after flipping and cropping, and the sample product images at different positions include at least: sample product images collected after translating and rotating the sample products.

[0014] As a further optimization, judging whether the sample product to be tested has defects according to the first semantic feature and the second semantic feature specifically includes:

[0015] For each pixel position, the first semantic feature and the second semantic feature corresponding thereto are differentiated, and whether the sample product to be inspected has defects is determined based on the differential result, wherein the differential includes: pixel differential or root mean square pixel differential.

[0016] As a further optimization, judging whether the sample product to be tested has defects according to the difference result specifically includes:

[0017] If the differential results corresponding to all pixel positions are less than the preset threshold, it is determined that the sample product to be tested has no defects. If there is a pixel position corresponding to a differential result greater than the preset threshold, it is determined that the pixel position corresponding to the sample product to be tested has defects.

[0018] As a further optimization, the difference formula is as follows:

[0019] diff(i,j)=|feature(i,j)-normal(i,j)|;

[0020] In the formula, diff represents the difference result, feature represents the second semantic feature, normal represents the first semantic feature, and (i, j) represents the coordinate position corresponding to the pixel position.

[0021] As a further optimization, the method further includes:

[0022] Step 6: Upsample the difference result to obtain an enlarged image corresponding to the image to be tested, so as to locate the defect area.

[0023] As a further optimization, the upsampling method is: adjacent value interpolation upsampling or quadratic linear interpolation upsampling.

[0024] As a further optimization, the upsampling formula is as follows:

[0025]

[0026] Where f() represents the upsampling function, Q 11 , Q 12 , Q 21 and Q 22 represents the four preset position coordinates in the image to be tested, (x, y) represents the position coordinates of any pixel position, where Q 11 =(x1,y1),Q 12 =(x1,y2),Q 21 =(x2,y1),Q 22 =(x2,y2).

[0027] As a further optimization, the deep learning network is: VGG network, ResNet network or MobileNet network.

[0028] In a second aspect, a defect detection system based on semantic alignment is provided, characterized in that it includes:

[0029] A data collection training module is used to visually collect sample product images at different angles and positions, and construct a multi-angle sample data set based on the collected sample product images; and select first sample data corresponding to a normal sample product from the multi-angle sample data set, and extract a normal first semantic feature based on the first sample data;

[0030] A semantic alignment module is used to construct a semantically aligned deep learning network, and obtain a semantically aligned network based on the multi-angle sample data set training; and obtain a test picture of the sample product to be tested, and obtain second sample data corresponding to the sample product to be tested and its corresponding second semantic features based on the test picture and the first sample data and the semantic alignment network, wherein the second sample data approximates the first sample data;

[0031] The differential comparison module is used to determine whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature.

[0032] The beneficial effect of the present invention is that the semantic alignment-based defect detection method and system of the present invention, after obtaining an unknown product image, aligns it with a normal product image based on a pre-trained semantic alignment network, thereby improving the accuracy of defect detection. In addition, the present invention also locates the defect area through upsampling, which facilitates subsequent product quality optimization tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a flow chart of a defect detection method based on semantic alignment according to an embodiment of the present invention;

[0034] Figure 2Another schematic diagram of a flow chart of a defect detection method based on semantic alignment according to an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of the structure of a defect detection system based on semantic alignment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] The defect detection method and system based on semantic alignment provided by the present invention have the following main technical solutions: first, visually collect sample product images at different angles and positions, and construct a multi-angle sample data set based on the collected sample product images; then, construct a semantically aligned deep learning network, and obtain a semantic alignment network based on the multi-angle sample data set training; then, select the first sample data corresponding to the normal sample product from the multi-angle sample data set, and extract the normal first semantic feature based on the first sample data; when it is necessary to perform defect detection on the sample product to be inspected, first obtain the image to be inspected of the sample product to be inspected, and obtain the second sample data corresponding to the sample product to be inspected and its corresponding second semantic feature based on the image to be inspected and the first sample data and the semantic alignment network, and the second sample data is close to the first sample data; finally, judge whether the sample product to be inspected has defects based on the first semantic feature and the second semantic feature.

[0038] When defect detection is performed on products based on images, it is mainly performed by comparing the pixels of the sample products to be detected with the pictures of the normal sample products. If the feature values ​​corresponding to the pixels are inconsistent, it is determined that the sample products to be detected have defects. In the actual application process of this method, it is necessary to ensure that the shooting angle and position of the sample product pictures to be detected are highly consistent with those of the normal sample product pictures. If they are inconsistent, it is easy to have inconsistent feature values ​​corresponding to the pixels, which leads to misjudgment. Based on this, the present invention constructs a multi-angle sample data set by shooting pictures of various angles and positions of the sample products, and trains a semantic alignment network that can realize the semantic alignment function according to the multi-angle sample data set. The pictures of the sample products to be detected and the pictures of the normal sample products are input into the semantic alignment network, and the pictures corresponding to the sample products to be detected that are close to the normal sample pictures can be determined. Close to the normal sample pictures means that the angle and position are close to the normal sample pictures, and then according to the second semantic feature corresponding to the picture and the first semantic feature corresponding to the normal sample picture, it can be judged whether the sample pictures to be detected have defects. When obtaining the sample pictures to be detected, the present invention does not need to consider the shooting angle and position. By semantically aligning the sample pictures to be detected and then comparing them with the normal sample pictures, the accuracy of defect detection is improved.

[0039] Example

[0040] The defect detection method based on semantic alignment described in the embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0041] Step 1: Visually collect sample product images at different angles and positions, and construct a multi-angle sample data set based on the collected sample product images;

[0042] In this embodiment, the sample product pictures may be normal sample product pictures or defective sample product pictures. The sample product pictures at different angles include at least: sample product pictures after flipping and cropping, and the sample product pictures at different positions include at least: sample product pictures collected after translating and rotating the sample product. In actual application, the camera and light source can be fixed, the sample product can be rotated and translated manually, and then pictures at different angles and positions can be collected. Since manual rotation and translation are only possible, image processing operations such as cropping and flipping are also used to enhance the sample data to improve the effect of semantic alignment.

[0043] Step 2: construct a semantically aligned deep learning network, and train the semantically aligned network based on the multi-angle sample data set;

[0044] In this embodiment, the deep learning network can be a VGG network, a ResNet network or a MobileNet network. In specific applications, such as Figure 2 As shown, the fully connected layers are first concatenated to construct the classification task, and pre-trained on a multi-angle sample data set to improve the network feature extraction capability. The fully connected layers are then removed, and the output of the last convolutional layer is taken as the semantic feature layer to obtain a semantic alignment network. The semantic alignment network in this embodiment is used for semantic alignment and semantic feature extraction.

[0045] Step 3: Selecting first sample data corresponding to a normal sample product from the multi-angle sample data set, and extracting a normal first semantic feature based on the first sample data;

[0046] It can be understood that the first sample data corresponding to the normal sample product only includes data corresponding to the normal sample product image at one position and angle, so as to facilitate alignment of the sample product image to be detected with the normal sample product image through the semantic alignment network.

[0047] Step 4: Obtain a picture of the sample product to be tested, and obtain second sample data corresponding to the sample product to be tested and its corresponding second semantic features according to the picture to be tested and the first sample data and based on the semantic alignment network, wherein the second sample data is close to the first sample data;

[0048] Specifically, when it is necessary to perform defect detection on a sample product to be inspected, first obtain a picture of the sample product to be inspected. The picture to be inspected here can be a picture of the sample product to be inspected at any angle or position. Input the data corresponding to the picture to be inspected and the first sample data into the semantic alignment network to make the picture to be inspected approximate the normal sample picture, realize the network semantic alignment function, and then obtain the second sample data corresponding to the sample product to be inspected that is approximate to the first sample data, and extract the second semantic feature corresponding to the second sample data through the semantic alignment network.

[0049] Step 5: Determine whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature.

[0050] Judging whether the sample product to be tested has defects according to the first semantic feature and the second semantic feature specifically includes:

[0051] For each pixel position, the first semantic feature and the second semantic feature corresponding thereto are differentiated, and whether the sample product to be inspected has defects is determined based on the differential result, wherein the differential includes: pixel differential or root mean square pixel differential.

[0052] In this embodiment, the difference formula is as follows:

[0053] diff(i,j)=|feature(i,j)-normal(i,j)|;

[0054] In the formula, diff represents the difference result, feature represents the second semantic feature, normal represents the first semantic feature, and (i, j) represents the coordinate position corresponding to the pixel position.

[0055] After obtaining the differential result, if the differential results corresponding to all pixel positions are less than the preset threshold, it is determined that the sample product to be tested has no defects. If there is a pixel position corresponding to a differential result greater than the preset threshold, it is determined that the pixel position corresponding to the sample product to be tested has defects.

[0056] In order to locate the defect area so that the user can view the defect location, this embodiment also includes:

[0057] Step 6: Upsample the difference result to obtain an enlarged image corresponding to the image to be tested, so as to locate the defect area. The upsampling method may be: adjacent value interpolation upsampling or quadratic linear interpolation upsampling.

[0058] In this embodiment, the upsampling formula is as follows:

[0059]

[0060] Where f() represents the upsampling function, Q11 , Q 12 , Q 21 and Q 22 represents the four preset position coordinates in the image to be tested, (x, y) represents the position coordinates after upsampling, where Q 11 =(x1,y1),Q 12 =(x1,y2),Q 21 =(x2,y1),Q 22 =(x2,y2).

[0061] It can be understood that upsampling is performed on the differential result to enlarge the image and display the defect position in the enlarged image, thereby locating the defect area.

[0062] Based on the above technical solution, this embodiment also proposes a defect detection system based on semantic alignment, such as Figure 3 As shown, including:

[0063] A data collection training module is used to visually collect sample product images at different angles and positions, and construct a multi-angle sample data set based on the collected sample product images; and select first sample data corresponding to a normal sample product from the multi-angle sample data set, and extract a normal first semantic feature based on the first sample data;

[0064] A semantic alignment module is used to construct a semantically aligned deep learning network, and obtain a semantically aligned network based on the multi-angle sample data set training; and obtain a test picture of the sample product to be tested, and obtain second sample data corresponding to the sample product to be tested and its corresponding second semantic features based on the test picture and the first sample data and the semantic alignment network, wherein the second sample data approximates the first sample data;

[0065] The differential comparison module is used to determine whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature.

[0066] It can be understood that since the semantic alignment-based defect detection system described in the embodiment of the present invention is a system for implementing the semantic alignment-based defect detection method described in the embodiment, for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method.

Claims

1. A defect detection method based on semantic alignment, characterized in that: The following steps are involved: Step 1: Visually collect sample product images at different angles and positions, and construct a multi-angle sample data set based on the collected sample product images; Step 2: construct a semantically aligned deep learning network, and train the semantically aligned network based on the multi-angle sample data set, where the semantically aligned network is used for semantic alignment and semantic feature extraction; Step 3: Selecting first sample data corresponding to a normal sample product from the multi-angle sample data set, and extracting a normal first semantic feature based on the first sample data; Step 4: Obtain a picture of the sample product to be tested, and obtain second sample data corresponding to the sample product to be tested and its corresponding second semantic features according to the picture to be tested and the first sample data and based on the semantic alignment network, wherein the second sample data is close to the first sample data; According to the image to be tested and the first sample data and based on the semantic alignment network, second sample data corresponding to the sample product to be tested and its corresponding second semantic feature are obtained, including: Inputting the data corresponding to the image to be tested and the first sample data into the semantic alignment network, obtaining the second sample data corresponding to the sample product to be tested and approximating the first sample data, and extracting the second semantic features corresponding to the second sample data through the semantic alignment network; Step 5: Determine whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature.

2. The defect detection method based on semantic alignment according to claim 1, characterized in that: The sample product pictures at different angles include at least: sample product pictures after flipping and cropping, and the sample product pictures at different positions include at least: sample product pictures collected after translating and rotating the sample products.

3. The defect detection method based on semantic alignment according to claim 1, characterized in that: Judging whether the sample product to be tested has defects according to the first semantic feature and the second semantic feature specifically includes: For each pixel position, the first semantic feature and the second semantic feature corresponding thereto are differentiated, and whether the sample product to be inspected has defects is determined based on the differential result, wherein the differential includes: pixel differential or root mean square pixel differential.

4. The defect detection method based on semantic alignment according to claim 3, characterized in that: The step of judging whether the sample product to be tested has defects according to the difference result specifically includes: If the differential results corresponding to all pixel positions are less than the preset threshold, it is determined that the sample product to be tested has no defects. If there is a pixel position corresponding to a differential result greater than the preset threshold, it is determined that the pixel position corresponding to the sample product to be tested has defects.

5. The defect detection method based on semantic alignment according to claim 3, characterized in that: The difference formula is as follows: diff(i,j)=|feature(i,j)-normal(i,j); In the formula, diff represents the difference result, feature represents the second semantic feature, normal represents the first semantic feature, and (i, j) represents the coordinate position corresponding to the pixel position.

6. The defect detection method based on semantic alignment according to claim 1, characterized in that: The method further includes: Step 6: Upsample the difference result to obtain an enlarged image corresponding to the image to be tested, so as to locate the defect area.

7. The defect detection method based on semantic alignment according to claim 6, characterized in that: The up-sampling method is: adjacent value interpolation up-sampling or quadratic linear interpolation up-sampling.

8. The semantic alignment-based defect detection method according to claim 7, characterized in that: The upsampling formula is as follows: Where f() represents the upsampling function, Q 11 , Q 12 , Q 21 and Q 22 represents the four preset position coordinates in the image to be tested, (x, y) represents the position coordinates of any pixel position, where Q 11 =(x1,y1),Q 12 =(x1,y2),Q 21 =(x2,y1),Q 22 =(x2,y2).

9. The defect detection method based on semantic alignment according to claim 1, characterized in that: The deep learning network is: VGG network, ResNet network or MobileNet network.

10. A defect detection system based on semantic alignment, characterized in that: include: The data collection training module is used to visually collect sample product images at different angles and positions, and to construct a multi-angle sample data set based on the collected sample product images; and selecting first sample data corresponding to a normal sample product from the multi-angle sample data set, and extracting a normal first semantic feature based on the first sample data; A semantic alignment module, used to construct a semantic alignment deep learning network, and to train a semantic alignment network based on the multi-angle sample data set, wherein the semantic alignment network is used to perform semantic alignment and semantic feature extraction; and obtaining a picture of the sample product to be tested, and obtaining second sample data corresponding to the sample product to be tested and its corresponding second semantic features according to the picture and the first sample data and based on the semantic alignment network, wherein the second sample data approximates the first sample data; Obtaining second sample data corresponding to the sample product to be tested and its corresponding second semantic feature according to the image to be tested and the first sample data and based on the semantic alignment network, including: Inputting the data corresponding to the image to be tested and the first sample data into the semantic alignment network, obtaining the second sample data corresponding to the sample product to be tested and approximating the first sample data, and extracting the second semantic features corresponding to the second sample data through the semantic alignment network; The differential comparison module is used to determine whether the sample product to be tested has defects based on the first semantic feature and the second semantic feature.

Images