Product appearance defect detection method and system

Abstract

The invention relates to the technical field of defect detection, and discloses a product appearance defect detection method and system, and the method specifically comprises the following steps: collecting images of the appearance of a product from different directions, and training a Yolov4 model and a CLGAN model; images of the appearance of a product are collected in the transverse direction and the longitudinal direction respectively, the transverse image and the longitudinal image are cut respectively and then input into the trained Yolov4 model and the trained CLGAN model, and a multi-channel image containing a defect rectangular marking frame and a binary image representing defects with white pixel points are output; and splicing the corresponding images to obtain a transverse detection result image and a longitudinal detection result image. According to the invention, the definition of product appearance image acquisition and the defect imaging effect are improved, and the product full-surface defect detection effect is improved by adopting a supervision and non-supervision combined mode.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and in particular to a method and system for detecting product appearance defects. Background Art

[0002] Defect detection is a technology widely used in manufacturing and engineering fields to detect defects and undesirable conditions in products or structures. Traditional manual defect detection methods mainly rely on artificial vision. The manual detection process has problems such as low efficiency and high misjudgment rate due to visual fatigue. There are also important problems such as the operation process cannot be recorded, cannot be repeated, and cannot be traced. In addition, manual defect detection requires a lot of human resources, which will place a huge burden on enterprises. With the continuous development and application of emerging technologies such as computer vision, deep learning, and artificial intelligence, defect detection technology has also shown new development trends. In recent years, deep learning has been applied well in many scenarios such as face recognition and autonomous driving. Some defect detection methods based on deep learning are also widely used in industrial scenarios. Based on deep learning There are usually two types of defect detection algorithms, one is an unsupervised defect detection algorithm and the other is a supervised defect detection algorithm. Both algorithms have their own advantages and disadvantages. The advantage of the supervised defect detection algorithm is that its effect is usually better than the unsupervised algorithm. The disadvantage is that the supervised algorithm requires a large amount of labeled data for training and cannot identify unlabeled defect types. However, in actual industrial inspection scenarios, the distribution of different types of surface defects is often unbalanced. Therefore, it is very difficult to have all types of defects in a labeled data set; the advantage of the unsupervised defect detection algorithm is that it does not require labeled data and can discover unknown defect types. The disadvantage is that the effect of defect identification is not as good as the supervised algorithm.

[0003] Product appearance defect detection primarily involves identifying scratches, punctures, pores, particles, and other defects. However, in actual testing, different areas of the product's appearance have different backgrounds and the number of defect samples varies significantly. Using only supervised or unsupervised algorithms for defect detection is difficult to achieve effective results. In actual production, the logo area has a lower probability of defects and a complex background, making it difficult to obtain sufficient labeled data, making it unsuitable for supervised algorithms. However, outside the logo area, defects are more numerous, making it easier to obtain sufficient labeled data and more suitable for supervised algorithms. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a product appearance defect detection method and system, which improves the clarity of product appearance image acquisition and the imaging effect of defects, and adopts a combination of supervised and unsupervised methods to improve the full-surface defect detection effect of products.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for detecting product appearance defects, comprising:

[0007] Step 1: Collect images of the product appearance from different directions, combine the images without the logo area into training set A1, and combine the images with the logo area into training set B1; perform manual defect annotation and channel separation on the images in training set A1, and form a data set with multiple single-channel grayscale images

[0008] Step 2: Based on the dataset Train the Yolov4 model; train the CLGAN model based on training set B1;

[0009] Step 3: Collect images of the product appearance from the horizontal and vertical directions respectively, and obtain the horizontal sub-image set S1 and the vertical sub-image set S2. Separate the sub-images that do not contain the logo area in S1 and S2, and form a data set with the obtained single-channel grayscale images. The sub-images containing the logo area in S1 and S2 form the data sets respectively.

[0010] The dataset Input the trained Yolov4 model and output a multi-channel image containing a defect rectangular annotation box; perform channel fusion on the multi-channel image output by the Yolov4 model to obtain a first result image containing a rectangular box annotating the defect;

[0011] The dataset The trained CLGAN model is input and outputs a binary image with white pixels representing the defect. The binary image is processed to obtain the minimum bounding rectangle of the defect area, and the minimum bounding rectangle of the defect area is marked on the output binary image to obtain a second result image.

[0012] The first result image and the second result image corresponding to S1 are spliced together to obtain a horizontal detection result image; the first result image and the second result image corresponding to S2 are spliced together to obtain a vertical detection result image; both the horizontal detection result image and the vertical detection result image contain rectangular boxes for marking defects.

[0013] In one embodiment, in step one, capturing images of the product's appearance from different directions specifically includes: a camera capturing images of the product's appearance from horizontal and vertical directions respectively, and a supporting assembly for placing the product shell is rotated 90 degrees along the vertical axis to achieve a transition from horizontal to vertical.

[0014] In one embodiment, the data set Training the Yolov4 model includes:

[0015] The loss function L1 used to calculate the loss between the predicted value and the true value output by the Yolov4 model is:

[0016]

[0017] k×k represents the number of grids into which the image is divided, and M represents the number of prediction boxes generated for each grid. Indicates that the jth candidate box of the i-th grid contains the target object, Indicates that the j-th candidate box of the i-th grid does not contain the target object; Represents the bounding box of the real object, Represents the bounding box of the predicted object, IOU is the intersection-union ratio of the bounding box of the predicted object to the bounding box of the real object, Ac represents the minimum rectangular area enclosed by the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, U represents the union area of the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, |·| represents the area of the region, Represents the confidence of the object predicted by the j-th candidate box of the i-th grid, represents the confidence of the object predicted by the j-th candidate box of the i-th grid, classes represents the set of all defect categories, and c represents one of the defects. Indicates the probability that the j-th candidate box of the i-th grid contains the target category c; Indicates that the j-th candidate box of the i-th grid contains the true value of category c. If it contains category c, the value is 1, and if it does not contain category c, it is 0.

[0018] In one embodiment, the CLGAN model includes: a generator and a discriminator.

[0019] In one embodiment, the training of the CLGAN model based on the training set B1 specifically includes:

[0020] The loss function L2 used to calculate the loss between the predicted value and the true value output by the CLGAN model is:

[0021] L2=λ x L PatchNCE (G,H,X)+λ y L PatchNCE (G,H,Y)+L GAN (G,D,X,Y);

[0022]

[0023] λ x ,λy is the weight coefficient, L PatchNCE is the PatchNCE loss, G represents the generator, H represents the fully connected layer, X represents the set of real images, Y represents the set of reconstructed repair images, and L GAN is the adversarial loss, D represents the discriminator, E(·) represents the expected value, l represents the number of layers, l∈{1,2,3,...,L}, L represents the total number of layers, f represents infoNCE, which is a contrast loss; S l represents the number of spatial positions in the lth layer, s represents a certain spatial position, x and After the corresponding

[0024] The output characteristics of the layer perceptron network are and where x and are samples of X and Y respectively; Represents the eigenvectors at other locations, Z l It is a set of multiple feature vectors transformed from x through the multi-layer perceptron network, H l is the multilayer perceptron network corresponding to the encoder layer l, Represents the encoder G enc The output features of layer l, yes A collection of multiple feature vectors converted after MLP.

[0025] In one embodiment, the data set Input the trained CLGAN model and output a binary image with white pixels representing defects, including:

[0026] Dataset An image to be detected I is input into the generator of the trained CLGAN model to obtain the reconstructed repair image I r , for the detected image I and the reconstructed repaired image I r Perform Gaussian filtering and calculate the residual to obtain the residual image, and then pass the Gaussian filter G guass Smoothed residual image:

[0027] I D1 =G guass (G guass (I)-G guass (I r ));

[0028] I D1 is the residual image after Gaussian filtering, G guass (I) and G guass (I r ) represent the image to be detected and the reconstructed and repaired image after Gaussian filtering respectively;

[0029] to I D1 Perform three-channel adaptive threshold segmentation to obtain the threshold image I D2 :

[0030]

[0031] is image I D1 In channel c, the value of pixel (x, y), T is the adaptive threshold, μ and σ are I D1 The mean and standard deviation of channel c; k is the standard deviation coefficient of threshold segmentation;

[0032] For the threshold image I D2 Perform grayscale processing to obtain a single-channel grayscale image I D3 :

[0033]

[0034] I D3 is a single-channel grayscale image, and They are image I D2 The R color channel, G color channel and B color channel of ω; r ,ω g and ω b They are image I D2 The weights of the three color channels;

[0035] to I D3 Perform single channel threshold segmentation to obtain a binary image I D4 ;

[0036] For binary image I D4 Perform a closing operation of expansion followed by corrosion to obtain a binary image I with white pixels representing defects. D5 :

[0037]

[0038] E is a 3×3 structural element; For expansion operation; For corrosion operations.

[0039] A product appearance defect detection system, comprising:

[0040] The data acquisition module collects images of the product appearance from different directions, combines the images without the logo area into training set A1, and combines the images with the logo area into training set B1; performs manual defect annotation and channel separation on the images in training set A1, and combines the obtained multiple single-channel grayscale images into a data set

[0041] Model training module, based on data set Train the Yolov4 model; train the CLGAN model based on training set B1;

[0042] The defect detection module collects images of the product appearance from the horizontal and vertical directions respectively, obtains the horizontal sub-image set S1 and the vertical sub-image set S2, separates the sub-images that do not contain the logo area in S1 and S2, and forms a data set with the obtained single-channel grayscale images The sub-images containing the logo area in S1 and S2 form the data sets respectively. The dataset Input the trained Yolov4 model and output a multi-channel image containing a defect rectangular annotation frame; perform channel fusion on the multi-channel image output by the Yolov4 model to obtain the first result image containing a rectangular frame annotating the defect; Input the trained CLGAN model and output a binary image with white pixels representing the defect; process the binary image to obtain the minimum bounding rectangle of the defect area, and mark the minimum bounding rectangle of the defect area on the output binary image to obtain a second result image; splice the first result image and the second result image corresponding to S1 to obtain a horizontal detection result image; splice the first result image and the second result image corresponding to S2 to obtain a vertical detection result image; both the horizontal detection result image and the vertical detection result image contain rectangular boxes that mark the defects.

[0043] In one embodiment, in step one, capturing images of the product's appearance from different directions specifically includes: a camera capturing images of the product's appearance from horizontal and vertical directions respectively, and a supporting assembly for placing the product shell is rotated 90 degrees along the vertical axis to achieve a transition from horizontal to vertical.

[0044] In one embodiment, the data set Training the Yolov4 model includes:

[0045] The loss function L1 used to calculate the loss between the predicted value and the true value output by the Yolov4 model is:

[0046]

[0047] k×k represents the number of grids into which the image is divided, and M represents the number of prediction boxes generated for each grid. Indicates that the jth candidate box of the i-th grid contains the target object, Indicates that the j-th candidate box of the i-th grid does not contain the target object; Represents the bounding box of the real object, Represents the bounding box of the predicted object, IOU is the intersection-union ratio of the bounding box of the predicted object to the bounding box of the real object, Ac represents the minimum rectangular area enclosed by the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, U represents the union area of the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, |·| represents the area of the region, Represents the confidence of the object predicted by the j-th candidate box of the i-th grid, represents the confidence of the object predicted by the j-th candidate box of the i-th grid, classes represents the set of all defect categories, and c represents one of the defects. Indicates the probability that the j-th candidate box of the i-th grid contains the target category c; Indicates that the j-th candidate box of the i-th grid contains the true value of category c. If it contains category c, the value is 1, and if it does not contain category c, it is 0.

[0048] In one embodiment, the training of the CLGAN model based on the training set B1 specifically includes:

[0049] The loss function L2 used to calculate the loss between the predicted value and the true value output by the CLGAN model is:

[0050] L2=λ x L PatchNCE (G,H,X)+λ y L PatchNCE (G,H,Y)+L GAN (G,D,X,Y);

[0051]

[0052] λ x ,λ y is the weight coefficient, L PatchNCE is the PatchNCE loss, G represents the generator, H represents the fully connected layer, X represents the set of real images, Y represents the set of reconstructed repair images, and L GAN is the adversarial loss, D represents the discriminator, E(·) represents the expected value, l represents the number of layers, l∈{1,2,3,...,L}, L represents the total number of layers, f represents infoNCE, which is a contrast loss; S l represents the number of spatial positions in the lth layer, s represents a certain spatial position, x and After the corresponding

[0053] The output characteristics of the layer perceptron network are and where x and are samples of X and Y respectively; Represents the eigenvectors at other locations, Z l It is a set of multiple feature vectors transformed from x through the multi-layer perceptron network, H l is the multilayer perceptron network corresponding to the encoder layer l, Represents the encoder G enc The output features of layer l, yes A collection of multiple feature vectors converted after MLP.

[0054] Compared with the prior art, the beneficial technical effects of the present invention are:

[0055] The clarity of product appearance image acquisition and defect imaging effects are improved, and a combination of supervised and unsupervised methods is used to improve the full-surface defect detection effect of the product. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a defect detection lighting device used in the present invention;

[0057] Figure 2 This is a schematic diagram of the network structure of the CLGAN model used in the present invention;

[0058] Figure 3 Schematic diagram of the encoder structure of the CLGAN model;

[0059] Figure 4 Schematic diagram of the decoder structure of the CLGAN model;

[0060] Figure 5 Schematic diagram of the structure of the discriminator of the CLGAN model;

[0061] Figure 6 Schematic diagram of the defect detection method of the present invention;

[0062] Figure 7 Schematic diagram of the defect detection process for the CLGAN model.

[0063] In the figure: 1. Main frame; 2. Detection component; 3. Support component; 4. Travel mechanism. DETAILED DESCRIPTION

[0064] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0065] like Figure 6 As shown, the product appearance defect detection method in the present invention specifically includes the following steps.

[0066] Step 1: Adjust the camera position, collect and preprocess the original image, and divide it into training set and test set; the specific steps include:

[0067] 1.1. Set the distance between the sample and the camera. The image should include the entire surface area. The image size is 3*2048*4500, with a height of 4500 and a width of 2048. Because some defects are not visible at certain angles and may be missed during inspection, images are collected from two directions: horizontally first and then vertically.

[0068] 1.2. Crop the image collected in step 1.1 to a size of 416×416.

[0069] 1.3. Divide the images processed in step 1.2 into two parts: one containing images outside the logo area, denoted as Dataset A, and the other containing images within the logo area, denoted as Dataset B. Dataset A is divided into two parts: the training set, which accounts for approximately 70%, denoted as A1, and the test set, which accounts for approximately 30%, denoted as A2. The defect-free images in Dataset B are divided into two parts in a 7:3 ratio: the training set, which accounts for 70%, denoted as B1, and the test set, which accounts for 30%, denoted as B2.

[0070] 1.4. Use the LabelImg annotation tool to manually annotate the images of datasets A1 and A2 with defects, that is, mark the defect locations with rectangular boxes.

[0071] 1.5, after processing in step 1.4, the dataset A1 is separated into multiple single-channel grayscale images through RGB channels. These single-channel grayscale images form a new dataset, denoted as The same data set A2 is also separated by channels to form a new data set, which is recorded as

[0072] like Figure 1As shown, in a preferred embodiment, the product appearance defect detection method proposed by the present invention adopts a defect detection lighting device, including a main frame 1, an isolation frame and a travel mechanism 4 are arranged in the middle of the main frame 1, a base plate is arranged at the bottom of the main frame 1, a control module is installed inside the base plate, a pair of support rods are fixedly connected in the middle of the isolation frame, a supporting assembly 3 is installed at the upper end of the support rods, a pair of slide rails are fixedly connected to the upper end of the main frame 1, and a detection assembly 2 is slidably connected to the slide rails, the detection assembly 2 includes a slide plate, a pair of mounting brackets are provided at the lower end of the slide plate, the pair of mounting brackets are connected to each other by a fixed rod, a linear array camera is installed at the lower end of the fixed rod, and a pair of linear light sources are installed at the lower end between the pair of mounting brackets. Through the cross-arranged linear light sources, the light on both sides is converged at a certain angle to the middle position of the light source, and the contrast of the defect is improved by reducing the background brightness, thereby highlighting the vertical scratches parallel to the direction of movement of the object, and cooperating with the linear array camera to improve the efficiency of defect detection.

[0073] The above device is applied to product appearance defect detection scenarios. A deep learning algorithm is used to automatically detect the captured images and output a visual output of the presence and location of defects on the product's appearance. In step 1.1, when capturing images from two directions, the image is first captured horizontally, then the support assembly 3 is rotated 90 degrees to capture the image vertically.

[0074] Step 2: Train and test the Yolov4 model and CLGAN model. The structure of the CLGAN model is shown in Figure 2 Specifically, the CLGAN model consists of two parts: a generator and a discriminator. The generator is responsible for image reconstruction and restoration, and consists of an encoder and a decoder. The discriminator distinguishes between real images and reconstructed images generated by the generator.

[0075] 2.1. Initialize the hyperparameters in the Yolov4 model, such as setting the learning rate to 0.001 and the batch size to 8. Then, use the training set processed in step 1.5. Input into the Yolov4 model for training. The loss between the predicted value and the true value output by the Yolov4 model is calculated as shown in formula (1). Backpropagation is performed based on the loss value to update the network parameters. When the network model converges to a certain extent and the loss no longer decreases, the training ends.

[0076] The loss function of Yolov4 model training is:

[0077] k×k represents the number of grids into which the image is divided, and M represents the number of prediction boxes generated for each grid. Indicates that the jth candidate box of the i-th grid contains the target object, Indicates that the j-th candidate box of the i-th grid does not contain the target object; Represents the bounding box of the real object, Represents the bounding box of the predicted object, IOU is the intersection-union ratio of the bounding box of the predicted object to the bounding box of the real object, Ac represents the minimum rectangular area enclosed by the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, and U represents the union area of the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image. Represents the confidence of the object predicted by the j-th candidate box of the i-th grid, represents the confidence of the real object of the jth candidate box of the i-th grid, classes represents the set of all defect categories, and c represents one of the defects. represents the probability that the j-th candidate box of the i-th grid contains the target category c, Indicates that the j-th candidate box of the i-th grid contains the true value of category c. If it contains category c, the value is 1, otherwise it is 0.

[0078] 2.2. First, initialize the hyperparameters in the CLGAN model, such as setting the learning rate to 0.001 and the batch size to 8. Then input the training set B1 into the model for training. The loss between the predicted value and the true value output by the CLGAN model is calculated as shown in formula (6). Backpropagation is performed based on the loss value to update the CLGAN model parameters. When the CLGAN model converges to a certain degree and L2 no longer decreases, the training ends.

[0079] L total =λ x L PatchNCE (G,H,X)+λ y L PatchNCE (G,H,Y)+L GAN (G,D,X,Y); (2)

[0080]

[0081] λ x ,λ y is the weight coefficient, L PatchNCE is the PatchNCE loss, G represents the generator, H represents the fully connected layer, X represents the set of real input images, Y represents the set of reconstructed repair images, and L GAN is the adversarial loss, D represents the discriminator, E(·) represents the expected value of the distribution function, l represents the number of layers, l∈{1,2,3,...,L}, L represents the total number of layers), f represents infoNCE, which is a contrast loss; S l Represents the number of spatial positions in the lth layer, s represents a certain spatial position, and the corresponding MLP network output feature is called Represents the eigenvectors at other locations, Z l is a set of multiple feature vectors transformed from x after MLP, where x and are samples of X and Y respectively, H l is the MLP network corresponding to the encoder layer l, Represents the encoder G enc The output features of layer l, yes A collection of multiple feature vectors converted after MLP.

[0082] In the present invention, the encoder in the model converts the input image into a latent space feature representation, and the decoder generates a reconstructed image based on this, and the generator strives to make it similar to the input image; the discriminator accepts the reconstructed image and the real image at the same time and distinguishes them, and its performance improves with training, prompting the generator to generate more realistic images; the real image and the reconstructed image obtain Pacth feature components through the same encoder, which are mapped to a vector matrix after encoding, and then projected to a shared feature space through MLP. By maximizing the mutual information between the input and output image patches, the comparative learning framework enhances the correlation between the generated image and the original image, and finally backpropagates and updates the model parameters based on the loss between the model output prediction value and the true value until the model converges.

[0083] 2.3, the test set The input is fed into the trained Yolov4 model for testing, and the Yolov4 model is evaluated based on the performance indicator mAP (i.e., the average value of various APs).

[0084] 2.4, the test set B2 is sent to the trained CLGAN model for testing, and the CLGAN model is evaluated based on the loss function obtained from the test.

[0085] Step 3: For a product appearance sample to be tested, use the trained Yolov4 and CLGAN models for detection.

[0086] 3.1. Collect an image of the sample to be tested according to step 1.1, one horizontal and one vertical image, and crop the two images separately. The original image is cropped into a set of 5×11 sub-images. If a sub-image is located in the mth row and nth column of the original image, it is labeled (m,n). After cropping the horizontal image and the vertical image, two sub-image sets are obtained, respectively, denoted as S1 and S2; the RGB channels of the part outside the logo area in S1 are separated to form multiple single-channel grayscale images to obtain a new set, denoted as Similarly, the part outside the logo area in S2 is also separated by channels to obtain a new set, which is recorded as The logo area images in S1 and S2 form a new set, which are respectively denoted as

[0087] 3.2, use two models to detect separately, and divide the dataset Input Yolov4 model, dataset Input the unsupervised detection model, namely the CLGAN model, for detection.

[0088] like Figure 7 As shown in Figure 2, the detection process of the unsupervised detection model is as follows:

[0089] (1) Dataset One of the images to be detected I is input into the generator of the CLGAN model trained in step 2.2 to obtain the reconstructed repaired image I r , the image to be detected I and the reconstructed repair image I r Perform Gaussian filtering and calculate the residual to obtain the residual image, and then smooth the residual image through Gaussian filtering:

[0090] I D1 =G guass (G guass (I)-G guass (I r )); (8)

[0091] Where: I D1 is the residual image after Gaussian filtering, G guass (I) and G guass (I r ) represent the image to be detected and the reconstructed image after Gaussian filtering, respectively.

[0092] (2) to I D1 Perform three-channel adaptive threshold segmentation, as shown in formula (9):

[0093]

[0094] Where: is image I D1 In channel c, the value of pixel (x, y), T is the adaptive threshold, μ and σ are the values of image I D1 The mean and standard deviation of channel c; k is the standard deviation coefficient of threshold segmentation; in the present invention, k is set to 3.

[0095] (3) For image I D2 Perform grayscale processing to obtain a single-channel grayscale image I D3 , the specific operation is shown in formula (10):

[0096]

[0097] Where: I D3is the grayscale image, and They are image I D2 Corresponding 3 RGB color channels; ω r 、ω g and ω b They are the weights of the three channels, generally set to ω r is 0.299, ω g is 0.587, ω b It is 0.114.

[0098] (4) to I D3 Perform single-channel threshold segmentation. The second segmentation can be performed directly using the mean plus standard deviation. The parameters c and k are both 1. The threshold segmentation is shown in formula (9).

[0099] For binary image I D4 Perform a closing operation of expansion followed by corrosion to obtain the detection result I D5 , as shown in formula (11):

[0100]

[0101] Where: E is a 3×3 structural element; For the expansion operation, For corrosion operations.

[0102] 3.3. After the two models have finished testing, the detection results of each model are obtained. If the Yolov4 model detects a defect, the output result is an image with the defect marked by a rectangular frame. If no defect is detected, the output result is an image without a rectangular frame. If the CLGAN model detects a defect, the output is a binary image with the defect represented by white pixels. If no defect is detected, the output is a binary image without white pixels.

[0103] 3.4, in the Yolov4 model, different channels of the same image are fused to obtain an RGB three-channel image.

[0104] 3.5. In order to make the final visualization results consistent, the output of the CLGAN unsupervised model is processed.

[0105] The specific steps include:

[0106] (1) Read the pixel matrix of the CLGAN model output result image and define the pixel phase with a value of 255.

[0107] The area formed by the adjacent areas is the defect area.

[0108] (2) Initialize a stack and a set. The stack is used to store access elements (i.e., matrix indexes), and the set is used to store popped elements. Start from the first row and first column and traverse in a row-first manner. When a pixel value of 255 is encountered in the matrix and the element has not been traversed, it indicates a defect area and the element is pushed into the stack.

[0109] (3) Repeat steps (4) to (8).

[0110] (4) Determine whether there is an unvisited element with a right adjacent pixel value of 255. If so, access the element and push it into the stack.

[0111] (5) If it does not exist, determine whether there is an unvisited element with a lower adjacent pixel value of 255 under the top element of the stack. If so, access the element and push it into the stack.

[0112] (6) If it does not exist, determine whether there is an unvisited element with a left adjacent pixel value of 255 to the top element of the stack. If so, access the element and push it into the stack.

[0113] (7) If it does not exist, determine whether there is an element with an adjacent pixel value of 255 that has not been visited. If so, visit the element and push it into the stack.

[0114] (8) If it does not exist, the top element of the stack is popped out and enters the collection. When the stack is empty, the loop is exited.

[0115] (9) According to step (3), the set saves all elements of the defect area, and finds the minimum and maximum values of the horizontal coordinate x in the set min 、x max , the minimum and maximum values of the vertical coordinate y min 、y max ; The upper left coordinate of the minimum bounding rectangle of the defect area is (x max ,y min ), the length and width of the rectangle are (y max -y min )、(x max -x min ).

[0116] (10) Mark the corresponding matrix on the RGB image according to the upper left coordinate, length, and width of the obtained minimum circumscribed matrix.

[0117] (11) Continue traversing from the position accessed in step ②. When an element with a pixel value of 255 that has not been traversed is accessed again, the stack and set in step ② are reinitialized, and the element is pushed into the stack and steps (3), (4), and (5) are repeated until all elements of the matrix have been accessed.

[0118] 3.6. The RGB image obtained in step 3.5 and the detection results of the sub-image belonging to S1 in the RGB image obtained in step 3.4 are spliced to obtain a horizontal detection result image. The binary images obtained in steps 3.4 and 3.5 are spliced with the detection results of the sub-image belonging to S2 in the RGB image to obtain a vertical detection result image.

[0119] 3.7, output the final result, which is the complete horizontal and vertical detection result image with the defects marked by rectangular boxes.

[0120] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0121] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for detecting product appearance defects, characterized in that: include: Step 1: Collect images of the product appearance from different directions, combine the images without the logo area into training set A1, and combine the images with the logo area into training set B1; perform manual defect annotation and channel separation on the images in training set A1, and form a data set with multiple single-channel grayscale images Step 2: Based on the dataset Train the Yolov4 model; train the CLGAN model based on training set B1; Step 3: Collect images of the product appearance from the horizontal and vertical directions respectively, and obtain the horizontal sub-image set S1 and the vertical sub-image set S2. Separate the sub-images that do not contain the logo area in S1 and S2, and form a data set with the obtained single-channel grayscale images. The sub-images containing the logo area in S1 and S2 form the data sets respectively. The dataset Input the trained Yolov4 model and output a multi-channel image containing a defect rectangular annotation box; perform channel fusion on the multi-channel image output by the Yolov4 model to obtain a first result image containing a rectangular box annotating the defect; The dataset The trained CLGAN model is input and outputs a binary image with white pixels representing the defect. The binary image is processed to obtain the minimum bounding rectangle of the defect area, and the minimum bounding rectangle of the defect area is marked on the output binary image to obtain a second result image. The first result image and the second result image corresponding to S1 are spliced together to obtain a horizontal detection result image; the first result image and the second result image corresponding to S2 are spliced together to obtain a vertical detection result image; both the horizontal detection result image and the vertical detection result image contain rectangular boxes for marking defects.

2. The product appearance defect detection method according to claim 1, characterized in that: In step one, the image capture of the product's appearance from different directions specifically includes: the camera captures images of the product's appearance from the horizontal and vertical directions respectively, and the supporting component for placing the product shell is rotated 90 degrees along the vertical axis to achieve the transition from horizontal to vertical.

3. The product appearance defect detection method according to claim 1, characterized in that: The dataset based Training the Yolov4 model includes: The loss function L1 used to calculate the loss between the predicted value and the true value output by the Yolov4 model is: k×k represents the number of grids into which the image is divided, and M represents the number of prediction boxes generated for each grid. Indicates that the jth candidate box of the i-th grid contains the target object, Indicates that the j-th candidate box of the i-th grid does not contain the target object; Represents the bounding box of the real object, Represents the bounding box of the predicted object, IOU is the intersection-union ratio of the bounding box of the predicted object to the bounding box of the real object, Ac represents the minimum rectangular area enclosed by the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, U represents the union area of the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, |·| represents the area of the region, Indicates the confidence of the object predicted by the j-th candidate box of the i-th grid, represents the confidence of the object predicted by the j-th candidate box of the i-th grid, classes represents the set of all defect categories, and c represents one of the defects. represents the probability that the j-th candidate box of the i-th grid contains the target category c; Indicates that the j-th candidate box of the i-th grid contains the true value of category c. If it contains category c, the value is 1, and if it does not contain category c, it is 0.

4. The product appearance defect detection method according to claim 1, characterized in that: The CLGAN model includes: a generator and a discriminator.

5. The product appearance defect detection method according to claim 1, characterized in that: The training of the CLGAN model based on the training set B1 specifically includes: The loss function L2 used to calculate the loss between the predicted value and the true value output by the CLGAN model is: L2=λ x L PatchNCE (G,H,X)+λ y L PatchNCE (G,H,Y)+L GAN (G,D,X,Y); λ x ,λ y is the weight coefficient, L PatchNCE is the PatchNCE loss, G represents the generator, H represents the fully connected layer, X represents the set of real images, Y represents the set of reconstructed repair images, and L GAN is the adversarial loss, D represents the discriminator, E(·) represents the expected value, l represents the number of layers, l∈{1,2,3,...,L}, L represents the total number of layers, f represents infoNCE, which is a contrast loss; S l represents the number of spatial positions in the lth layer, s represents a certain spatial position, x and The features output by the corresponding multi-layer perceptron network are: and where x and are samples of X and Y respectively; Represents the eigenvectors at other locations, Z l It is a set of multiple feature vectors transformed from x through the multi-layer perceptron network, H l is the multilayer perceptron network corresponding to the encoder layer l, Represents the encoder G enc The output features of layer l, yes A collection of multiple feature vectors converted after MLP.

6. The product appearance defect detection method according to claim 1, characterized in that: The dataset Input the trained CLGAN model and output a binary image with white pixels representing defects, including: Dataset An image to be detected I is input into the generator of the trained CLGAN model to obtain the reconstructed repair image I r , for the detected image I and the reconstructed repaired image I r Perform Gaussian filtering and calculate the residual to obtain the residual image, and then pass the Gaussian filter G guass Smoothed residual image: I D1 =G guass (G giass (I)-G guass (I r )); I D1 is the residual image after Gaussian filtering, G guass (I) and G guass (I r ) represent the image to be detected and the reconstructed and repaired image after Gaussian filtering respectively; to I D1 Perform three-channel adaptive threshold segmentation to obtain the threshold image I D2 : is image I D1 In channel c, the value of pixel (x, y), T is the adaptive threshold, μ and σ are I D1 The mean and standard deviation of channel c; k is the standard deviation coefficient of threshold segmentation; For the threshold image I D2 Perform grayscale processing to obtain a single-channel grayscale image I D3 : I D3 is a single-channel grayscale image, and They are image I D2 The R color channel, G color channel and B color channel of ω; r ,ω g and ω b They are image I D2 The weights of the three color channels; to I D3 Perform single channel threshold segmentation to obtain a binary image I D4 ; For binary image I D4 Perform a closing operation of expansion followed by corrosion to obtain a binary image I with white pixels representing defects. D5 : E is a 3×3 structural element; For expansion operation; For corrosion operation.

7. A product appearance defect detection system, characterized in that: include: The data acquisition module collects images of the product appearance from different directions, combines the images without the logo area into training set A1, and combines the images with the logo area into training set B1; performs manual defect annotation and channel separation on the images in training set A1, and combines the obtained multiple single-channel grayscale images into a data set Model training module, based on data set Train the Yolov4 model; train the CLGAN model based on training set B1; The defect detection module collects images of the product appearance from the horizontal and vertical directions respectively, obtains the horizontal sub-image set S1 and the vertical sub-image set S2, separates the sub-images that do not contain the logo area in S1 and S2, and forms a data set with the obtained single-channel grayscale images The sub-images containing the logo area in S1 and S2 form the data sets respectively. The dataset Input the trained Yolov4 model and output a multi-channel image containing a defect rectangular annotation frame; perform channel fusion on the multi-channel image output by the Yolov4 model to obtain the first result image containing a rectangular frame annotating the defect; The trained CLGAN model is input and a binary image with white pixels representing defects is output. The binary image is processed to obtain the minimum bounding rectangle of the defect area, and the minimum bounding rectangle of the defect area is marked on the output binary image to obtain a second result image. The first result image and the second result image corresponding to S1 are spliced to obtain a horizontal detection result image. The first result image and the second result image corresponding to S2 are spliced to obtain a vertical detection result image. Both the horizontal detection result image and the vertical detection result image contain rectangular boxes that mark the defects.

8. The product appearance defect detection system according to claim 7, characterized in that: In step one, the image capture of the product's appearance from different directions specifically includes: the camera captures images of the product's appearance from the horizontal and vertical directions respectively, and the supporting component for placing the product shell is rotated 90 degrees along the vertical axis to achieve the transition from horizontal to vertical.

9. The product appearance defect detection system according to claim 7, characterized in that: The dataset based Training the Yolov4 model includes: The loss function L1 used to calculate the loss between the predicted value and the true value output by the Yolov4 model is: k×k represents the number of grids into which the image is divided, and M represents the number of prediction boxes generated for each grid. Indicates that the jth candidate box of the i-th grid contains the target object, Indicates that the j-th candidate box of the i-th grid does not contain the target object; Represents the bounding box of the real object, Represents the bounding box of the predicted object, IOU is the intersection-union ratio of the bounding box of the predicted object to the bounding box of the real object, Ac represents the minimum rectangular area enclosed by the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, U represents the union area of the rectangular box at a certain position in the predicted image and the rectangular box at the position in the actual image, |·| represents the area of the region, Indicates the confidence of the object predicted by the j-th candidate box of the i-th grid, represents the confidence of the object predicted by the j-th candidate box of the i-th grid, classes represents the set of all defect categories, and c represents one of the defects. represents the probability that the j-th candidate box of the i-th grid contains the target category c; Indicates that the j-th candidate box of the i-th grid contains the true value of category c. If it contains category c, the value is 1, and if it does not contain category c, it is 0.

10. The product appearance defect detection system according to claim 7, characterized in that: The training of the CLGAN model based on the training set B1 specifically includes: The loss function L2 used to calculate the loss between the predicted value and the true value output by the CLGAN model is: L2=λ x L PatchNCE (G,H,X)+λ y L PatchNCE (G,H,Y)+L GAN (G,D,X,Y); λ x ,λ y is the weight coefficient, L PatchNCE is the PatchNCE loss, G represents the generator, H represents the fully connected layer, X represents the set of real images, Y represents the set of reconstructed repair images, and L GAN is the adversarial loss, D represents the discriminator, E(·) represents the expected value, l represents the number of layers, l∈{1,2,3,...,L}, L represents the total number of layers, f represents infoNCE, which is a contrast loss; S l represents the number of spatial positions in the lth layer, s represents a certain spatial position, x and The features output by the corresponding multi-layer perceptron network are: and where x and are samples of X and Y respectively; Represents the eigenvectors at other locations, Z l It is a set of multiple feature vectors transformed from x through the multi-layer perceptron network, H l is the multilayer perceptron network corresponding to the encoder layer l, Represents the encoder G enc The output features of layer l, yes A collection of multiple feature vectors converted after MLP.

Images