Product appearance quality judgment method based on industrial vision

Abstract

The invention provides a product appearance quality determination method based on industrial vision, and the method comprises the steps: constructing a multi-scale feature pyramid for a preprocessed image according to the characteristics of different sizes and various forms of flow mark defects, extracting flow mark features of different scales through convolution kernels of different scales, capturing flow mark regions of different sizes and forms, and determining the appearance quality of a product according to the flow mark regions. Obtaining a multi-scale feature map; on the basis of the potential flow mark area, aiming at the characteristics that the distinction degree between the flow mark defect and the background is not high and unified feature representation is lacked, the weights of different areas in the feature map are adjusted through the long-range dependency relationship between image areas, and the feature representation of the flow mark area is enhanced; on the basis of a lightweight feature map, aiming at the difficulty of constructing a flow mark defect training data set, training samples are enhanced and expanded by adopting data, and diversified flow mark forms and distribution positions are simulated by carrying out random rotation, translation, scaling and overturning transformation on original defect samples.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for judging product appearance quality using industrial vision. Background Art

[0002] In the appearance quality inspection of industrial visual products, flow mark defects are a common and difficult problem. Flow marks present diverse morphological features on the surface of products, with different sizes and random distribution positions, which brings many challenges to the design of defect detection algorithms. First, the contrast of flow mark defects against the background of the product surface is low, the edge contour is not obvious, and the distinction from the background is not high. Traditional edge detection and segmentation algorithms are difficult to accurately extract the flow mark area. Secondly, due to the diverse shapes and sizes of flow marks and the lack of unified feature representation, the detection method based on feature matching is difficult to work. Furthermore, the random distribution position of flow marks and the lack of prior position information increase the difficulty of positioning. Traditional detection methods based on low-level visual features such as grayscale, texture, and edge have poor universality and robustness in the face of weak contrast, multi-scale, and randomly distributed defects such as flow marks. Although the defect detection method based on deep learning can learn more robust and abstract feature representations, in the face of complex defects such as flow marks, how to design a sophisticated detection network structure, balance detection accuracy and real-time performance, and build a high-quality flow mark defect training data set is also an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a method for judging product appearance quality based on industrial vision, which mainly includes:

[0004] The product appearance images are collected by high-resolution industrial cameras and telecentric lenses. In view of the low contrast and unclear edge contour of flow mark defects against the product surface background, the images are pre-processed to enhance the contrast between the flow mark area and the background and highlight the visual characteristics of the edge contour information;

[0005] For the preprocessed images, a multi-scale feature pyramid is constructed to solve the problem that flow mark defects have different sizes and shapes. Flow mark features of different scales are extracted through convolution kernels of different scales, and flow mark areas of different sizes and shapes are captured to obtain a multi-scale feature map.

[0006] According to the multi-scale feature map, the target detection framework based on the region proposal network is adopted to locate the potential flow mark area by densely sampling candidate regions of different scales and aspect ratios on the convolutional feature map, considering the random distribution of flow mark defects and the lack of prior position information.

[0007] Based on the potential flow mark area, the flow mark defects are not easily distinguished from the background and lack unified feature representation. The weights of different areas in the feature map are adjusted through the long-range dependency between image areas to enhance the feature representation of the flow mark area.

[0008] According to the image of the feature representation of the enhanced flow mark area, in view of the balance between detection accuracy and real-time requirements of flow mark defect detection, a lightweight backbone network is used as a feature extractor to reduce the number of flow mark feature model parameters and calculation complexity, thereby improving the detection speed while ensuring detection accuracy and obtaining a lightweight feature map.

[0009] Based on lightweight feature maps, data enhancement is used to expand training samples to address the difficulty of constructing flow mark defect training data sets. By randomly rotating, translating, scaling, and flipping the original defect samples, diverse flow mark morphologies and distribution positions are simulated.

[0010] In the training process of simulating diverse flow mark morphologies and distribution positions, the difficult samples with large losses are recorded in detail to improve the model’s detection capabilities for weak-contrast and multi-scale flow mark defects. The defect classification and positioning tasks are jointly optimized, and the lightweight feature maps are classified and regressed to obtain the category and location information of flow mark defects. Combined with the preset judgment rules, the industrial vision product appearance quality judgment is completed.

[0011] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0012] The present invention discloses a method for judging the appearance quality of products using industrial vision. In view of the characteristics of flow mark defects on the product surface, such as low contrast, unclear edges, different sizes, diverse shapes, and random distribution, the present invention uses image preprocessing to enhance contrast, constructs a multi-scale feature pyramid to capture features of different scales, uses a region proposal network to locate potential defects, enhances flow mark feature representation through long-range dependencies, uses a lightweight backbone network to balance accuracy and speed, and uses data enhancement to expand samples and focus on difficult examples. The present invention effectively improves the detection capability of flow mark defects by integrating multiple technical means, realizes the accurate judgment of the appearance quality of industrial products, and can significantly improve the level of product quality control. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flowchart of a method for judging product appearance quality based on industrial vision. DETAILED DESCRIPTION

[0014] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] like Figure 1 In this embodiment, a method for judging product appearance quality by industrial vision may specifically include:

[0016] S101. Capture product appearance images through high-resolution industrial cameras and telecentric lenses. Considering the low contrast and unclear edge contour of flow mark defects against the product surface background, pre-process the images to enhance the contrast between the flow mark area and the background and highlight the visual features of the edge contour information.

[0017] The original image of the product appearance captured by a high-resolution industrial camera and a telecentric lens is obtained, and the original image is grayed to obtain a gray image. The gray image is subjected to contrast-limited adaptive histogram equalization to obtain a contrast-enhanced image. The contrast-enhanced image is locally contrast-enhanced using a Gaussian difference filter to obtain a locally enhanced image. Canny edge detection is performed on the locally enhanced image to obtain a binary edge image. For the binary edge image, background modeling and difference method are used to separate the background to obtain a foreground image. If there is a flow mark defect in the foreground image, a logical AND operation is performed between the foreground image and the binary edge image to obtain a flow mark defect detection result image. According to the flow mark defect detection result image, it is determined whether there is a flow mark defect on the product appearance.

[0018] Specifically, the original image of the product appearance collected by a high-resolution industrial camera and a telecentric lens is obtained, and the original image is grayed. The noise in the image is removed by a median filter with a window size of 5x5 to obtain a smoothed gray image. The smoothed gray image is processed by contrast-limited adaptive histogram equalization (CLAHE), and the grid size is set to 8x8 and the contrast limit parameter is set to 3.0 to improve the overall contrast of the image and enhance the difference between the flow mark area and the background to obtain a contrast-enhanced image. The Gaussian difference filter is used to perform local contrast enhancement on the contrast-enhanced image, and the standard deviation is set to 1 and 2 respectively. The flow mark area is specifically enhanced to obtain a locally enhanced image. Canny edge detection is performed on the locally enhanced image, and the low threshold is set to 50 and the high threshold is set to 150 to highlight the edge contour information of the flow mark defect, and a binary edge image is obtained. The binary edge image is morphologically processed, and a 3x3 rectangular structure element is used to perform an open operation to remove small noise points, and a 5x5 rectangular structure element is used to perform a closed operation to connect the edge breaks to obtain a continuous and complete flow mark defect contour image. In view of the complex background of different product surfaces, background modeling and difference method are used for background separation. The background update rate is set to 0.01, the threshold is set to 20, the flow mark area is highlighted, and the separated foreground image is obtained. The separated foreground image is logically ANDed with the continuous and complete flow mark defect contour image to obtain the final flow mark defect detection result image. The original image of the product appearance collected by a high-resolution industrial camera (resolution of 4096x3072 pixels) and a telecentric lens (magnification of 0.5 times) is obtained, and the original image is grayed to convert the RGB three-channel image into a single-channel grayscale image. The noise in the image is removed by median filtering with a window size of 5x5. During the filtering process, the pixel values ​​in the 5x5 area around each pixel are sorted, and the middle value is selected as the new value of the pixel to obtain a smoothed grayscale image. The smoothed grayscale image was processed with contrast-limited adaptive histogram equalization (CLAHE). The grid size was set to 8x8, the contrast limit parameter was set to 3.0, and the image was divided into 64 regions. The histograms were calculated and equalized respectively. At the same time, the amplitude of contrast enhancement was limited to avoid excessive noise enhancement. The contrast-enhanced image was locally enhanced using a Gaussian difference filter. The standard deviations were set to 1 and 2 respectively. The edge and detail information in the image were highlighted by subtracting the results of two Gaussian filters of different scales. Canny edge detection was performed on the locally enhanced image. The low threshold was set to 50 and the high threshold was set to 150. The image was first smoothed using a Gaussian filter, and then the image gradient was calculated and non-maximum suppression was performed. Finally, a binary edge image was obtained by double thresholding and edge connection.The binary edge image is morphologically processed, and a 3x3 rectangular structural element is used to perform an opening operation to remove small noise points. A 5x5 rectangular structural element is used to perform a closing operation to connect the edge breaks. During the opening operation, corrosion is first performed and then expansion is performed. During the closing operation, expansion is first performed and then corrosion is performed to obtain a continuous and complete flow mark defect contour image. In view of the complex background of different product surfaces, background modeling and difference method are used for background separation. The background update rate is set to 0.01 and the threshold is set to 20. The background model is established through the Gaussian mixture model (GMM). 3-5 Gaussian distributions are established for each pixel. The foreground is extracted according to the difference between the current frame and the background model to obtain the separated foreground image. The separated foreground image is logically ANDed with the continuous and complete flow mark defect contour image, and the foreground parts of both images are retained to obtain the final flow mark defect detection result image.

[0019] S102. For the preprocessed image, a multi-scale feature pyramid is constructed according to the characteristics of flow mark defects of different sizes and shapes. Flow mark features of different scales are extracted through convolution kernels of different scales, flow mark areas of different sizes and shapes are captured, and a multi-scale feature map is obtained.

[0020] A Gaussian pyramid is constructed for the preprocessed image, and image layers of different resolutions are generated through downsampling operations to obtain a multi-scale image pyramid; for each layer of the image in the multi-scale image pyramid, convolution operations are performed using convolution kernels of different sizes to extract flow mark features of different scales to obtain a feature map; based on the feature map, a ReLU function is used to perform nonlinear activation processing on each pixel value in the feature map to obtain an activated feature map; an upsampling operation is performed on the activated feature map, and a bilinear interpolation method is used to enlarge the low-resolution feature map to the same size as the original image to obtain a feature map of uniform size; all feature maps of uniform size are spliced ​​in the channel dimension to obtain a fused feature map; based on the fused feature map, a directional gradient histogram feature extraction method is used to divide the image into cells of a preset size, the gradient amplitude and direction of the pixels in each cell are calculated, and a histogram of a preset number of directions is generated to obtain a multi-scale and multi-morphological flow mark feature representation.

[0021] Specifically, a Gaussian pyramid is constructed for the preprocessed image, using a 5x5 Gaussian kernel with a standard deviation of 1.0. Image layers of different resolutions are generated through downsampling operations, and the size of each layer is 1 / 2 of the previous layer. A total of 4 layers of images are generated to obtain a multi-scale image pyramid. For each layer of images in the multi-scale image pyramid, three different sizes of convolution kernels are designed, namely 3x3, 5x5, and 7x7. The convolution kernel weights are initialized using Gaussian distribution, and the standard deviation is set to 0.01. Convolution operations are performed on each layer of images, and zero padding is used to keep the size of the feature map unchanged. Flow mark features of different scales are extracted to obtain 12 feature maps. Nonlinear activation processing is performed on the 12 feature maps, and the ReLU function f(x)=max(0,x) is used to process each pixel value in the feature map to enhance the expression ability of the feature and obtain the activated feature map. The activated feature map is upsampled to unify the size of the feature maps of each layer. The low-resolution feature map is enlarged to the same size as the original image using the bilinear interpolation method. The specific implementation process is as follows: the four nearest neighbor pixel positions of the target pixel in the original image are calculated, the interpolation weight is calculated according to the distance, and the interpolation result is obtained by weighted average. Then all feature maps are spliced ​​in the channel dimension to obtain a fused feature map containing multi-scale flow mark features. The fused feature map is processed using the directional gradient histogram feature extraction method, the image is divided into 8x8 pixel cells, the gradient amplitude and direction of the pixels in each cell are calculated, and histograms of 9 directions are generated to capture the morphological diversity of flow mark defects and obtain the final multi-scale and multi-morphological flow mark feature representation. A Gaussian pyramid is constructed for the preprocessed 1024x1024 pixel flow mark defect image. A 5x5 Gaussian kernel with a standard deviation of 1.0 is used to generate image layers of different resolutions through downsampling operations, and images of 512x512, 256x256, and 128x128 pixels are obtained in turn. A total of 4 layers of images constitute a multi-scale image pyramid. For each layer of images in the pyramid, three different sizes of convolution kernels are designed: 3x3, 5x5, and 7x7. The convolution kernel weights are initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Convolution operations are performed on each layer of images, and zero padding is used to keep the size of the feature map unchanged. For example, for a 3x3 convolution kernel, a zero value of 1 pixel is filled at the edge of the image. Flow mark features of different scales are extracted to obtain 12 feature maps. ReLU activation processing is performed on these 12 feature maps, f(x) = max(0, x), and values ​​less than 0 are set to 0, and values ​​greater than 0 remain unchanged, to enhance the nonlinear expression ability of the features. The activated feature map is upsampled, and the low-resolution feature map is enlarged to 1024x1024 pixels using a bilinear interpolation method.In the specific implementation, for the target pixel (x, y), find the four nearest pixels (x1, y1), (x1, y2), (x2, y1), (x2, y2) in the original image, calculate the interpolation weights w1 = (x2-x)(y2-y), w2 = (x2-x)(y-y1), w3 = (x-x1)(y2-y), w4 = (x-x1)(y-y1), and get the interpolation result by weighted average. All feature maps are spliced ​​in the channel dimension to obtain a fused feature map containing multi-scale flow mark features. The fused feature map is processed using the histogram of oriented gradients. The image is divided into cells of 8x8 pixels. The gradient amplitude and direction of the pixels in each cell are calculated to generate histograms in nine directions (0°, 20°, 40°, 60°, 80°, 100°, 120°, 140°, and 160°). The accumulated gradient value in each direction is used as a bin of the histogram to capture the morphological diversity of flow mark defects and obtain the final multi-scale and multi-morphological flow mark feature representation.

[0022] S103. According to the multi-scale feature map, in view of the random distribution of flow mark defects and the lack of prior position information, a target detection framework based on a region proposal network is adopted to locate potential flow mark areas by densely sampling candidate regions of different scales and aspect ratios on the convolutional feature map.

[0023] A convolution operation is performed on the multi-scale feature map. The convolution operation uses a 3x3 convolution kernel to perform a sliding window scan on the feature map to obtain a dense feature point set. Each feature point in the feature point set corresponds to a receptive field of the original image. Multiple anchor frames of different scales and aspect ratios are generated according to the feature point set. The scale range of the anchor frame is a preset pixel value, and the aspect ratio range is a preset ratio value. A preset number of anchor frames are generated for each feature point to obtain a candidate region set. The feature representation of each anchor frame in the candidate region set is calculated using a region of interest pooling operation. The region of interest pooling operation maps anchor frames of different sizes to a feature map of a fixed size. The feature representation is classified and regressed. If the classification and regression processing uses a fully connected layer, the fully connected layer contains a preset number of neurons, and a refined target detection frame and a corresponding confidence score are determined. The overlapping detection frames are processed using a non-maximum suppression algorithm. The non-maximum suppression algorithm sets a preset IoU threshold and confidence threshold, selects the detection frame with the highest score, and suppresses other detection frames with high overlap.

[0024] Specifically, a convolution operation is performed on the multi-scale feature map, and a 3x3 convolution kernel is used to perform a sliding window scan on the feature map. The convolution kernel weights are initialized using the He method, and the activation function uses ReLU to generate dense feature points. Each feature point corresponds to a receptive field of the original image, and a feature point set is obtained. Multiple anchor frames of different scales and aspect ratios are generated on each feature point. By calculating the statistical information of flow mark defects in the image, the scale range of the anchor frame is dynamically adjusted to {64, 128, 256, 512} pixels, and the aspect ratio range is {1:1, 1:2, 2:1, 1:3, 3:1}. For each feature point, 20 anchor frames are generated to form a candidate region set. The feature representation of each anchor frame in the candidate region set is calculated, and the region of interest pooling operation is used to map anchor frames of different sizes to a fixed-size 7x7 feature map. In the specific implementation, bilinear interpolation is first used to adjust the anchor frame area to 14x14 size, and then a 2x2 maximum pooling operation is performed to obtain a unified dimensional regional feature. The regional features are classified and regressed. Two fully connected layers are used. The first layer contains 1024 neurons and the second layer contains 512 neurons. The activation function is ReLU. Each regional feature is classified to obtain the category probability distribution. At the same time, the translation and scaling offsets of the anchor box are generated to obtain the refined target detection box and the corresponding confidence score. The non-maximum suppression algorithm is used to process the overlapping detection boxes. The IoU threshold is set to 0.5 and the confidence threshold is set to 0.7. The detection box with the highest score is selected and other detection boxes with high overlap are suppressed. The attention map is generated by the U-Net image segmentation network to highlight the areas in the image where flow marks may exist. The attention map is multiplied with the feature map to guide the region proposal network to focus on the key areas and improve the detection accuracy. The multi-scale feature map of 1024x1024 pixels is convolved. A 3x3 convolution kernel is used. The weights are initialized using the He method (the standard deviation is sqrt(2 / 9)). The feature map is scanned by sliding window with a step size of 16 pixels to generate 64x64 feature points. 20 anchor boxes are generated at each feature point. The size distribution of flow mark defects is determined by analyzing the image histogram, and the anchor box scale is dynamically adjusted to {64, 128, 256, 512} pixels with an aspect ratio of {1:1, 1:2, 2:1, 1:3, 3:1}. For each of the 81920 (64x64x20) candidate regions, bilinear interpolation is used to adjust it to a size of 14x14, and then 2x2 maximum pooling is performed to obtain a unified feature map of 7x7. Two layers of fully connected layers are used to process the feature map, with 1024 neurons in the first layer and 512 neurons in the second layer. The weight initialization uses the Xavier method and the bias is initialized to 0. The output layer contains category probabilities (background and flow marks, a total of 2 categories) and bounding box regression values ​​(center point coordinates, width and height adjustment, a total of 4 values).The non-maximum suppression algorithm is applied to the overlapping detection boxes, and the IoU between the two is calculated. If it is greater than 0.5, the box with high confidence is retained. Finally, the U-Net is used to generate the attention map. The encoder contains 4 convolution blocks, each with 2 3x3 convolution layers and 1 2x2 maximum pooling layer. The decoder uses transposed convolution upsampling, skipconnection connects the corresponding encoding layer, and outputs an attention map of the same size as the input image, which is multiplied pixel by pixel with the original feature map to highlight the potential flow mark area.

[0025] S104. Based on the potential flow mark area, in view of the low distinction between flow mark defects and background and the lack of unified feature representation, the weights of different areas in the feature map are adjusted through the long-range dependency between image areas to enhance the feature representation of the flow mark area.

[0026] Perform self-attention calculation on the feature map of the potential flow mark area, obtain the feature map and reshape it into a HW×C matrix, and obtain the three matrices Q, K, and V through linear transformation. Use spatial pyramid pooling to extract multi-scale features from the feature map, and perform average pooling operation on each sub-region according to the preset pooling size to obtain a multi-scale feature representation. Perform non-local operations on the multi-scale feature representation. If the dot product similarity between each position in the feature map and all other positions is greater than the preset threshold, the features are weighted averaged according to the generated similarity matrix. The original feature map and the enhanced feature map are feature fused, and the two feature maps are channel-reduced by 1×1 convolution, and the features are merged by element-by-element addition. The feature distinction between the flow mark area and the background is enhanced by the contrastive learning method. The features of the flow mark area and the background area are randomly selected from the enhanced feature map as positive and negative sample pairs. It is judged whether the positive and negative sample pairs meet the preset contrastive learning conditions. If they do, the feature representation is optimized using the InfoNCE loss function.

[0027] Specifically, self-attention calculation is performed on the feature map of the potential flow mark area, and the feature map is reshaped into a matrix of HW×C. The three matrices Q, K, and V are generated through linear transformation. The weight initialization adopts the Xavier method and the dimension is set to 512. Eight attention heads are set, and the scaling factor is 1 / sqrt(64). The attention weight matrix is ​​obtained by matrix multiplication, and the weighted sum of V is obtained to obtain the enhanced feature map. Spatial pyramid pooling is used to extract multi-scale features from the feature map. The pooling size is set to 1×1, 2×2, and 4×4. The average pooling operation is performed on each sub-region to obtain the feature vectors of 1, 4, and 16 sub-regions. These feature vectors are concatenated to obtain multi-scale feature representation. Non-local operations are performed on the multi-scale feature representation. The dot product similarity is used to calculate the similarity between each position in the feature map and all other positions, and a similarity matrix is ​​generated. The features are weighted averaged according to the similarity matrix to obtain a feature map containing global context information. The original feature map and the enhanced feature map are fused, and the channel dimension reduction of the two feature maps is performed by 1×1 convolution. The number of input channels is 1024, the number of output channels is 512, and the activation function is ReLU. Then, the features are merged by element-by-element addition. Finally, the number of feature channels is adjusted by 3×3 convolution. The number of input channels is 512, the number of output channels is 256, and the activation function is ReLU. The final enhanced feature map of the flow mark area is obtained. The feature distinction between the flow mark area and the background is enhanced by the contrastive learning method. The features of the flow mark area and the background area are randomly selected from the enhanced feature map as positive and negative sample pairs. The feature representation is optimized using the InfoNCE loss function. The temperature parameter is set to 0.07 and the batch size is 256. The 256×256×512 potential flow mark area feature map is processed and first reshaped into a 65536×512 matrix. The Q, K, and V matrices are generated by three independent linear transformation layers. The weight matrix size of each layer is 512×512, initialized using the Xavier method, and the standard deviation is

[0028] sqrt(2 / (512+512)). Set 8 attention heads, each with a dimension of 64 and a scaling factor of 1 / sqrt(64)≈0.125. Calculate the attention weight matrix A=softmax(QK^T / 0.125) with a size of 65536×65536, and then calculate AV to obtain the enhanced features. Perform spatial pyramid pooling on the enhanced features, 1×1 pooling to obtain a 1×512 vector, 2×2 pooling to obtain a 4×512 matrix, and 4×4 pooling to obtain a 16×512 matrix. These features are concatenated into a 21×512 multi-scale feature representation. Calculate the dot product similarity matrix S, where S_ij=the inner product of the i-th feature and the j-th feature, to generate a 21×21 similarity matrix. Perform softmax normalization on S, and then perform matrix multiplication with the original feature to obtain a feature containing global information. The original features and enhanced features are fused using 1×1 convolution, with 512 convolution kernels, 1 stride, no padding, 1024 input channels, 512 output channels, and ReLU activation. The number of channels is then adjusted using 3×3 convolution, with 256 convolution kernels, 1 stride, 1 padding, 512 input channels, 256 output channels, and ReLU activation. Finally, 128 flow mark region features and 128 background region features are randomly selected from the 256×256×256 feature map to form 256 sample pairs. The InfoNCE loss function L=-log(exp(p·p+ / 0.07) / ∑exp(p·p_i / 0.07)) is used to optimize the feature representation, where p and p+ are positive sample pairs and p_i is all samples.

[0029] S105. Based on the image represented by the features of the enhanced flow mark area, a lightweight backbone network is used as a feature extractor to reduce the number of flow mark feature model parameters and computational complexity in order to improve the detection speed while ensuring the detection accuracy and obtain a lightweight feature map in view of the need to balance the detection accuracy and real-time requirements of flow mark defect detection.

[0030] The input feature map is replaced by the standard convolution with depthwise separable convolution to perform channel-by-channel convolution. The depthwise separable convolution includes a 3x3 convolution kernel, a convolution layer with a stride of 1 and a padding of 1, and a 1x1 convolution layer. According to the feature map after the depthwise separable convolution, the channel attention mechanism is used to perform channel weighting. The channel weighting obtains the channel descriptor through global average pooling, and the channel weight is generated through two layers of fully connected layers. The number of neurons in the first layer is 1 / 16 of the number of channels in the feature map, and the number of neurons in the second layer is the same as the number of channels in the feature map. A skip connection structure is introduced to the channel-enhanced feature map. The feature maps of the third and fifth layers of the network are selected as shallow features and are fused with the feature maps of the last layer. After adjusting the number of channels with 1x1 convolution, the features are merged by element-by-element addition. The fused feature map is pruned, and the L1 norm of each convolution kernel is calculated as the importance score. The important convolution kernels are screened according to the preset threshold, and the convolution kernels with scores below the threshold are removed. The teacher-student network structure is used to perform knowledge distillation on the sparse feature map. The original model is used as the teacher network and the lightweight network is used as the student network. Distillation is performed through soft label and feature map matching losses to obtain a compressed lightweight model.

[0031] Specifically, the standard convolution is replaced by the depthwise separable convolution, and the input feature map is convolved channel by channel. The convolution kernel size is 3x3, the stride is 1, and the padding is 1. Then, 1x1 convolution is used for feature combination to obtain the feature map after the depthwise separable convolution. The channel attention mechanism is used to weight the feature map, and the channel descriptor is obtained by global average pooling. The channel weight is generated through two layers of fully connected layers. The number of neurons in the first layer is 1 / 16 of the number of channels in the feature map, and the number of neurons in the second layer is the same as the number of channels in the feature map. The activation function uses ReLU to weight each channel of the feature map to obtain a channel-enhanced feature map. The skip connection structure is introduced, and the feature maps of the 3rd and 5th layers of the network are selected as shallow features. The features are fused with the feature maps of the last layer. After adjusting the number of channels by 1x1 convolution, the features are merged by element-by-element addition to enhance the multi-scale expression ability of the features and obtain the fused feature map. The fused feature map is pruned, and the L1 norm of each convolution kernel is calculated as the importance score. The important convolution kernels are screened according to the preset threshold, and the convolution kernels with scores below the threshold are removed to obtain a sparse feature map. The feature map and model parameters are compressed through 8-bit fixed-point quantization, and the floating-point numbers are converted to fixed-point numbers to reduce storage space and calculation amount. The teacher-student network structure is used for knowledge distillation, and the original model is used as the teacher network and the lightweight network is used as the student network. Distillation is performed through soft label and feature map matching loss to obtain a compressed lightweight model. A single-stage detector is used as a lightweight detection head, and a 3x3 convolution is used to generate a predicted feature map. The target position and category probability are directly regressed to reduce the amount of calculation and obtain the final lightweight feature map and detection results. Apply depthwise separable convolution to the input feature map of 256x256x64, use 64 3x3 convolution kernels for channel-by-channel convolution, with a step size of 1 and padding of 1, keep the feature map size unchanged, and then use 64 1x1 convolution kernels for feature combination to obtain an output feature map of 256x256x64. Apply the channel attention mechanism to the output feature map, first perform global average pooling to obtain a 1x1x64 channel descriptor, and pass through two layers of fully connected layers, the first layer has 4 neurons (64 / 16), the second layer has 64 neurons, use the ReLU activation function, generate 64 channel weights, and perform channel weighting on the feature map. The feature maps of the 3rd layer (64x64x256) and the 5th layer (32x32x512) of the network are selected as shallow features and fused with the feature map of the last layer (256x256x64). The number of shallow feature channels is adjusted to 64 using 1x1 convolution, and then upsampled to 256x256 size, and the three layers of features are merged by element-by-element addition.The fused 256x256x64 feature map is pruned, and the L1 norm of 64 convolution kernels is calculated as the importance score. The threshold is set to 0.01, and the convolution kernels with scores below the threshold are pruned. Assuming that 48 convolution kernels remain, a sparse feature map of 256x256x48 is obtained. The feature map and model parameters are quantized to 8 bits, and the floating point numbers in the range of [-1, 1] are mapped to the integer range of [-128, 127]. The original model (teacher network) and the lightweight model (student network) are used for knowledge distillation, with the temperature parameter set to 3 and the weight to 0.5. The student network is trained by combining hard label loss and soft label loss. Finally, a 3x3 convolution is used to generate the predicted feature map, and the number of output channels is 15 (assuming that 3 types of targets are detected, each with 5 parameters: 4 position parameters and 1 confidence), and a lightweight detection feature map of 256x256x15 is obtained.

[0032] S106. Based on lightweight feature maps, in view of the difficulty of constructing flow mark defect training data sets, data enhancement is used to expand training samples. By randomly rotating, translating, scaling, and flipping the original defect samples, diverse flow mark morphologies and distribution positions are simulated.

[0033] The original defect samples are geometrically transformed, and the transformed sample images are obtained by random rotation angles, pixel translation, scaling and random flipping operations to generate a sample set enhanced by geometric transformation. According to the sample set enhanced by geometric transformation, the image blending technology is used to alpha blend different defect samples according to the preset ratio to obtain new composite defect samples. For the composite defect samples, the sample images are Gaussian blurred, sharpened and Gaussian noise is added using image filtering and noise injection methods to obtain a sample set enhanced by filtered noise. If the sample set enhanced by filtered noise meets the preset conditions, the overall brightness and contrast level of the sample image are changed by adjusting the contrast and brightness using a linear transformation method to generate a sample set enhanced by illumination changes. For the sample set enhanced by illumination changes, the data enhancement method is applied to the lightweight feature map, and the enhancement operation is performed in the feature space. The feature map is subjected to random channel dropout and eigenvalue perturbation to obtain the robustness and generalization ability of the enhanced feature map.

[0034] Specifically, the original defect samples were geometrically transformed, and the transformed sample images were generated by random rotation angles (0-360 degrees), pixel translation (±20 pixels), scaling (0.8-1.2) and random flipping operations to obtain a sample set enhanced by geometric transformation. Image blending technology was used to alpha blend different defect samples at a ratio of 0.3-0.7 to generate new composite defect samples, increasing the diversity and complexity of the samples. The composite defect samples contained the superposition of multiple flow mark features. Image filtering and noise injection methods were used to perform Gaussian blur (kernel size 5x5, standard deviation 1.5), sharpening, and adding Gaussian noise (mean 0, variance 0.01) to the sample images to simulate the flow mark defect characteristics under different imaging conditions, and obtain a sample set enhanced by filtering noise. By adjusting the contrast and brightness, the linear transformation method was used to change the overall brightness (±50) and contrast level (0.5-1.5) of the sample images to simulate the flow mark defect performance under different lighting conditions, and generate a sample set enhanced by lighting changes. The above data enhancement method is applied to lightweight feature maps, and enhancement operations are performed in the feature space. Random channel dropout (discarding probability 0.1-0.3) and eigenvalue perturbation (Gaussian noise with mean 0 and standard deviation 0.05) are performed on the feature maps to enhance the robustness and generalization ability of the feature maps. Data enhancement is performed on the original flow mark defect sample of 256x256 pixels. First, geometric transformation is applied: randomly select a rotation angle between 0-360 degrees, such as 45 degrees; translate the image within the range of ±20 pixels, such as +10 pixels in the horizontal direction and -15 pixels in the vertical direction; randomly select a scaling ratio between 0.8-1.2, such as 1.1; and perform horizontal flipping with a 50% probability. Then, image blending is performed, two different defect samples are selected, and alpha blending is performed using weights of 0.4 and 0.6 to generate a composite sample containing multiple flow mark features. Then, a Gaussian blur kernel of 5x5 size and standard deviation of 1.5 is applied to the image for smoothing, and then the Laplacian operator is used for sharpening, and finally Gaussian noise with mean 0 and variance 0.01 is added. The image contrast and brightness are adjusted, the brightness value is increased by 30, and the contrast coefficient is set to 1.2. For the 64x64x256 lightweight feature map, 20% of the channels (51 channels) are randomly selected for dropout, the values ​​of these channels are set to 0, and then Gaussian noise with mean 0 and standard deviation 0.05 is added to the remaining channels.

[0035] S107. In the process of simulating the diverse flow mark morphology and distribution position training, focus on recording the difficult samples with large losses to improve the model's detection ability for weak contrast and multi-scale flow mark defects. Jointly optimize the defect classification and positioning tasks, classify and regress the lightweight feature map, obtain the category and location information of the flow mark defect, and complete the industrial vision product appearance quality judgment in combination with the preset judgment rules.

[0036] A 3x3 convolution operation is performed on the lightweight feature map, with a step size of 1 and a padding of 1, to obtain a classification branch and a regression branch; the classification branch outputs the probability of the flow mark category at each position, and the regression branch outputs the coordinate offset of the flow mark bounding box. A feature pyramid network structure is used to fuse feature maps of different scales to obtain the feature representation of multi-scale flow mark defects. According to the outputs of the classification branch and the regression branch, the classification loss and regression loss are calculated; the samples are sorted according to the total loss value, and the samples with the top 20% of the loss value are selected as difficult samples; if the sample is a difficult sample, it is given a 2-fold weight, and its sampling frequency is increased to 1.5 times the original in the next round of training. The detection results are post-processed, and the detection box with the highest confidence is screened out by the non-maximum suppression algorithm; the IoU threshold of the non-maximum suppression algorithm is set to 0.5, and all detection boxes are traversed, the box with the highest score is retained, and other boxes with high overlap are suppressed.

[0037] Specifically, a 3x3 convolution operation is performed on the lightweight feature map with a step size of 1 and a padding of 1 to generate a classification branch and a regression branch. The classification branch outputs the probability of the flow mark category at each position, and the regression branch outputs the coordinate offset of the flow mark bounding box. A feature pyramid network structure is used to fuse feature maps of different scales to handle multi-scale flow mark defects. The classification loss and regression loss are calculated, and focalloss is used as the classification loss function, the gamma value is set to 2, the alpha value is set to 0.25, smoothL1loss is used as the regression loss function, and the beta value is set to 1.0. The two losses are weighted and summed to obtain the total loss. The samples are sorted according to the total loss value, and the samples with the top 20% of the loss value are selected as difficult examples. These samples are given a weight of 2 times, and their sampling frequency is increased to 1.5 times the original in the next round of training. The detection results are post-processed, and the detection box with the highest confidence is screened out by the non-maximum suppression algorithm. The IoU threshold is set to 0.5, and all detection boxes are traversed to retain the box with the highest score and suppress other boxes with high overlap. The severity of the flow mark defect is judged according to the preset area threshold, length threshold and depth threshold, and the final quality judgment result is output. The visibility of weak contrast flow marks is enhanced by adaptive histogram equalization, and the local contrast of the image is enhanced. The block size is set to 8x8 pixels and the contrast limit parameter is set to 3.0 to enhance the detectability of weak contrast flow marks. A 3x3 convolution is applied to the lightweight feature map of 256x256x64 with a step size of 1 and a padding of 1 to generate 128 classification channels and 256 regression channels. A feature pyramid network is used to fuse feature maps of three scales: 32x32x256, 64x64x128 and 128x128x64. The number of channels is adjusted by 1x1 convolution, and the elements are added after upsampling. Calculate focalloss, set gamma = 2, alpha = 0.25, for misclassified samples with a confidence of 0.3, the loss value is about 0.0879; calculate smoothL1loss, set beta = 1.0, for samples with a prediction deviation of 5 pixels, the loss value is 4.5. For samples with a batch size of 256, select the 51 with the highest loss values ​​as hard examples, increase their weights from 1 to 2, and increase the sampling frequency by 50%. Perform non-maximum suppression, set the IoU threshold to 0.5, for 10 overlapping detection boxes, assume that the highest score is 0.9, and the rest are 0.8 and 0.7 respectively. Keep the box with 0.9 and suppress other boxes with an IoU greater than 0.5. Set the flow mark area threshold to 100 square pixels, the length threshold to 20 pixels, and the depth threshold estimated by the gray value difference to 30.Adaptive histogram equalization is performed on the original image. The 256x256 image is divided into 1024 8x8 blocks. Histogram equalization is performed on each block, the contrast gain is limited to 3.0, the overflowed histogram counts are redistributed, and finally bilinear interpolation is performed on the block boundaries to obtain the enhanced image and improve the visibility of weak contrast flow marks.

[0038] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for judging product appearance quality based on industrial vision, characterized in that: The method comprises: The product appearance images are collected by high-resolution industrial cameras and telecentric lenses. In view of the low contrast and unclear edge contour of flow mark defects against the product surface background, the images are pre-processed to enhance the contrast between the flow mark area and the background and highlight the visual characteristics of the edge contour information; For the preprocessed images, a multi-scale feature pyramid is constructed to solve the problem that flow mark defects have different sizes and shapes. Flow mark features of different scales are extracted through convolution kernels of different scales, and flow mark areas of different sizes and shapes are captured to obtain a multi-scale feature map. According to the multi-scale feature map, the target detection framework based on the region proposal network is adopted to locate the potential flow mark area by densely sampling candidate regions of different scales and aspect ratios on the convolutional feature map, considering the random distribution of flow mark defects and the lack of prior position information. Based on the potential flow mark area, the flow mark defects are not easily distinguished from the background and lack unified feature representation. The weights of different areas in the feature map are adjusted through the long-range dependency between image areas to enhance the feature representation of the flow mark area. According to the image of the feature representation of the enhanced flow mark area, in view of the balance between detection accuracy and real-time requirements of flow mark defect detection, a lightweight backbone network is used as a feature extractor to reduce the number of flow mark feature model parameters and calculation complexity, thereby improving the detection speed while ensuring detection accuracy and obtaining a lightweight feature map. Based on lightweight feature maps, data enhancement is used to expand training samples to address the difficulty of constructing flow mark defect training data sets. By randomly rotating, translating, scaling, and flipping the original defect samples, diverse flow mark morphologies and distribution positions are simulated. In the training process of simulating diverse flow mark morphologies and distribution positions, the difficult samples with large losses are recorded in detail to improve the model’s detection capabilities for weak-contrast and multi-scale flow mark defects. The defect classification and positioning tasks are jointly optimized, and the lightweight feature maps are classified and regressed to obtain the category and location information of flow mark defects. Combined with the preset judgment rules, the industrial vision product appearance quality judgment is completed.

2. The method according to claim 1, wherein: The product appearance image is collected by a high-resolution industrial camera and a telecentric lens. In view of the low contrast and unclear edge contour of the flow mark defect against the product surface background, the image is preprocessed to enhance the contrast between the flow mark area and the background and highlight the visual features of the edge contour information, including: Obtain the original image of the product appearance captured by a high-resolution industrial camera and a telecentric lens, and grayscale the original image to obtain a grayscale image; Perform contrast-limited adaptive histogram equalization on the grayscale image to obtain a contrast-enhanced image; The contrast-enhanced image is locally enhanced by using a Gaussian difference filter to obtain a locally enhanced image; Perform Canny edge detection on the local enhanced image to obtain a binary edge image; For the binary edge image, background modeling and difference method are used to separate the background and obtain the foreground image; If there is a flow mark defect in the foreground image, a logical AND operation is performed on the foreground image and the binary edge image to obtain a flow mark defect detection result image; According to the flow mark defect detection result image, determine whether there is a flow mark defect on the product appearance.

3. The method according to claim 1, wherein: For the preprocessed image, a multi-scale feature pyramid is constructed according to the characteristics of flow mark defects of different sizes and shapes. Flow mark features of different scales are extracted through convolution kernels of different scales, flow mark areas of different sizes and shapes are captured, and a multi-scale feature map is obtained, including: A Gaussian pyramid is constructed for the preprocessed image, and image layers of different resolutions are generated through downsampling operations to obtain a multi-scale image pyramid; For each layer of the multi-scale image pyramid, convolution kernels of different sizes are used to perform convolution operations to extract flow mark features of different scales and obtain feature maps; According to the feature map, the ReLU function is used to perform nonlinear activation processing on each pixel value in the feature map to obtain the activated feature map; The activated feature map is upsampled, and the low-resolution feature map is enlarged to the same size as the original image using a bilinear interpolation method to obtain a feature map of uniform size; All feature maps of uniform size are concatenated in the channel dimension to obtain a fused feature map; According to the fused feature map, the directional gradient histogram feature extraction method is adopted to divide the image into cells of preset size. The gradient amplitude and direction of the pixels in each cell are calculated, and the histogram of the preset number of directions is generated to obtain the multi-scale and multi-morphological flow mark feature representation.

4. The method according to claim 1, wherein: According to the multi-scale feature map, the target detection framework based on the region proposal network is adopted to locate the potential flow mark area by densely sampling candidate areas of different scales and aspect ratios on the convolution feature map, targeting the random distribution of flow mark defects and the lack of prior position information, including: The multi-scale feature map is convolved. The convolution operation uses a 3x3 convolution kernel to perform a sliding window scan on the feature map to obtain a dense set of feature points. Each feature point in the feature point set corresponds to a receptive field of the original image. Generate a plurality of anchor frames of different scales and aspect ratios according to the feature point set, wherein the scale range of the anchor frame is a preset pixel value, and the aspect ratio range is a preset ratio value, and generate a preset number of anchor frames for each feature point to obtain a candidate region set; The region of interest pooling operation is used to calculate the feature representation of each anchor box in the candidate region set. The region of interest pooling operation maps anchor boxes of different sizes to a feature map of a fixed size. Performing classification and regression processing on the feature representation, if the classification and regression processing adopts a fully connected layer, the fully connected layer includes a preset number of neurons, and a refined target detection frame and a corresponding confidence score are determined; The non-maximum suppression algorithm is used to process overlapping detection frames. The non-maximum suppression algorithm sets a preset IoU threshold and confidence threshold, selects the detection frame with the highest score and suppresses other detection frames with high overlap.

5. The method according to claim 1, wherein: Based on the potential flow mark area, the weights of different areas in the feature map are adjusted through the long-range dependency between image areas to enhance the feature representation of the flow mark area, aiming at the characteristics that the flow mark defect has low distinction between the background and the flow mark defect and lacks a unified feature representation, including: Perform self-attention calculation on the feature map of the potential flow mark area, obtain the feature map and reshape it into a HW×C matrix, and obtain the three matrices Q, K and V through linear transformation; Using spatial pyramid pooling to extract multi-scale features from the feature map, performing an average pooling operation on each sub-region according to a preset pooling size, and obtaining a multi-scale feature representation; Perform non-local operations on multi-scale feature representations. If the dot product similarity between each position in the feature map and all other positions is greater than a preset threshold, the features are weighted averaged according to the generated similarity matrix. The original feature map and the enhanced feature map are fused, and the channel dimension of the two feature maps is reduced by 1×1 convolution, and the features are merged by element-by-element addition; The feature distinction between the flow mark area and the background is enhanced through the contrastive learning method. The features of the flow mark area and the background area are randomly selected from the enhanced feature map as positive and negative sample pairs. It is judged whether the positive and negative sample pairs meet the preset contrastive learning conditions. If so, the InfoNCE loss function is used to optimize the feature representation.

6. The method according to claim 1, wherein: According to the image represented by the features of the enhanced flow mark area, a lightweight backbone network is used as a feature extractor to balance the detection accuracy and real-time requirements of flow mark defect detection, reduce the number of flow mark feature model parameters and calculation complexity, improve the detection speed while ensuring the detection accuracy, and obtain a lightweight feature map, including: The input feature map is replaced by a depthwise separable convolution to perform channel-by-channel convolution. The depthwise separable convolution includes a 3x3 convolution kernel, a convolution layer with a stride of 1 and a padding of 1, and a 1x1 convolution layer. According to the feature map after the depthwise separable convolution, the channel attention mechanism is used to perform channel weighting. The channel weighting obtains the channel descriptor through global average pooling, and the channel weight is generated through two layers of fully connected layers. The number of neurons in the first layer is 1 / 16 of the number of channels in the feature map, and the number of neurons in the second layer is the same as the number of channels in the feature map. A skip connection structure is introduced to the channel-enhanced feature map. The feature maps of the 3rd and 5th layers of the network are selected as shallow features and fused with the feature maps of the last layer. After adjusting the number of channels using 1x1 convolution, the features are merged by element-by-element addition. Perform feature map pruning on the fused feature map, calculate the L1 norm of each convolution kernel as the importance score, filter out important convolution kernels according to the preset threshold, and remove convolution kernels with scores lower than the threshold; The teacher-student network structure is used to perform knowledge distillation on the sparse feature map. The original model is used as the teacher network and the lightweight network is used as the student network. Distillation is performed through soft label and feature map matching losses to obtain a compressed lightweight model.

7. The method according to claim 1, wherein: Based on the lightweight feature map, the method aims to solve the difficulty of constructing the flow mark defect training data set by using data enhancement to expand the training samples. By randomly rotating, translating, scaling, and flipping the original defect samples, the method simulates various flow mark morphologies and distribution positions, including: Perform geometric transformation on the original defect sample, obtain the transformed sample image through random rotation angle, pixel translation, scaling and random flipping operations, and generate a sample set enhanced by geometric transformation; According to the sample set enhanced by geometric transformation, the image blending technology is used to alpha blend different defect samples according to the preset ratio to obtain new composite defect samples; For the composite defect samples, the image filtering and noise injection methods are used to perform Gaussian blurring, sharpening and adding Gaussian noise to the sample images to obtain a sample set with enhanced filtering noise. If the sample set enhanced by the filtered noise meets the preset conditions, the overall brightness and contrast level of the sample image are changed by using a linear transformation method through contrast and brightness adjustment to generate a sample set enhanced by illumination changes; For the sample set with enhanced illumination changes, the data augmentation method is applied to the lightweight feature map, and the enhancement operation is performed in the feature space. The feature map is subjected to random channel dropout and eigenvalue perturbation to obtain the robustness and generalization ability of the enhanced feature map.

8. The method according to claim 1, wherein: In the training process of simulating diverse flow mark morphologies and distribution positions, the difficult samples with large losses are recorded in detail to improve the model's detection capability for weak contrast and multi-scale flow mark defects; the defect classification and positioning tasks are jointly optimized, and the lightweight feature maps are classified and regressed to obtain the category and location information of the flow mark defects. Combined with the preset judgment rules, the product appearance quality judgment of industrial vision is completed, including: Perform a 3x3 convolution operation on the lightweight feature map, with a step size of 1 and a padding of 1 to obtain the classification branch and the regression branch; Among them, the classification branch outputs the flow mark category probability at each position, and the regression branch outputs the coordinate offset of the flow mark bounding box; The feature pyramid network structure is used to fuse feature maps of different scales to obtain the feature representation of multi-scale flow mark defects. Calculate the classification loss and regression loss based on the outputs of the classification branch and the regression branch; Sort the samples according to the total loss value, and select the top 20% of the samples as difficult samples; If the sample is a difficult example, it will be given a weight of 2 and its sampling frequency will be increased to 1.5 times the original in the next round of training; Post-process the detection results and select the detection box with the highest confidence through the non-maximum suppression algorithm; The IoU threshold of the non-maximum suppression algorithm is set to 0.5, and all detection boxes are traversed, the box with the highest score is retained, and other boxes with high overlap are suppressed.

Images