A method and system for detecting composite defects in images based on semantic segmentation

Abstract

The present invention relates to the field of screen detection technology, and discloses a method and system for detecting composite defects of images based on semantic segmentation, comprising the following steps: S1. constructing a semantic segmentation network model; S2. obtaining a first image set, and inputting the first image data set into the initialized semantic segmentation network model for pre-training; S3. setting the number of channels of the output layer of the pre-trained semantic segmentation network model according to the number of defect types that need to be detected; S4. obtaining a second image data set, and inputting the second image data set into the pre-trained semantic segmentation network model for training to obtain a semantic segmentation network model with composite defect detection capability; S5. inputting the image data to be detected into the semantic segmentation network model with composite defect detection capability to obtain a defect prediction map. The present invention solves the problem that the prior art relies on artificially designed defect recognition algorithms and can only detect single defects, and has the characteristics of high efficiency and small error.

Description

Technical Field

[0001] The present invention relates to the field of screen detection technology, and more specifically, to a method and system for detecting composite image defects based on semantic segmentation. Background Art

[0002] In the field of mobile phone production, LCD screens are used in large quantities and have high quality requirements. Therefore, it is necessary to perform defect detection on LCD screens to eliminate defective LCD screens. When testing LCD screens, it is necessary to meet both the speed and accuracy of LCD screen defect detection.

[0003] Existing technologies usually use deep learning semantic segmentation methods to detect the acquired LCD screen images. Semantic segmentation is an important branch of computer vision. Its main task is to label each pixel of the input image with the corresponding class based on the input image.

[0004] The application of deep learning in the field of semantic segmentation is reflected in the use of convolutional layers to up- and down-sample image features, which does not rely on pixel block methods. This method has a fixed receptive field for a single pixel and it is difficult to achieve a good level of detection speed and accuracy at the same time. Semantic segmentation mainly extracts low-dimensional features to high-dimensional features through down-sampling of convolutional layers, retains the mapping relationship between the position information of each pixel and the category, and classifies pixels according to the high-dimensional features corresponding to each pixel through up-sampling.

[0005] The main challenges of LCD screen defect detection are the texture background on the mobile phone screen, which has a negative impact on image preprocessing and the judgment of whether defects exist or not; there are many types of LCD screen defects, and the image features of each defect are significantly different, making it difficult to detect and classify all defects through a single algorithm; there are few defective samples of LCD screens, and the ratio of defective products to qualified products is seriously unbalanced. The resulting detection algorithm has low generalization ability and is prone to overfitting problems.

[0006] Traditional surface defect detection algorithms generally artificially design the recognition algorithm for each defect based on the defect feature information, generally using image filtering, morphological operations, image binarization processing, edge computing, etc. Traditional surface defect detection algorithms can generally adjust different parameters to achieve faster recognition of screen defects, but they have the disadvantages of requiring a separate recognition algorithm for each defect with different characteristics, poor generalization ability, high imaging requirements, and the need to adjust the algorithm parameters in a timely manner when background information changes to achieve the best detection effect.

[0007] According to some existing semantic segmentation models for defect detection, such as the classic U-Net model, it can effectively detect a single defect. However, when multiple defects exist, since the output of the U-Net model is continuous, we artificially mark multiple defects as discrete values. The same model has poor detection effect at the edge of the defect and the cross-overlapping area of ​​the defect, and more often a U-Net model is used for each defect. Therefore, using the U-Net model to detect multiple defects will result in slow calculation speed, high resource usage, and high hardware requirements for the environment. At the same time, the detection networks between each defect are independent of each other, and the composition of the data set also needs to ensure the number of each defect.

[0008] The prior art has a multi-level smartphone screen defect detection method, which includes the following steps: manually marking defects in an image, and then separating the foreground defect image and the background image; using a deep residual network and a feature pyramid network to perform multi-scale defect area feature extraction on a data set to obtain a multi-scale feature image; and extracting a target interest detection area.

[0009] However, the existing technology relies on artificially designed defect recognition algorithms when detecting defects, and can only detect single defects. Therefore, how to invent a screen surface composite defect detection method with small error, high efficiency, and no reliance on artificial design is an urgent problem to be solved in this technical field. Summary of the invention

[0010] In order to solve the problem in the prior art that only a single defect can be detected by relying on a manually designed defect recognition algorithm, the present invention provides a method for detecting composite defects in images based on semantic segmentation, which has the characteristics of high efficiency and small error.

[0011] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0012] A method for detecting composite defects in an image based on semantic segmentation comprises the following steps:

[0013] S1. Construct a semantic segmentation network model based on the classic DeepLab framework, where the number of channels in the output layer of the semantic segmentation network model is set to 1 for detecting a single defect;

[0014] S2. Obtain a first image set, wherein the first image data set includes a plurality of first images, wherein the number of defect types in the first image is 1, and input the first image data set into the initialized semantic segmentation network model for pre-training to obtain a pre-trained semantic segmentation network model;

[0015] S3. According to the number of defect types n to be detected, the number of channels of the output layer of the pre-trained semantic segmentation network model is set to n accordingly;

[0016] S4. Obtain a second image data set, wherein the second image data set includes a plurality of second images, and the number of defect types of the second images is n. The second image data set is input into the pre-trained semantic segmentation network model for training. After the training is completed, a semantic segmentation network model with a composite defect detection capability is obtained;

[0017] S5. Input the image data to be detected into the semantic segmentation network model with composite defect detection capability to perform defect detection and obtain a defect prediction map.

[0018] Preferably, the semantic segmentation network model includes an encoder that extracts high-dimensional features from low-dimensional features by downsampling and retains the position information of each pixel and the category mapping relationship; and a decoder that fuses low-dimensional features with high-dimensional features to improve the accuracy of segmentation boundaries.

[0019] Furthermore, the encoder includes a backbone network for extracting a first feature map from input image data and a spatial pyramid pooling module with a dilated convolution; the spatial pyramid pooling module is used to obtain and fuse information from different receptive fields, and extract the input first feature map to obtain a second feature map with multi-scale information.

[0020] Furthermore, the spatial pyramid pooling module includes 4 convolutional layers and 1 pooling layer; the 4 convolutional layers are respectively a 3×3 normal convolutional layer, a 3×3 dilated convolutional layer with a sparsity of 6, a 3×3 dilated convolutional layer with a sparsity of 12, and a 3×3 dilated convolutional layer with a sparsity of 18.

[0021] Furthermore, the decoder includes a 1×1 convolution layer for converting a plurality of feature maps of different scales into feature maps of the same scale, and a 3×3 convolution layer as an output layer;

[0022] The 1×1 convolution layer is used to transform the input image data, the first feature map, and the second feature map. Figure 3 Images of different scales are converted into feature maps of the same scale and superimposed;

[0023] The superimposed feature map is output through a 3×3 convolutional layer with a size of [output image length, output image width, number of defect types].

[0024] Furthermore, in step S1, the convolution with a kernel size of length, width of 3 and a number of channels greater than 16 in the semantic segmentation network model based on the classic DeepLab framework is replaced with a depthwise separable convolution;

[0025] The depth-wise separable convolution is specifically as follows: first, 3×3 convolution is performed on the feature map channel by channel, and then 1×1 convolution is performed across channels to achieve separate convolution of horizontal and vertical information;

[0026] The calculation parameter of the depth-separable convolution is: F ·W F ·D I ·(H C ·W C +D O );

[0027] Among them, H C Represents the length of the convolution kernel, W C Represents the width of the convolution kernel, H F Represents the length of the feature map, W F Represents the width of the feature map, D I Denotes the number of input channels of the depthwise separable convolution, D O Indicates the number of output channels of depthwise separable convolution.

[0028] Furthermore, in step S5, the second image data set is subjected to data enhancement and then input into the pre-trained semantic segmentation network model for training. The specific steps of the data enhancement are as follows:

[0029] B1. randomly dividing the images in the second image data set into a plurality of basic groups each consisting of 4 images;

[0030] B2. In each basic group, a center point is randomly selected as the stitching center of the four images;

[0031] B3. Stretch, scale, crop, rotate, and add noise to the four images in each base group;

[0032] B4. stitch together the four images in each base group to obtain an enhanced data map;

[0033] B5. Combine several enhanced data graphs into an enhanced data set, thereby completing data enhancement.

[0034] Furthermore, in step S5, the second image data set is input into the pre-trained semantic segmentation network model for training, as follows:

[0035] Set the number of iterative training times T, where T is a positive integer greater than 50;

[0036] In the first 50 iterations of training, the encoder in the semantic segmentation network model is frozen and only the decoder is trained;

[0037] After the first 50 iterations of training are completed, the semantic segmentation network model is iteratively trained as a whole;

[0038] During the training process, according to the set loss function, the error between the predicted image output by the semantic segmentation network model and the second image data set with defects is calculated, and the error is back-propagated to update the parameters of the semantic segmentation network model.

[0039] Furthermore, in step S5, during training, the model is updated by back propagation of the loss function, and the expression of the loss function is as follows:

[0040]

[0041] Among them, Loss is the loss value, λ recall is the recall regularization coefficient, λ class is the group regularization coefficient, It is the superposition of all defect areas in the prediction map output by the semantic segmentation deep learning model during training. is the superposition of all defect areas in the training image, represents the i-th defect map of the prediction map, represents the i-th defect image in the second image data set, and k is an integer constant.

[0042] A detection system for composite defects on screen surface based on semantic segmentation, comprising an image acquisition module for acquiring image data to be detected; a semantic segmentation deep learning module for detecting surface defects of screen images to be detected acquired by the acquisition module; and a training module for training the semantic segmentation deep learning module through diverse samples using common learning.

[0043] The beneficial effects of the present invention are as follows:

[0044] The semantic segmentation network model is pre-trained using a first image data set, where the number of defect types in the first image of the first image data set is 1. The pre-trained semantic segmentation network model is then retrained using a second image data set, where the number of defect types in the second image of the second image data set is n. The model is trained twice using different data sets, which achieves "common learning" between different tasks and solves the problem in the prior art that defect recognition algorithms rely on artificially designed algorithms and can only detect single defects. The algorithm has the characteristics of high efficiency and small error. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the method for detecting composite defects in images based on semantic segmentation.

[0046] Figure 2 It is a schematic diagram of the residual module used in the backbone network in Example 1.

[0047] Figure 3 Schematic diagram of the dilated convolution in the encoder’s pyramid module.

[0048] Figure 4 It is a schematic diagram of the data enhancement process.

[0049] Figure 5 It is a schematic diagram of the semantic segmentation deep learning model. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] like Figure 1 As shown, a method for detecting composite defects in an image based on semantic segmentation comprises the following steps:

[0053] S1. Construct a semantic segmentation network model based on the classic DeepLab framework, where the number of channels in the output layer of the semantic segmentation network model is set to 1 for detecting a single defect;

[0054] S2. Obtain a first image set, wherein the first image data set includes a plurality of first images, wherein the number of defect types in the first image is 1, and input the first image data set into the initialized semantic segmentation network model for pre-training to obtain a pre-trained semantic segmentation network model;

[0055] S3. According to the number of defect types n to be detected, the number of channels of the output layer of the pre-trained semantic segmentation network model is set to n accordingly;

[0056] S4. Obtain a second image data set, wherein the second image data set includes a plurality of second images, and the number of defect types of the second images is n. The second image data set is input into the pre-trained semantic segmentation network model for training. After the training is completed, a semantic segmentation network model with a composite defect detection capability is obtained;

[0057] S5. Input the image data to be detected into the semantic segmentation network model with composite defect detection capability to perform defect detection and obtain a defect prediction map.

[0058] like Figure 5 As shown, in a specific embodiment, the method for detecting composite defects in images based on semantic segmentation, the semantic segmentation network model includes an encoder that extracts high-dimensional features from low-dimensional features by downsampling and retains the position information and category mapping relationship of each pixel; and a decoder that fuses low-dimensional features with high-dimensional features to improve the accuracy of segmentation boundaries.

[0059] In this embodiment, the method for detecting composite image defects is applied to the complex surface of an LCD screen.

[0060] In a specific embodiment, the encoder includes a backbone network for extracting a first feature map from input image data and a spatial pyramid pooling module with a dilated convolution; the spatial pyramid pooling module is used to obtain and fuse information of different receptive fields, and extract the input first feature map to obtain a second feature map with multi-scale information.

[0061] In a specific embodiment, the spatial pyramid pooling module includes 4 convolution layers and 1 pooling layer; the 4 convolution layers are respectively a 3×3 normal convolution layer, a 3×3 dilated convolution layer with a sparsity of 6, a 3×3 dilated convolution layer with a sparsity of 12, and a 3×3 dilated convolution layer with a sparsity of 18.

[0062] In a specific embodiment, the decoder includes a 1×1 convolution layer for converting a plurality of feature maps of different scales into feature maps of the same scale, and a 3×3 convolution layer as an output layer;

[0063] The 1×1 convolution layer is used to transform the input image data, the first feature map, and the second feature map. Figure 3 Images of different scales are converted into feature maps of the same scale and superimposed;

[0064] The superimposed feature map is output through a 3×3 convolutional layer with a size of [output image length, output image width, number of defect types].

[0065] In this embodiment, the main function of the decoder is to improve the accuracy of the segmentation boundary. In the decoder, the feature maps of multiple sub-steps in the encoder are converted into feature maps of the same scale through 1×1 convolution, which are superimposed together and then output through a 3×3 convolution to output a feature map with dimensional information including the length of the input image, the width of the input image, and the number of defect types. The feature map is subsequently processed to obtain the corresponding prediction map for each defect.

[0066] like Figure 2 As shown, in this embodiment, the backbone network adopts the classification network ResNet50, which is provided with a residual learning module. ResNet50 solves the problem that the model performance decreases as the network deepens through residual learning. When the input is x, it is F(x) after passing through the convolution layer. After the input is connected through shortcut, the module output is F(x)+x. Through the inter-layer residual skip connection, the front information is introduced, which suppresses the gradient dissipation phenomenon caused by the error back propagation due to the excessive number of layers, and realizes more effective feature learning.

[0067] like Figure 3As shown, in this embodiment, the information of different receptive fields can be obtained and fused through the pyramid pooling module. First, information of different scales is obtained through 4 convolutional layers and 1 pooling layer, and then multiple feature maps of the same scale are superimposed. The four convolutional layers use 3×3 ordinary convolution, 3×3 dilated convolution with a sparsity of 6, 3×3 dilated convolution with a sparsity of 12, and 3×3 dilated convolution with a sparsity of 18. The sparsity of the dilated convolution is the interval between each point of the current convolution, which mainly controls the size of the receptive field. The larger the sparsity, the larger the receptive field. By improving the receptive field, the mapping area of ​​the current pixel on the input image is improved.

[0068] Example 2

[0069] like Figure 1 As shown, a method for detecting composite defects in an image based on semantic segmentation comprises the following steps:

[0070] S1. Construct a semantic segmentation network model based on the classic DeepLab framework, where the number of channels in the output layer of the semantic segmentation network model is set to 1 for detecting a single defect;

[0071] S2. Obtain a first image set, wherein the first image data set includes a plurality of first images, wherein the number of defect types in the first image is 1, and input the first image data set into the initialized semantic segmentation network model for pre-training to obtain a pre-trained semantic segmentation network model;

[0072] S3. According to the number of defect types n to be detected, the number of channels of the output layer of the pre-trained semantic segmentation network model is set to n accordingly;

[0073] S4. Obtain a second image data set, wherein the second image data set includes a plurality of second images, and the number of defect types of the second images is n. The second image data set is input into the pre-trained semantic segmentation network model for training. After the training is completed, a semantic segmentation network model with a composite defect detection capability is obtained;

[0074] S5. Input the image data to be detected into the semantic segmentation network model with composite defect detection capability to perform defect detection and obtain a defect prediction map.

[0075] like Figure 5 As shown, in a specific embodiment, the method for detecting composite defects in images based on semantic segmentation, the semantic segmentation network model includes an encoder that extracts high-dimensional features from low-dimensional features by downsampling and retains the position information and category mapping relationship of each pixel; and a decoder that fuses low-dimensional features with high-dimensional features to improve the accuracy of segmentation boundaries.

[0076] In a specific embodiment, the encoder includes a backbone network for extracting a first feature map from input image data and a spatial pyramid pooling module with a dilated convolution; the spatial pyramid pooling module is used to obtain and fuse information of different receptive fields, and extract the input first feature map to obtain a second feature map with multi-scale information.

[0077] In a specific embodiment, the spatial pyramid pooling module includes 4 convolution layers and 1 pooling layer; the 4 convolution layers are respectively a 3×3 normal convolution layer, a 3×3 dilated convolution layer with a sparsity of 6, a 3×3 dilated convolution layer with a sparsity of 12, and a 3×3 dilated convolution layer with a sparsity of 18.

[0078] In a specific embodiment, the decoder includes a 1×1 convolution layer for converting a plurality of feature maps of different scales into feature maps of the same scale, and a 3×3 convolution layer as an output layer;

[0079] The 1×1 convolution layer is used to transform the input image data, the first feature map, and the second feature map. Figure 3 Images of different scales are converted into feature maps of the same scale and superimposed;

[0080] The superimposed feature map is output through a 3×3 convolutional layer with a size of [output image length, output image width, number of defect types].

[0081] In a specific embodiment, in step S1, a convolution with a kernel size of length, width of 3 and a number of channels greater than 16 in a semantic segmentation network model based on a classic DeepLab framework is replaced with a depthwise separable convolution;

[0082] The depth-wise separable convolution is specifically as follows: first, 3×3 convolution is performed on the feature map channel by channel, and then 1×1 convolution is performed across channels to achieve separate convolution of horizontal and vertical information;

[0083] The calculation parameter of the depth-separable convolution is: F ·W F ·D I ·(H C ·W C +D O );

[0084] Among them, H C Represents the length of the convolution kernel, W C Represents the width of the convolution kernel, H F Represents the length of the feature map, W F Represents the width of the feature map, D I Denotes the number of input channels of the depthwise separable convolution, D O Indicates the number of output channels of depthwise separable convolution.

[0085] In this embodiment, the depth-wise separable convolution replaces the original 3×3 cross-channel convolution by depth-wise convolution and then point-wise convolution, so as to separate the horizontal and vertical information convolution. The length and width of the original convolution kernel are H C ,W C , the length and width of the feature map H F ,W F , the number of input and output channels are D I ,D O , the calculation parameter of normal convolution is H C ·W C ·H F ·W F ·D I ·D O ; The number of computational parameters for depthwise separable convolution is H F ·W F ·D I ·(H C ·W C +D O ), the calculated parameter quantity is the original The convolution kernel size replaced in the backbone network is 3×3 and the depth is 64 to 256, so the number of parameters is reduced to about the original

[0086] In this embodiment, a first image set is obtained, the first image data set includes several first images, the number of defect types of the first images is 1, and the first image data set of this embodiment adopts the AITEX fabric defect data set, specifically: the images in the AITEX fabric defect data set are used as training images, the initialized semantic segmentation deep learning model is input, the number of iterations is set, and the initialized semantic segmentation deep learning model is iteratively trained by using the images in the AITEX fabric defect data set as training images.

[0087] In this embodiment, the AITEX fabric defect data set has a large amount of data, and the obtained model can accurately complete image feature extraction, fuse low-dimensional features with high-dimensional features, and accurately segment boundaries.

[0088] Example 3

[0089] like Figure 1 As shown, a method for detecting composite defects in an image based on semantic segmentation comprises the following steps:

[0090] S1. Construct a semantic segmentation network model based on the classic DeepLab framework, where the number of channels in the output layer of the semantic segmentation network model is set to 1 for detecting a single defect;

[0091] S2. Initialize the semantic segmentation network model;

[0092] S3. Obtain a first image set, wherein the first image data set includes a plurality of first images, wherein the number of defect types in the first image is 1, and input the first image data set into the initialized semantic segmentation network model for pre-training to obtain a pre-trained semantic segmentation network model;

[0093] S4. According to the number of defect types n to be detected, the number of channels of the output layer of the pre-trained semantic segmentation network model is set to n accordingly;

[0094] S5. Obtain a second image data set, wherein the second image data set includes a plurality of second images, and the number of defect types of the second images is n. The second image data set is input into the pre-trained semantic segmentation network model for training. After the training is completed, a semantic segmentation network model with a composite defect detection capability is obtained;

[0095] S6. Input the image data to be detected into the semantic segmentation network model with composite defect detection capability to perform defect detection and obtain a defect prediction map.

[0096] like Figure 5 As shown, in a specific embodiment, the method for detecting composite defects in images based on semantic segmentation, the semantic segmentation network model includes an encoder that extracts high-dimensional features from low-dimensional features by downsampling and retains the position information and category mapping relationship of each pixel; and a decoder that fuses low-dimensional features with high-dimensional features to improve the accuracy of segmentation boundaries.

[0097] In a specific embodiment, the encoder includes a backbone network for extracting a first feature map from input image data and a spatial pyramid pooling module with a dilated convolution; the spatial pyramid pooling module is used to obtain and fuse information of different receptive fields, and extract the input first feature map to obtain a second feature map with multi-scale information.

[0098] In a specific embodiment, the spatial pyramid pooling module includes 4 convolution layers and 1 pooling layer; the 4 convolution layers are respectively a 3×3 normal convolution layer, a 3×3 dilated convolution layer with a sparsity of 6, a 3×3 dilated convolution layer with a sparsity of 12, and a 3×3 dilated convolution layer with a sparsity of 18.

[0099] In a specific embodiment, the decoder includes a 1×1 convolution layer for converting a plurality of feature maps of different scales into feature maps of the same scale, and a 3×3 convolution layer as an output layer;

[0100] The 1×1 convolution layer is used to transform the input image data, the first feature map, and the second feature map. Figure 3Images of different scales are converted into feature maps of the same scale and superimposed;

[0101] The superimposed feature map is output through a 3×3 convolutional layer with a size of [output image length, output image width, number of defect types].

[0102] In a specific embodiment, in step S1, a convolution with a kernel size of length, width of 3 and a number of channels greater than 16 in a semantic segmentation network model based on a classic DeepLab framework is replaced with a depthwise separable convolution;

[0103] The depth-wise separable convolution is specifically as follows: first, 3×3 convolution is performed on the feature map channel by channel, and then 1×1 convolution is performed across channels to achieve separate convolution of horizontal and vertical information;

[0104] The calculation parameter of the depth-separable convolution is: F ·W F ·D I ·(H C ·W C +D O );

[0105] Among them, H C Represents the length of the convolution kernel, W C Represents the width of the convolution kernel, H F Represents the length of the feature map, W F Represents the width of the feature map, D I Denotes the number of input channels of the depthwise separable convolution, D O Indicates the number of output channels of depthwise separable convolution.

[0106] like Figure 4 As shown, in a specific embodiment, in step S5, the second image data set is data enhanced and then input into the pre-trained semantic segmentation network model for training, and the data enhancement is specifically performed in the following steps:

[0107] B1. randomly dividing the images in the second image data set into a plurality of basic groups each consisting of 4 images;

[0108] B2. In each basic group, a center point is randomly selected as the stitching center of the four images;

[0109] B3. Stretch, scale, crop, rotate, and add noise to the four images in each base group;

[0110] B4. stitch together the four images in each base group to obtain an enhanced data map;

[0111] B5. Combine several enhanced data graphs into an enhanced data set, thereby completing data enhancement.

[0112] In this embodiment, the purpose of enhancing the data is to increase the amount of training data, improve the generalization ability of the model, increase noise data, and enhance the robustness of the model in complex surface backgrounds.

[0113] In a specific embodiment, in step S5, the second image data set is input into the pre-trained semantic segmentation network model for training, specifically as follows:

[0114] Set the number of iterative training times T, where T is a positive integer greater than 50;

[0115] In the first 50 iterations of training, the encoder in the semantic segmentation network model is frozen and only the decoder is trained;

[0116] After the first 50 iterations of training are completed, the semantic segmentation network model is iteratively trained as a whole;

[0117] During the training process, according to the set loss function, the error between the predicted image output by the semantic segmentation network model and the second image data set with defects is calculated, and the error is back-propagated to update the parameters of the semantic segmentation network model.

[0118] In this embodiment, after replacing the output layer of the model trained on the AITEX fabric defect dataset, the new model on the first image dataset is continued to be trained. In order to solve the problems of insufficient datasets and variable number of classifications in engineering problems, by continuing to train the new model based on the overall results of the pre-trained model, or selectively retraining only some parameters of the pre-trained model, the initial performance of the model can be higher and the convergence can be better.

[0119] In a specific embodiment, in step S5, during training, the model is updated by back propagation of the loss function, and the expression of the loss function is as follows:

[0120]

[0121] Among them, Loss is the loss value, λ recall is the recall regularization coefficient, λ class is the group regularization coefficient, It is the superposition of all defect areas in the prediction map output by the semantic segmentation deep learning model during training. is the superposition of all defect areas in the training image, represents the i-th defect map of the prediction map, represents the i-th defect image in the second image data set, and k is an integer constant.

[0122] In this embodiment, the detection error between the predicted defect mask output by the decoder and the annotated defect label is calculated according to the set defect detection loss function Loss during iterative training, and the error is back-propagated to update the model parameters. The loss function calculation can be divided into the defect detection rate part and the classification accuracy part. By additionally calculating the IOU of the defect area in the prediction and annotation, the defect detection effect of the semantic segmentation model is guaranteed.

[0123] In this embodiment, the surface composite defects of the screen image to be detected after the enhancement are detected by using the trained semantic segmentation deep learning model, and the specific steps are as follows:

[0124] C1. The screen image to be detected is input into the backbone network of the encoder to extract features;

[0125] C2. Input the extracted features into the pyramid pooling module, obtain features of five different scales through the four convolutional layers and one pooling layer of the pyramid pooling module, and superimpose the features of different scales to obtain multi-scale features;

[0126] C3. The screen image to be detected, the multi-scale features, and the features extracted by the backbone network are converted into three feature maps of the same scale through the three 1×1 convolutional layers of the decoder;

[0127] C4. Superimpose three feature maps of the same scale and pass them through the 3×3 convolution layer of the decoder to obtain the composite defect mask prediction map.

[0128] The present invention pre-trains a semantic segmentation network model through a first image data set, and the number of defect types of the first image of the first image data set is 1. The pre-trained semantic segmentation network model is then re-trained through a second image data set, and the number of defect types of the second image of the second image data set is n. The model is trained twice through different data sets to achieve "common learning" between different tasks; and the present invention also achieves the fusion of high-dimensional features through the cooperation of "encoder" and "decoder", and realizes multi-dimensional feature extraction through the backbone network and the pyramid network in the encoder, and improves the computational efficiency by using degree-separable convolution, thereby realizing the detection of composite defects on the screen surface through a deep learning model, solving the problem that the defect detection in the prior art relies on artificially designed defect recognition algorithms and can only detect single defects, and has the characteristics of high efficiency and small error.

[0129] Example 4

[0130] A detection system for composite defects on screen surface based on semantic segmentation, comprising an image acquisition module for acquiring image data to be detected; a semantic segmentation deep learning module for detecting surface defects of screen images to be detected acquired by the acquisition module; and a training module for training the semantic segmentation deep learning module through diverse samples using common learning.

[0131] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for detecting composite defects in images based on semantic segmentation, characterized in that: The following steps are involved: S1. Constructing a semantic segmentation network model based on the classic DeepLab framework, wherein the number of channels of the output layer of the semantic segmentation network model is set to 1 for detecting a single defect; S2. Obtaining a first image set, wherein the first image data set comprises a plurality of first images, wherein the number of defect types of the first image is 1, and inputting the first image data set into the initialized semantic segmentation network model for pre-training to obtain a pre-trained semantic segmentation network model; S3. According to the number n of defect types to be detected, the number of channels of the output layer of the pre-trained semantic segmentation network model is set to n accordingly; S4. Obtaining a second image data set, wherein the second image data set includes a plurality of second images, the number of defect types of the second image is n, and the second image data set is input into the pre-trained semantic segmentation network model for training. After the training is completed, a semantic segmentation network model having a composite defect detection capability is obtained; S5. Input the image data to be detected into the semantic segmentation network model with composite defect detection capability to perform defect detection and obtain a defect prediction map.

2. The method for detecting composite defects in images based on semantic segmentation according to claim 1, characterized in that: The semantic segmentation network model includes an encoder that extracts high-dimensional features from low-dimensional features through downsampling and retains the position information of each pixel and the category mapping relationship; and a decoder that is used to fuse low-dimensional features with high-dimensional features to improve the accuracy of segmentation boundaries.

3. The method for detecting composite defects in images based on semantic segmentation according to claim 2, characterized in that: The encoder includes a backbone network for extracting a first feature map from input image data and a spatial pyramid pooling module with a dilated convolution; the spatial pyramid pooling module is used to obtain and fuse information of different receptive fields, and extract the input first feature map to obtain a second feature map with multi-scale information.

4. The method for detecting composite defects in images based on semantic segmentation according to claim 3, characterized in that: The spatial pyramid pooling module includes 4 convolutional layers and 1 pooling layer; the 4 convolutional layers are respectively a 3×3 normal convolutional layer, a 3×3 dilated convolutional layer with a sparsity of 6, a 3×3 dilated convolutional layer with a sparsity of 12, and a 3×3 dilated convolutional layer with a sparsity of 18.

5. The method for detecting composite defects in images based on semantic segmentation according to claim 3, characterized in that: The decoder includes a 1×1 convolution layer for converting multiple feature maps of different scales into feature maps of the same scale, and a 3×3 convolution layer as an output layer; The 1×1 convolution layer is used to convert the input image data, the first feature map, and the second feature map into feature maps of the same scale and superimpose them. The superimposed feature map is output through a 3×3 convolutional layer with a size of [output image length, output image width, number of defect types].

6. The method for detecting composite defects in images based on semantic segmentation according to claim 1, characterized in that: In step S1, the convolution with a kernel size of length, width of 3 and a number of channels greater than 16 in the semantic segmentation network model based on the classic DeepLab framework is replaced with a depthwise separable convolution; The depth-wise separable convolution is specifically as follows: first, 3×3 convolution is performed on the feature map channel by channel, and then 1×1 convolution is performed across channels to achieve separate convolution of horizontal and vertical information; The calculation parameter quantity of the depth-separable convolution is: ; in, represents the length of the convolution kernel, represents the width of the convolution kernel, represents the length of the feature map, represents the width of the feature map, represents the number of input channels of the depthwise separable convolution, Indicates the number of output channels of depthwise separable convolution.

7. The method for detecting composite defects in images based on semantic segmentation according to claim 1, characterized in that: In step S4, the second image data set is subjected to data enhancement and then input into the pre-trained semantic segmentation network model for training. The data enhancement specifically comprises the following steps: B1. randomly dividing the images in the second image data set into a plurality of basic groups each consisting of 4 images; B2. In each basic group, a center point is randomly selected as the stitching center of the four images; B3. Stretch, scale, crop, rotate, and add noise to the four images in each base group; B4. Concatenate the four images in each basic group to obtain an enhanced data map; B5. Combine several enhanced data graphs into an enhanced data set, thereby completing data enhancement.

8. The method for detecting composite defects in images based on semantic segmentation according to claim 2, characterized in that: In step S4, the second image data set is input into the pre-trained semantic segmentation network model for training, as follows: Set the number of iterative training times T, where T is a positive integer greater than 50; In the first 50 iterations of training, the encoder in the semantic segmentation network model is frozen and only the decoder is trained; After the first 50 iterations of training are completed, the semantic segmentation network model is iteratively trained as a whole; During the training process, according to the set loss function, the error between the predicted image output by the semantic segmentation network model and the second image data set with defects is calculated, and the error is back-propagated to update the parameters of the semantic segmentation network model.

9. The method for detecting composite defects in images based on semantic segmentation according to claim 8, characterized in that: In step S4, during training, the model is updated by back propagation of the loss function, and the expression of the loss function is as follows: in, is the loss value, , is the group regularization coefficient, It is the superposition of all defect areas in the prediction map output by the semantic segmentation deep learning model during training. is the superposition of all defect areas in the training image, Represents the prediction graph Defect map, represents the first Defect map, is an integer constant.

10. A detection system for composite defects on screen surface based on semantic segmentation, used to implement the detection method for composite defects on image based on semantic segmentation as claimed in any one of claims 1 to 9, characterized in that: It includes an image acquisition module for acquiring image data to be detected; A semantic segmentation deep learning module for detecting surface defects of the screen image to be detected acquired by the acquisition module; A training module for training the semantic segmentation deep learning module using commonality learning and diverse samples.

Images