Composite defect detection method based on semantic segmentation and object detection fusion model

Abstract

The present invention provides a composite defect detection method based on a semantic segmentation and target detection fusion model, which performs composite defect image preprocessing on the original image to obtain multiple defect images, thereby obtaining the defect edge shape in the image and distinguishing the defect type of the image; according to the defect type of the image, the original image is preprocessed with a defect image and the processing result is input into the Unet semantic segmentation network to build and train, and a defect mask map of the corresponding image is obtained; the similarity between the defect mask map of the corresponding image and the mask label of the corresponding defect of the original image is calculated, and a superimposed image is obtained according to the similarity, and the superimposed image is input into the Faster R-CNN network for defect target detection, and the Faster R-CNN network detection result map is obtained to complete the detection of composite defects. The present invention proposes a composite defect detection method, which can effectively solve the defect detection problem of mobile phone screen images with overlapping defects of multiple types by separately detecting various defects in the original image.

Description

Technical Field

[0001] The present invention relates to the technical field of screen surface defect detection, and in particular to a composite defect detection method based on a semantic segmentation and target detection fusion model. Background Art

[0002] During the use of mobile phones, defects are inevitable on the screen surface. Existing mobile phone screen surface defects can be mainly divided into single-type defects, multiple-type defects and mixed defects according to the existence of defects. Among the three defect situations, the detection process of mixed defects is relatively difficult. Single-type defects refer to the existence of only one type of defects on a mobile phone screen image. In this case, the detection difficulty of the detection model is relatively low; multiple-type defects refer to the existence of multiple different types of defects on a mobile phone screen image, but there is no mixed overlap in these defect images. In this case, the detection difficulty of the detection model will be greater than that of a single defect detection; multiple types of defects overlap, usually manifested as a certain area on the mobile phone screen image with multiple defects overlapping and interlacing at the same time, and the form of the defects is difficult to judge. In this case, the detection difficulty of the detection model is the highest among the three defect existence situations, and it is difficult to successfully and accurately detect all defects in the image. In addition to the above-mentioned situations, the surface of the mobile phone screen may also have the following situations that lead to low detection accuracy of the detection model. For example: 1. The chromaticity value of the pixel points of some defects is relatively low, and the features are extremely unclear. 2. The sizes of various defects on mobile phone screens vary greatly, and the feature sizes extracted from various defects are not uniform, making it difficult to detect targets. 3. The division of positive and negative samples is not particularly reasonable, especially in some special cases. These conditions on mobile phone screens may make detection more difficult, reduce detection accuracy, and result in poor detection results.

[0003] According to some existing detection models, such as the classic target detection network Faster R-CNN, after multiple training iterations, it can achieve good detection effects for images of single-type defects and multiple-type defects. However, there is no very suitable detection model for the overlapping of multiple types of defects. For the overlapping of multiple types of defects, it is obvious that the detection ability of general detection models is insufficient. The main reasons are: First, when there are multiple types of defects overlapping in the same area, the target detection network may use the feature extraction network to extract features based on the most obvious defects among all the defects in an area, and then use the extracted features for subsequent detection tasks, and then perform classification prediction and position prediction. However, there are multiple types of defects in this area, so it leads to the problem of missed detection or the defect cannot be detected due to the mixing of defect features. Secondly, the pixel chromaticity value of the defects in the image of the mobile phone screen surface is low, which leads to the feature extraction network in the target detection network failing to extract the feature map in the image, making it difficult for the subsequent network of the target detection network to detect defects. Finally, the sizes of various types of defects vary greatly. If the features of the defects are used for subsequent target detection, the detection effect may be poor due to the differences in the sizes of the defects. Large defects need to be judged with deep features, while small defects need to be judged with shallow features. Summary of the invention

[0004] In order to solve at least one of the above technical defects, the present invention provides a composite defect detection method based on semantic segmentation and target detection fusion model, which can effectively solve the defect detection problem of mobile phone screen images with overlapping defects of multiple types.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] The composite defect detection method based on the semantic segmentation and target detection fusion model includes the following steps:

[0007] S1: Obtain an original image and use an image chromaticity value analysis method to calculate the number ratio of low chromaticity value pixels in the original image to obtain an original image with a chromaticity value in a normal range;

[0008] S2: performing composite defect image preprocessing on the original image with chromaticity values ​​in the normal range to obtain multiple defect images;

[0009] S3: Perform image edge detection on each defect image to obtain the defect edge shape in each image;

[0010] S4: performing shape judgment on each image according to the defect edge shape to distinguish the defect type of the image;

[0011] S5: performing defect image preprocessing on the original image according to the defect type of the image to obtain a corresponding preprocessed image;

[0012] S6: Build and train the Unet semantic segmentation network, and use the preprocessed image as the input of the Unet network to predict the corresponding image defects and obtain the defect mask map of the corresponding image;

[0013] S7: using a histogram similarity algorithm, calculating the similarity between the defect mask image of the corresponding image and the mask label of the corresponding defect of the original image, and obtaining the similarity of each defect;

[0014] S8: setting a corresponding similarity threshold for each defect, comparing the similarity of each defect with the similarity threshold of the corresponding defect, and if the similarity of the defect is greater than the set similarity threshold, superimposing the defect mask image corresponding to the similarity with the original image to obtain a superimposed image; if the similarity of the defect is less than or equal to the set similarity threshold, returning to step S5 to adjust the parameters of the defect image preprocessing of the defect type corresponding to the similarity;

[0015] S9: Input all the obtained superimposed images into the constructed Faster R-CNN network to perform defect target detection, obtain multiple detection result images, and display the detection results of the multiple detection result images together on the original image to obtain the Faster R-CNN network detection result image, and complete the detection of composite defects.

[0016] In the above scheme, the image chromaticity value analysis method is used to count the chromaticity value of each pixel of the entire original image, and perform corresponding subsequent image processing according to the distribution of the chromaticity value. In this scheme, if the chromaticity value distribution of the entire original image is generally low, the image will not be processed subsequently. If the chromaticity value distribution of the entire original image is relatively normal, the image will be processed normally.

[0017] In the above scheme, by separately detecting various defects in the original image, the defect detection problem of mobile phone screen images with overlapping types of defects can be effectively solved; at the same time, by applying multiple defect image preprocessing methods, the problem that the pixel chromaticity value of some defects is relatively low and the features are extremely unclear can be solved.

[0018] Wherein, in the step S2, the composite defect image preprocessing includes scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing and pinhole defect image preprocessing; wherein, the scratch defect image preprocessing can highlight the scratch defects in the original image to obtain an SP-1 defect image; the bubble defect image preprocessing can highlight the bubble defects in the original image to obtain a BP-1 defect image; the tin ash defect image preprocessing can highlight the tin ash defects in the original image to obtain a TP-1 defect image; the pinhole defect image preprocessing can highlight the image of the pinhole defects in the original image to obtain a PP-1 defect image.

[0019] In the above scheme, the methods of scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing and pinhole defect image preprocessing are all composed of a combination of multiple different image preprocessing methods, which can better highlight the corresponding defects of the image. The initial parameters of the corresponding methods here are all set to a more general value.

[0020] Wherein, in the step S3, an edge detection operator is used to perform image edge detection on each defect image, and the contour edge of the defect in the corresponding image is accurately measured and located to obtain the defect edge shape of each image.

[0021] Among them, in the step S4, the scratch shape judgment method is used to judge whether the edge shape in the SP-1 defect image is a scratch defect. If so, it is determined that the original image has a scratch defect and enters step S5 for processing; otherwise, the subsequent processing of the scratch defect is not performed on the original image; similarly, the bubble shape judgment method is used to judge whether the edge shape in the BP-1 defect image is a bubble defect, the tin ash shape judgment method is used to judge whether the edge shape in the TP-1 defect image is a tin ash defect, and the pinhole shape judgment method is used to judge whether the edge shape in the PP-1 defect image is a pinhole defect; at this point, the defect type of the original image is obtained.

[0022] In the above scheme, the scratch shape judgment method is to judge whether the defect edge shape obtained after the SP-1 defect image and image edge detection processing is consistent with the shape and area of ​​the real scratch, so as to determine whether the original image has a scratch defect; the judgment process of the bubble shape judgment method, the tin ash shape judgment method, and the pinhole shape judgment method is the same.

[0023] Wherein, in the step S5, the original image is subjected to defect image preprocessing according to the defect type determined in the step S4; wherein, if the original image has a scratch defect, the scratch defect image is preprocessed to obtain an SP-2 image; if the original image has a bubble defect, the bubble defect image is preprocessed to obtain a BP-2 image; if the original image has a tin ash defect, the tin ash defect image is preprocessed to obtain a TP-2 image; if the original image has a pinhole defect, the pinhole defect image is preprocessed to obtain a PP-2 image; the SP-2 image, BP-2 image, TP-2 image and PP-2 image are the corresponding preprocessed images.

[0024] In the above scheme, the methods used for scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing, and pinhole defect image preprocessing are basically the same as the composite defect image preprocessing described in step S2. The difference is that the initial parameters of the corresponding method here are set to a smaller value, with the purpose of returning the adjustment parameters in step S8.

[0025] Wherein, in the step S6, the Unet semantic segmentation network includes a feature extraction part and an enhanced feature extraction part; it first uses the feature extraction part to extract features of the input image to obtain a feature map of the image, and then the enhanced feature extraction part performs upsampling and multi-layer feature fusion operations on the obtained feature map to obtain a final fused feature map, and finally regresses the fused feature map to the classification of pixel points of the same size as the input image; the PP-2 image; the SP-2 image, BP-2 image, TP-2 image and PP-2 image are used as input images, and after prediction by the Unet semantic segmentation network, mask maps of different defects are obtained, and the SP-2 image, BP-2 image, TP-2 image, and PP-2 image correspond to scratch defect mask maps, bubble defect mask maps, tin ash defect mask maps and pinhole defect mask maps.

[0026] In the above scheme, when the SP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted scratch mask; when the BP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted bubble mask; when the TP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted tin ash mask; when the PP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted pinhole mask. Among them, the attention mechanism module is embedded in the trained Unet semantic segmentation network, which is used to weigh the importance of each feature layer for weight allocation, adjust the weight coefficients of different features, and facilitate the segmentation of a good mask map, which can improve the accuracy of the prediction.

[0027] Among them, in the step S8, the scratch mask predicted by the Unet semantic segmentation network only calculates the similarity with the scratch mask label of the original image; the bubble mask predicted by the Unet semantic segmentation network only calculates the similarity with the bubble mask label of the original image; the tin ash mask predicted by the Unet semantic segmentation network only calculates the similarity with the tin ash mask label of the original image; the pinhole mask predicted by the Unet semantic segmentation network only calculates the similarity with the pinhole mask label of the original image.

[0028] Wherein, in the step S9, the Faster R-CNN network includes a feature extraction network, a candidate region generation network (Region Proposal Networks), a ROI Pooling layer, and a classification and regression layer (Classification and Regression); wherein: the feature extraction network is used to extract the features of the image, the input is the superimposed image, and the output is the extracted feature map; the candidate region generation network is used to generate candidate regions and screen the candidate regions, the input is the original image, and the output is the screened candidate frame; the ROI Pooling layer is used to collect the extracted feature map and the screened candidate frame, synthesize them, extract the features corresponding to the candidate frame, and obtain a feature map of a fixed size; the classification and regression layer outputs the category to which the final candidate region belongs and the precise position of the candidate region in the image according to the obtained fixed-size feature map, and uses the detection classification probability (Softmax Loss) and the detection frame regression (Smooth L1 Loss) to jointly train the classification probability and the frame regression, so as to obtain the category of the candidate region, and at the same time obtain the final precise position of the prediction frame, that is, to obtain multiple detection result images; and the obtained multiple detection result images are displayed on the original image as the Faster R-CNN network detection result image, and the detection of composite defects is completed.

[0029] In the above scheme, the superimposed image obtained in the previous step is used as the input of the Faster R-CNN network. In the Faster R-CNN network: first, the feature extraction network is used to extract features from the superimposed image to obtain a feature map; then, the candidate region generation network is used according to the feature map to generate candidate frames and screen the generated candidate frames; then, the selected candidate frames and feature maps are comprehensively selected using the ROI Pooling layer to screen the remaining candidate frames and obtain a feature map of a fixed size; in the candidate frame generation process, the candidate frame size is not set manually, but the isodata self-iterative clustering algorithm is used to cluster the candidate frame size; finally, according to the obtained fixed-size feature map, it is input into two fully connected layers for classification prediction and position prediction, and the prediction frames of the prediction map of the obtained Faster R-CNN network are obtained, and then these prediction frames are displayed on the original image.

[0030] Among them, in the step S9, the candidate region generation network of the Faster R-CNN network uses the isodata self-iterative clustering algorithm to cluster the candidate frame sizes to generate corresponding candidate frames during the process of generating candidate frames.

[0031] In the above solution, the isodata self-iterative clustering algorithm is used to replace the manual setting of the candidate box size in the RPN part of the Faster R-CNN network, and the clustering algorithm is used to replace the manual setting to improve the accuracy and efficiency of the border regression. It can effectively solve the problem that the sizes of various defects on existing mobile phone screens vary greatly and the feature sizes extracted from various defects are not uniform.

[0032] The composite defect detection method based on the semantic segmentation and target detection fusion model further includes the following steps:

[0033] S10: Using the intersection-over-union and similarity synthesis method, calculations are performed based on the prediction boxes obtained by the Faster R-CNN network to evaluate the performance of the Faster R-CNN network model.

[0034] Among them, in the step S10, the intersection-over-union and similarity comprehensive method includes an intersection-over-union calculation part and a similarity calculation part using a twin neural network. Through the comprehensive calculation of these two parts, the number of true positive examples, false negative examples, false positive examples and true negative examples in the Faster R-CNN network model is detected, thereby realizing the evaluation of the performance of the Faster R-CNN network model; specifically:

[0035] The position, size and category information of the real frame are obtained through the xml file, and the position, size and category information of the predicted frame are obtained through step S9; the intersection and union ratio of the real frame and the predicted frame are calculated through the intersection and union ratio calculation part, and according to the set intersection and union ratio standard, if the intersection and union ratio is greater than the target, it is classified as a true positive example; all prediction frames with an intersection and union ratio between 0.3 and 0.5 are extracted, and the extracted prediction frames and their corresponding real frames are input into the twin neural network similarity calculation part to calculate the similarity. If the similarity between the two reaches 0.7, the prediction frame is determined to be a true positive example, otherwise, it is a false positive example; finally, the number of true positive examples, false negative examples, false positive examples and true negative examples is used to evaluate the performance of the Faster R-CNN network model.

[0036] The Siamese neural network (Siamese neural network), also known as the twin neural network, used in the above scheme is a coupling framework based on two artificial neural networks. The twin neural network takes two samples as input, and its two sub-networks each receive an input and output their representation embedded in a high-dimensional space. The distance between the two representations, such as the Euclidean distance, is calculated to compare the similarity between the two samples. In this scheme, the twin neural network is mainly used to compare the similarity between the prediction of the FasterR-CNN network and the corresponding real box. There are two sub-networks with the same structure and shared weights in the twin neural network, which receive two inputs X1 and X2 respectively, convert them into vectors Gw(X1) and Gw(X2), and then calculate the distance Ew of the two output vectors by some distance metric.

[0037] In the above scheme, by adopting the intersection-over-union and similarity comprehensive method, the problem that the existing division method of true positive examples, false negative examples, false positive examples and true negative examples is not particularly reasonable can be solved.

[0038] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0039] The present invention proposes a composite defect detection method based on a semantic segmentation and target detection fusion model. By separately detecting various defects in the original image, the defect detection problem of mobile phone screen images with overlapping types of defects can be effectively solved. At the same time, by applying multiple defect image preprocessing methods, the problem that the pixel chromaticity values ​​of some defects are relatively low and the features are extremely unclear can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0041] Figure 2 This is a flow chart of step S1 to step S4 in one embodiment of the present invention;

[0042] Figure 3 This is a flow chart of step S5 to step S9 in one embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the Unet semantic segmentation network structure in one embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the Faster R-CNN target detection network structure in one embodiment of the present invention;

[0045] Figure 6 Schematic diagram of the twin neural network structure in one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0047] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0048] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0049] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] Example 1

[0051] The existing mobile phone screen surface defect detection technology usually adopts some popular target detection networks such as YoLo V3 single-stage target detection network and Faster R-CNN two-stage target detection network. Ordinary target detection networks can achieve good results in detecting single-type defects and multiple-type defect images, but for overlapping images of multiple types of defects and problems in the detection process, ordinary detection networks cannot solve them well. Therefore, this paper proposes a composite defect detection method based on semantic segmentation and target detection fusion model, such as Figure 1 As shown. The following steps are included:

[0052] S1: Obtain an original image and use an image chromaticity value analysis method to calculate the number ratio of low chromaticity value pixels in the original image to obtain an original image with a chromaticity value in a normal range;

[0053] S2: performing composite defect image preprocessing on the original image with chromaticity values ​​in the normal range to obtain multiple defect images;

[0054] S3: Perform image edge detection on each defect image to obtain the defect edge shape in each image;

[0055] S4: performing shape judgment on each image according to the defect edge shape to distinguish the defect type of the image;

[0056] S5: performing defect image preprocessing on the original image according to the defect type of the image to obtain a corresponding preprocessed image;

[0057] S6: Build and train the Unet semantic segmentation network, and use the preprocessed image as the input of the Unet network to predict the corresponding image defects and obtain the defect mask map of the corresponding image;

[0058] S7: using a histogram similarity algorithm, calculating the similarity between the defect mask image of the corresponding image and the mask label of the corresponding defect of the original image, and obtaining the similarity of each defect;

[0059] S8: setting a corresponding similarity threshold for each defect, comparing the similarity of each defect with the similarity threshold of the corresponding defect, and if the similarity of the defect is greater than the set similarity threshold, superimposing the defect mask image corresponding to the similarity with the original image to obtain a superimposed image; if the similarity of the defect is less than or equal to the set similarity threshold, returning to step S5 to adjust the parameters of the defect image preprocessing of the defect type corresponding to the similarity;

[0060] S9: Input all the obtained superimposed images into the constructed Faster R-CNN network to perform defect target detection, obtain multiple detection result images, and display the detection results of the multiple detection result images together on the original image to obtain the Faster R-CNN network detection result image, and complete the detection of composite defects.

[0061] In the specific implementation process, the image chromaticity value analysis method is used to count the chromaticity value of each pixel of the entire original image, and perform corresponding subsequent image processing according to the distribution of the chromaticity value. In this solution, if the chromaticity value distribution of the entire original image is generally low, the image will not be processed subsequently. If the chromaticity value distribution of the entire original image is relatively normal, the image will be processed normally in the subsequent process.

[0062] In the specific implementation process, the present invention adopts a Faster R-CNN network, embeds a Unet semantic segmentation network, adds image chromaticity value analysis method, composite defect image preprocessing method, defect image preprocessing method, histogram similarity algorithm, etc., to form an enhanced hybrid model, which realizes the detection of composite defects on mobile phone screens. In this hybrid model, the detection problem of mobile phone screen images with overlapping defects of various types can be effectively solved; secondly, the situation where the pixel chromaticity value of some defects is relatively low and the features are extremely unclear can also be solved well.

[0063] Example 2

[0064] More specifically, based on Example 1, Figure 2 As shown, Figure 2 A schematic diagram showing the flow of steps S1 to S4.

[0065] Wherein, in the step S2, the composite defect image preprocessing includes scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing and pinhole defect image preprocessing; wherein, the scratch defect image preprocessing can highlight the scratch defects in the original image to obtain an SP-1 defect image; the bubble defect image preprocessing can highlight the bubble defects in the original image to obtain a BP-1 defect image; the tin ash defect image preprocessing can highlight the tin ash defects in the original image to obtain a TP-1 defect image; the pinhole defect image preprocessing can highlight the image of the pinhole defects in the original image to obtain a PP-1 defect image.

[0066] In the specific implementation process, the scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing and pinhole defect image preprocessing methods are all composed of a combination of multiple different image preprocessing methods, which can better highlight the corresponding defects of the image. The initial parameters of the corresponding methods here are all set to a more general value.

[0067] More specifically, in step S3, an edge detection operator is used to perform image edge detection on each defect image, and the contour edge of the defect in the corresponding image is accurately measured and located to obtain the defect edge shape of each image.

[0068] More specifically, in step S4, a scratch shape judgment method is used to judge whether the edge shape in the SP-1 defect image is a scratch defect. If so, it is determined that the original image has a scratch defect and the process proceeds to step S5; otherwise, no subsequent processing of the scratch defect is performed on the original image; similarly, a bubble shape judgment method is used to judge whether the edge shape in the BP-1 defect image is a bubble defect, a tin ash shape judgment method is used to judge whether the edge shape in the TP-1 defect image is a tin ash defect, and a pinhole shape judgment method is used to judge whether the edge shape in the PP-1 defect image is a pinhole defect; at this point, the defect type of the original image is obtained.

[0069] In the specific implementation process, the scratch shape judgment method is to judge whether the defect edge shape obtained after the SP-1 defect image and image edge detection processing is consistent with the shape and area of ​​the real scratch, so as to determine whether the original image has a scratch defect; the judgment process of the bubble shape judgment method, the tin ash shape judgment method, and the pinhole shape judgment method is the same.

[0070] More specifically, Figure 3 As shown, Figure 3 A schematic diagram showing the flow of steps S5 to S9.

[0071] Wherein, in the step S5, the original image is subjected to defect image preprocessing according to the defect type determined in the step S4; wherein, if the original image has a scratch defect, the scratch defect image is preprocessed to obtain an SP-2 image; if the original image has a bubble defect, the bubble defect image is preprocessed to obtain a BP-2 image; if the original image has a tin ash defect, the tin ash defect image is preprocessed to obtain a TP-2 image; if the original image has a pinhole defect, the pinhole defect image is preprocessed to obtain a PP-2 image; the SP-2 image, BP-2 image, TP-2 image and PP-2 image are the corresponding preprocessed images.

[0072] In the specific implementation process, the methods used for the scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing, and pinhole defect image preprocessing are basically the same as the composite defect image preprocessing described in step S2. The difference is that the initial parameters of the corresponding method here are set to a smaller value, with the purpose of returning the adjustment parameters in step S8.

[0073] More specifically, in step S6, Figure 4 As shown, the Unet semantic segmentation network includes a feature extraction part and an enhanced feature extraction part; it first uses the feature extraction part to extract features of the input image to obtain a feature map of the image, and then the enhanced feature extraction part performs upsampling and multi-layer feature fusion operations on the obtained feature map to obtain the final fused feature map, and finally regresses the fused feature map to the classification of pixel points of the same size as the input image; the PP-2 image; the SP-2 image, BP-2 image, TP-2 image and PP-2 image are used as input images, and after prediction by the Unet semantic segmentation network, mask maps of different defects are obtained, and the SP-2 image, BP-2 image, TP-2 image, and PP-2 image correspondingly obtain scratch defect mask maps, bubble defect mask maps, tin ash defect mask maps and pinhole defect mask maps.

[0074] In the specific implementation process, when the SP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted scratch mask; when the BP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted bubble mask; when the TP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted tin ash mask; when the PP-2 image is predicted by the Unet semantic segmentation network, the prediction result only needs to display the predicted pinhole mask. Among them, the attention mechanism module is embedded in the trained Unet semantic segmentation network, which is used to weigh the importance of each feature layer for weight distribution, adjust the weight coefficients of different features, and facilitate the segmentation of a good mask map, which can improve the accuracy of the prediction.

[0075] In the specific implementation process, the attention mechanism is usually an additional neural network that can rigidly select certain parts of the input or assign different weights to different parts of the input. The attention mechanism can filter out important information from a large amount of information. SE-net is used in this solution. The importance of each feature channel is obtained through automatic learning (implemented by another new neural network), and then this importance is used to assign a weight value to each feature channel, so that the neural network focuses on certain feature channels, that is, to enhance the feature channels that are useful for the current task and suppress the feature channels that are not used for the current task.

[0076] More specifically, in step S8, the scratch mask predicted by the Unet semantic segmentation network only calculates the similarity with the scratch mask label of the original image; the bubble mask predicted by the Unet semantic segmentation network only calculates the similarity with the bubble mask label of the original image; the tin ash mask predicted by the Unet semantic segmentation network only calculates the similarity with the tin ash mask label of the original image; the pinhole mask predicted by the Unet semantic segmentation network only calculates the similarity with the pinhole mask label of the original image.

[0077] More specifically, in step S9, Figure 5 As shown, the Faster R-CNN network includes a feature extraction network, a candidate region generation network (Region Proposal Networks), a ROI Pooling layer, and a classification and regression layer (Classification and Regression); wherein: the feature extraction network is used to extract the features of the image, the input is the superimposed image, and the output is the extracted feature map; the candidate region generation network is used to generate candidate regions and screen the candidate regions, the input is the original image, and the output is the screened candidate frame; the ROI Pooling layer is used to collect the extracted feature map and the screened candidate frame, synthesize them, extract the features corresponding to the candidate frame, and obtain a feature map of a fixed size; the classification and regression layer outputs the category to which the final candidate region belongs and the precise position of the candidate region in the image according to the obtained fixed-size feature map, and uses the detection classification probability (Softmax Loss) and the detection frame regression (Smooth L1 Loss) to jointly train the classification probability and the frame regression, so as to obtain the category of the candidate region, and at the same time obtain the final precise position of the prediction frame, that is, to obtain multiple detection result images; and the obtained multiple detection result images are displayed on the original image as the Faster R-CNN network detection result image to complete the detection of composite defects.

[0078] In the specific implementation process, the superimposed image obtained in the previous step is used as the input of the Faster R-CNN network. In the Faster R-CNN network: first, the feature extraction network is used to extract features from the superimposed image to obtain a feature map; then, the candidate region generation network is used according to the feature map to generate candidate frames and screen the generated candidate frames; then, the selected candidate frames and feature maps are comprehensively selected using the ROI Pooling layer to screen the remaining candidate frames and obtain a feature map of a fixed size; in the candidate frame generation process, the candidate frame size is not set manually, but the isodata self-iterative clustering algorithm is used to cluster the candidate frame size; finally, according to the obtained fixed-size feature map, it is input into two fully connected layers for classification prediction and position prediction, and the prediction frames of the prediction map of the obtained Faster R-CNN network are obtained, and then these prediction frames are displayed on the original image.

[0079] More specifically, in step S9, the candidate region generation network of the Faster R-CNN network uses the isodata self-iterative clustering algorithm to cluster the candidate frame sizes to generate corresponding candidate frames during the process of generating candidate frames.

[0080] In the specific implementation process, the isodata self-iterative clustering algorithm is used to replace the manual setting of the candidate box size in the RPN part of the Faster R-CNN network, and the clustering algorithm is used to replace the manual setting to improve the accuracy and efficiency of the border regression. It can effectively solve the problem that the sizes of various defects on existing mobile phone screens vary greatly and the feature sizes extracted from various defects are not uniform.

[0081] Example 3

[0082] More specifically, the composite defect detection method based on the semantic segmentation and target detection fusion model further includes the following steps:

[0083] S10: Using the intersection-over-union and similarity synthesis method, calculations are performed based on the prediction boxes obtained by the Faster R-CNN network to evaluate the performance of the Faster R-CNN network model.

[0084] More specifically, in step S10, the intersection-over-union and similarity comprehensive method includes an intersection-over-union calculation part and a similarity calculation part using a twin neural network. Through the comprehensive calculation of these two parts, the number of true positive examples, false negative examples, false positive examples, and true negative examples in the Faster R-CNN network model is detected, thereby realizing the evaluation of the performance of the Faster R-CNN network model; specifically:

[0085] The position, size and category information of the real frame are obtained through the xml file, and the position, size and category information of the predicted frame are obtained through step S9; the intersection and union ratio of the real frame and the predicted frame are calculated through the intersection and union ratio calculation part, and according to the set intersection and union ratio standard, if the intersection and union ratio is greater than the target, it is classified as a true positive example; all prediction frames with an intersection and union ratio between 0.3 and 0.5 are extracted, and the extracted prediction frames and their corresponding real frames are input into the twin neural network similarity calculation part to calculate the similarity. If the similarity between the two reaches 0.7, the prediction frame is determined to be a true positive example, otherwise, it is a false positive example; finally, the number of true positive examples, false negative examples, false positive examples and true negative examples is used to evaluate the performance of the Faster R-CNN network model.

[0086] The Siamese neural network (also known as the twin neural network) used in the above scheme is a coupling framework based on two artificial neural networks. Its structure is as follows: Figure 6 As shown. The twin neural network takes two samples as input, and its two sub-networks each receive an input and output their representations embedded in a high-dimensional space. The distance between the two representations, such as the Euclidean distance, is calculated to compare the similarity between the two samples. In this solution, the twin neural network is mainly used to compare the similarity between the prediction of the Faster R-CNN network and the corresponding real box. There are two sub-networks with the same structure and shared weights in the twin neural network, which receive two inputs X1 and X2 respectively, convert them into vectors Gw(X1) and Gw(X2), and then calculate the distance Ew between the two output vectors by some distance metric.

[0087] In the specific implementation process, by adopting the intersection-over-union and similarity comprehensive method, the problem that the existing division method of true positive examples, false negative examples, false positive examples and true negative examples is not particularly reasonable can be solved.

[0088] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A composite defect detection method based on a semantic segmentation and target detection fusion model, characterized in that: The following steps are involved: S1: Obtain an original image and use an image chromaticity value analysis method to calculate the number ratio of low chromaticity value pixels in the original image to obtain an original image with a chromaticity value in a normal range; S2: performing composite defect image preprocessing on the original image with chromaticity values ​​in the normal range to obtain multiple defect images; S3: Perform image edge detection on each defect image to obtain the defect edge shape in each image; S4: performing shape judgment on each image according to the defect edge shape to distinguish the defect type of the image; S5: performing defect image preprocessing on the original image according to the defect type of the image to obtain a corresponding preprocessed image; S6: Build and train the Unet semantic segmentation network, and use the preprocessed image as the input of the Unet network to predict the corresponding image defects and obtain the defect mask map of the corresponding image; S7: using a histogram similarity algorithm, calculating the similarity between the defect mask image of the corresponding image and the mask label of the corresponding defect of the original image, and obtaining the similarity of each defect; S8: setting a corresponding similarity threshold for each defect, comparing the similarity of each defect with the similarity threshold of the corresponding defect, and if the similarity of the defect is greater than the set similarity threshold, superimposing the defect mask image corresponding to the similarity with the original image to obtain a superimposed image; if the similarity of the defect is less than or equal to the set similarity threshold, returning to step S5 to adjust the parameters of the defect image preprocessing of the defect type corresponding to the similarity; S9: Input all the obtained superimposed images into the constructed Faster R-CNN network to perform defect target detection, obtain multiple detection result images, and display the detection results of the multiple detection result images together on the original image to obtain the Faster R-CNN network detection result image, and complete the detection of composite defects.

2. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 1 is characterized in that: In the step S2, the composite defect image preprocessing includes scratch defect image preprocessing, bubble defect image preprocessing, tin ash defect image preprocessing and pinhole defect image preprocessing; wherein the scratch defect image preprocessing can highlight the scratch defects in the original image to obtain an SP-1 defect image; the bubble defect image preprocessing can highlight the bubble defects in the original image to obtain a BP-1 defect image; the tin ash defect image preprocessing can highlight the tin ash defects in the original image to obtain a TP-1 defect image; the pinhole defect image preprocessing can highlight the image of the pinhole defects in the original image to obtain a PP-1 defect image.

3. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 2 is characterized in that: In step S3, edge detection is performed on each defect image using an edge detection operator, the contour edge of the defect in the corresponding image is accurately measured and located, and the defect edge shape of each image is obtained.

4. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 3 is characterized in that: In step S4, a scratch shape determination method is used to determine whether the edge shape in the SP-1 defect image is a scratch defect. If so, it is determined that the original image has a scratch defect, and the process proceeds to step S5; Otherwise, no subsequent processing of the scratch defect is performed on the original image; similarly, the bubble shape judgment method is used to determine whether the edge shape in the BP-1 defect image is a bubble defect, the tin ash shape judgment method is used to determine whether the edge shape in the TP-1 defect image is a tin ash defect, and the pinhole shape judgment method is used to determine whether the edge shape in the PP-1 defect image is a pinhole defect; at this point, the defect type of the original image is obtained.

5. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 4 is characterized in that: In step S5, defect image preprocessing is performed on the original image according to the defect type determined in step S4; if the original image has a scratch defect, scratch defect image preprocessing is performed on it to obtain an SP-2 image; if the original image has a bubble defect, bubble defect image preprocessing is performed on it to obtain a BP-2 image; if the original image has a tin ash defect, tin ash defect image preprocessing is performed on it to obtain a TP-2 image; if the original image has a pinhole defect, pinhole defect image preprocessing is performed on it to obtain a PP-2 image; the SP-2 image, BP-2 image, TP-2 image and PP-2 image are the corresponding preprocessed images.

6. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 5 is characterized in that: In the step S6, the Unet semantic segmentation network includes a feature extraction part and an enhanced feature extraction part; First, the feature extraction part extracts features from the input image to obtain the feature map of the image. Then, the enhanced feature extraction part performs upsampling and multi-layer feature fusion operations on the obtained feature map to obtain the final fused feature map. Finally, the fused feature map is regressed to the classification of pixels of the same size as the input image. The PP-2 image; The SP-2 image, BP-2 image, TP-2 image and PP-2 image are used as input images, and after prediction by the Unet semantic segmentation network, mask images of different defects are obtained. The SP-2 image, BP-2 image, TP-2 image, and PP-2 image correspondingly obtain scratch defect mask images, bubble defect mask images, tin ash defect mask images, and pinhole defect mask images.

7. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 6 is characterized in that: In step S9, the Faster R-CNN network includes a feature extraction network, a candidate region generation network, a ROI Pooling layer, and a classification and regression layer; wherein: the feature extraction network is used to extract features of the image, the input is a superimposed image, and the output is an extracted feature map; the candidate region generation network is used to generate candidate regions and screen the candidate regions, the input is an original image, and the output is a screened candidate frame; the ROI Pooling layer is used to collect the extracted feature maps and the screened candidate frames, synthesize them, extract features corresponding to the candidate frames, and obtain a feature map of a fixed size; the classification and regression layer outputs the category to which the final candidate region belongs and the precise position of the candidate region in the image according to the obtained feature map of a fixed size, and uses the detection classification probability and the detection frame regression to jointly train the classification probability and the frame regression, so as to obtain the category of the candidate region, and at the same time obtain the final precise position of the prediction frame, that is, to obtain multiple detection result images; and the obtained multiple detection result images are displayed on the original image as the Faster R-CNN network detection result image to complete the detection of composite defects.

8. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 7 is characterized in that: In step S9, the candidate region generation network of the Faster R-CNN network uses the isodata self-iterative clustering algorithm to cluster the candidate frame sizes and generate corresponding candidate frames during the process of generating candidate frames.

9. The composite defect detection method based on the semantic segmentation and target detection fusion model according to any one of claims 1 to 8, characterized in that: The following steps are also included: S10: The performance of the Faster R-CNN network model is evaluated by using the intersection-over-union and similarity combined method based on the prediction box obtained by the Faster R-CNN network.

10. The composite defect detection method based on semantic segmentation and target detection fusion model according to claim 9, characterized in that: In step S10, the intersection-over-union and similarity comprehensive method includes an intersection-over-union calculation part and a similarity calculation part using a twin neural network. Through the comprehensive calculation of these two parts, the number of true positive examples, false negative examples, false positive examples, and true negative examples in the Faster R-CNN network model is detected, thereby realizing the evaluation of the performance of the Faster R-CNN network model; specifically: The position, size and category information of the real frame are obtained through the xml file, and the position, size and category information of the predicted frame are obtained through step S9; the intersection and union ratio of the real frame and the predicted frame are calculated through the intersection and union ratio calculation part, and according to the set intersection and union ratio standard, if the intersection and union ratio is greater than the target, it is classified as a true positive example; all prediction frames with an intersection and union ratio between 0.3 and 0.5 are extracted, and the extracted prediction frames and their corresponding real frames are input into the twin neural network similarity calculation part to calculate the similarity. If the similarity between the two reaches 0.7, the prediction frame is determined to be a true positive example, otherwise, it is a false positive example; finally, the number of true positive examples, false negative examples, false positive examples and true negative examples is used to evaluate the performance of the Faster R-CNN network model.

Images