A method for detecting defects of a digital tube chip

Abstract

The present application relates to a kind of defect detection methods of nixie tube chip, comprising: obtaining source domain nixie tube chip image dataset;YOLO V5 model is trained according to source domain nixie tube chip image dataset, and first defect detection model is obtained;Target domain nixie tube chip image dataset is obtained, and first defect detection model is transferred to target domain based on transfer learning;Part of target domain nixie tube chip image is labeled using labelImg;The first defect detection model is pre-trained to obtain second defect detection model using the labeled target domain nixie tube chip image;Using the unlabelled target domain nixie tube chip image set based on semi-supervised method, second defect detection model is retrained, and the final defect detection model is obtained;The defect of target domain to be measured nixie tube chip image is detected by the final defect detection model, and the class and the position of the defect in target domain to be measured chip are obtained.

Description

Technical Field

[0001] This invention belongs to the field of digital tube chip defect detection, and specifically relates to a method for detecting defects in digital tube chips. Background Technology

[0002] Digital tube displays are widely used semiconductor display devices. Their production process can lead to a series of appearance defects that affect product quality due to various issues. With continuous advancements in production technology and increasing automation, traditional manual and automated inspection methods are gradually becoming insufficient to meet the needs of the digital tube manufacturing industry. Machine vision-based product inspection technology, with its advantages of long continuous working time and fast judgment speed, is an ideal alternative to manual inspection.

[0003] Machine vision-based product inspection technology mainly trains a neural network-based defect detection model using a large amount of product image data. The trained defect detection model extracts features from the product images. To ensure the accuracy and high reliability of the trained defect detection model, there must be enough usable training samples to learn a good defect detection model. In actual production processes, due to different production environments and conditions, the collection and labeling of training data requires a lot of manpower, material resources, financial resources, and time. Moreover, the learning of a large number of features also has disadvantages such as long learning time and poor robustness. Summary of the Invention

[0004] To address the problems existing in the background art, the present invention provides a defect detection method for digital tube chips, comprising:

[0005] S1: Obtain the source domain digital tube chip image dataset, which includes images of various digital tube chips in different orientations;

[0006] S2: Use labelImg to annotate the source domain digital tube chip image to generate corresponding defect category labels and defect location labels;

[0007] S3: Train the YOLO V5 model based on the source domain digital tube chip image dataset to obtain the first defect detection model;

[0008] S4: Obtain the target domain digital tube chip image dataset, and transfer the first defect detection model to the target domain based on transfer learning;

[0009] S5: Use labelImg to annotate part of the target domain digital tube chip image to obtain the defect category label and defect location label of the target domain digital tube chip image;

[0010] S6: Use the labeled target domain digital tube chip image to pre-train the first defect detection model to obtain the second defect detection model;

[0011] S7: The second defect detection model is retrained using an unlabeled target domain digital tube chip image set based on a semi-supervised method to obtain the final defect detection model; the final defect detection model is used to detect defects in the target domain digital tube chip image to obtain the type and location of defects in the target domain chip.

[0012] Preferably, the YOLO V5 model includes: an input terminal, a Backbone network, a Neck network, and a Prediction network.

[0013] Preferably, training the YOLO V5 model based on the source domain digital tube chip image dataset includes:

[0014] S31: Mosaic is used for data augmentation at the input of the YOLO V5 model. Four source domain digital tube chip images are randomly sampled from the source domain digital tube chip image dataset. The four source domain digital tube chip images and their corresponding defect location labels are randomly scaled and then randomly distributed and stitched together to obtain the first source domain digital tube chip augmented image and its corresponding defect location labels.

[0015] S32: Input the enhanced image of the first source domain digital tube chip into the YOLO V5 model, and obtain three feature maps of different scales through the Backbone network and Neck network;

[0016] S33: Input the feature maps of three different scales into the three prediction heads of the Prediction network to predict the type and location of defects in the enhanced image of the first source domain digital tube chip;

[0017] S34: Construct a binary cross-entropy loss based on the category prediction results of defects in the enhanced image of the first source domain digital tube chip;

[0018] S35: Construct the CIoU-Loss loss function based on the location prediction results of defects in the enhanced image of the first source domain digital tube chip;

[0019] S36: Construct the loss function of the YOLO V5 model based on the binary cross-entropy loss and CIoU-Loss loss function, and update the parameters of the YOLO V5 model to obtain the first defect detection model.

[0020] Preferably, the binary cross-entropy loss includes:

[0021] Loss1=-(y·log(p)+(1-y)·log(1-p))

[0022] Where loss1 represents the binary cross-entropy loss, y represents the category label of the defect in the enhanced image of the first source domain digital tube chip, and p represents the category prediction result of the defect in the enhanced image of the first source domain digital tube chip.

[0023] Preferably, the CIoU-Loss loss function includes:

[0024]

[0025]

[0026] Where A is the predicted location of the defect in the enhanced image of the first source domain digital tube chip, B is the location label of the defect in the enhanced image of the first source domain digital tube chip, and C is the minimum convex set of the predicted box and the actual box.

[0027] Preferably, the semi-supervised method for training the second defect detection model using unlabeled target domain digital tube chip images includes:

[0028] S71: At the input end of the second defect detection model, Mosaic is used for data augmentation. Four target domain digital tube chip images are randomly sampled from the unlabeled target domain digital tube chip images. The four sampled target domain digital tube chip images are randomly scaled and then randomly distributed and stitched together to obtain the first target domain digital tube chip augmented image.

[0029] S72: Randomly crop two image blocks x with overlapping areas from the image of the first target domain digital tube chip. u1 and x u2 ;

[0030] S73: Move image patch x u1 and x u2 The images are input into the second defect detection model and processed through the Backbone network and Neck network to obtain image patch x. u1 The corresponding first intermediate feature maps and image patches x at three different scales u2 The corresponding second intermediate feature maps at three different scales;

[0031] S74: Move image patch x u1 The first comprehensive feature map is obtained by concatenating the first intermediate feature maps of the three different scales using the Concat function; the image patch x u2 The three corresponding second intermediate feature maps at different scales are concatenated using the Concat function to obtain the second comprehensive feature map;

[0032] S75: Upsample the first comprehensive feature map and the second comprehensive feature map to the same size as the image of the first target domain digital tube chip to obtain the first comprehensive reconstructed feature map and the second comprehensive reconstructed feature map;

[0033] S76: Based on the feature representations of the overlapping regions in the first and second comprehensive reconstruction feature maps, a directional contrast loss function is constructed to update the parameters of the second defect detection model, thus obtaining the final defect detection model.

[0034] Preferably, the directional contrast loss function includes:

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] in, The loss direction is represented by C(), the classifier is represented by γ, the threshold for filtering low-confidence positive samples is represented by N, the number of sample points in the overlapping region is represented by h and w, the length and width of the overlapping region are represented by h and w, and the set of pixel features in the non-overlapping region is represented by Fu. and Represents image patch x u1 and x u2 The overlapping region in β u1 =β(f u1 ) and β u2 =β(f u2 In the feature representation, β represents the nonlinear mapping function, f u1 and f u2 This represents the first and second comprehensive reconstructed feature maps. and The pseudo-label represents the pixel features of the overlapping region in the first and second comprehensive reconstructed feature maps, y n Let represent the pseudo-labels of non-overlapping pixel features in the first and second comprehensive reconstructed feature maps, r() represent the similarity function, max represent the maximum value function, and B represent the training batch size. This represents the directional contrast loss function.

[0043] The present invention has at least the following beneficial effects

[0044] This invention presents a first defect detection model trained on a source domain digital tube chip image dataset based on the idea of ​​transfer learning. This first defect detection model can be transferred to different hardware devices. A small amount of target domain digital tube chip image dataset is obtained for the application scenario of the corresponding device. The parameters of the first defect detection model are fine-tuned using labeled target domain digital tube chip images to obtain a second defect detection model, making it applicable to the target domain application scenario. Simultaneously, this invention randomly crops two patches from an unlabeled image and restricts them to an overlapping region. This overlapping region is placed in two different environments. In this case, even if the ground truth is unknown, the pixel-level one-to-one correspondence between the overlapping regions of the two patches maintains the consistency of high-level features in different environments. Furthermore, to achieve consistency, a directional contrastive loss function is used to align low-quality features with high-quality features. This invention can effectively learn from unlabeled images, reducing the workload of labeling data and enhancing the robustness of the defect detection model. Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0046] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0047] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0048] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0049] Please see Figure 1 This invention provides a defect detection method for digital tube chips, comprising:

[0050] S1: Obtain the source domain digital tube chip image dataset, which includes images of various digital tube chips in different orientations;

[0051] In this embodiment, various digital tube chip images are downloaded through the API interface provided by the social network platform. By adding interference and noise to the downloaded digital tube chip images, the pixels in some areas of the digital tube chip images are changed to simulate different types of defects in the digital tube chips, thus creating a source domain digital tube chip image dataset.

[0052] S2: Use labelImg to annotate the source domain digital tube chip image to generate corresponding defect category labels and defect location labels;

[0053] In this embodiment, the defect categories of the digital tube chip include: dotted glue overflow, large areas of glue overflow, glue sticking to pins or misaligned pins, etc. The specific defect categories can be defined and classified according to experts in the field. The defect location label is marked with a detection box containing the defect. The position of the detection box is determined by the coordinates of the diagonal point of the detection box, and then the position of the defect is determined.

[0054] S3: Train the YOLO V5 model based on the source domain digital tube chip image dataset to obtain the first defect detection model;

[0055] S4: Obtain the target domain digital tube chip image dataset, and transfer the first defect detection model to the target domain based on transfer learning;

[0056] In this embodiment, the target domain digital tube chip image dataset is obtained by acquiring chip images of the target domain digital tube using a smart industrial camera.

[0057] S5: Use labelImg to annotate part of the target domain digital tube chip image to obtain the defect category label and defect location label of the target domain digital tube chip image;

[0058] S6: Use the labeled target domain digital tube chip image to pre-train the first defect detection model to obtain the second defect detection model;

[0059] In this embodiment, the method of pre-training the first defect detection model using labeled target domain digital tube chip images is the same as the method of training the YOLO V5 model based on the source domain digital tube chip image dataset. In this embodiment, the parameters of the YOLO V5 model are fine-tuned using labeled target domain digital tube chip images so that it can be applied to the target domain for detection.

[0060] S7: The second defect detection model is retrained using an unlabeled target domain digital tube chip image set based on a semi-supervised method to obtain the final defect detection model; the final defect detection model is used to detect defects in the target domain digital tube chip image to obtain the type and location of defects in the target domain chip.

[0061] Preferably, the YOLO V5 model includes: an input terminal, a Backbone network, a Neck network, and a Prediction network.

[0062] Preferably, training the YOLO V5 model based on the source domain digital tube chip image dataset includes:

[0063] S31: Mosaic is used for data augmentation at the input of the YOLO V5 model. Four source domain digital tube chip images are randomly sampled from the source domain digital tube chip image dataset. The four source domain digital tube chip images and their corresponding defect location labels are randomly scaled and then randomly distributed and stitched together to obtain the first source domain digital tube chip augmented image and its corresponding defect location labels.

[0064] S32: Input the enhanced image of the first source domain digital tube chip into the YOLO V5 model, and obtain three feature maps of different scales through the Backbone network and Neck network;

[0065] S33: Input the feature maps of three different scales into the three prediction heads of the Prediction network to predict the type and location of defects in the enhanced image of the first source domain digital tube chip;

[0066] S34: Construct a binary cross-entropy loss based on the category prediction results of defects in the enhanced image of the first source domain digital tube chip;

[0067] S35: Construct the CIoU-Loss loss function based on the location prediction results of defects in the enhanced image of the first source domain digital tube chip;

[0068] S36: Construct the loss function of the YOLO V5 model based on the binary cross-entropy loss and CIoU-Loss loss function, and update the parameters of the YOLO V5 model to obtain the first defect detection model.

[0069] Preferably, the binary cross-entropy loss includes:

[0070] Loss1=-(y·log(p)+(1-y)·log(1-p))

[0071] Where loss1 represents the binary cross-entropy loss, y represents the category label of the defect in the enhanced image of the first source domain digital tube chip, and p represents the category prediction result of the defect in the enhanced image of the first source domain digital tube chip.

[0072] Preferably, the CIoU-Loss loss function includes:

[0073]

[0074]

[0075] Where A is the predicted location of the defect in the enhanced image of the first source domain digital tube chip, B is the location label of the defect in the enhanced image of the first source domain digital tube chip, and C is the minimum convex set of the predicted box and the actual box.

[0076] Preferably, the semi-supervised method for training the second defect detection model using unlabeled target domain digital tube chip images includes:

[0077] S71: At the input end of the second defect detection model, Mosaic is used for data augmentation. Four target domain digital tube chip images are randomly sampled from the unlabeled target domain digital tube chip images. The four sampled target domain digital tube chip images are randomly scaled and then randomly distributed and stitched together to obtain the first target domain digital tube chip augmented image.

[0078] S72: Randomly crop two image blocks x with overlapping areas from the image of the first target domain digital tube chip. u1 and x u2 ;

[0079] S73: Move image patch x u1 and x u2 The images are input into the second defect detection model and processed through the Backbone network and Neck network to obtain image patch x. u1 The corresponding first intermediate feature maps and image patches x at three different scales u2 The corresponding second intermediate feature maps at three different scales;

[0080] S74: Move image patch xu1 The first comprehensive feature map is obtained by concatenating the first intermediate feature maps of the three different scales using the Concat function; the image patch x u2 The three corresponding second intermediate feature maps at different scales are concatenated using the Concat function to obtain the second comprehensive feature map;

[0081] S75: Upsample the first comprehensive feature map and the second comprehensive feature map to the same size as the image of the first target domain digital tube chip to obtain the first comprehensive reconstructed feature map and the second comprehensive reconstructed feature map;

[0082] S76: Based on the feature representations of the overlapping regions in the first and second comprehensive reconstruction feature maps, a directional contrast loss function is constructed to update the parameters of the second defect detection model, thus obtaining the final defect detection model.

[0083] Preferably, the directional contrast loss function includes:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] in, The loss direction is represented by C(), the classifier is represented by γ, the threshold for filtering low-confidence positive samples is represented by N, the number of sample points in the overlapping region is represented by h and w, the length and width of the overlapping region are represented by h and w, and the set of pixel features in the non-overlapping region is represented by Fu. and Represents image patch x u1 and x u2 The overlapping region in β u1 =β(f u1 ) and β u2 =β(f u2 In the feature representation, β represents the nonlinear mapping function, f u1 and f u2 This represents the first and second comprehensive reconstructed feature maps. and The pseudo-label represents the pixel features of the overlapping region in the first and second comprehensive reconstructed feature maps, yn Let represent the pseudo-labels of non-overlapping pixel features in the first and second comprehensive reconstructed feature maps, r() represent the similarity function, max represent the maximum value function, and B represent the training batch size. This represents the directional contrast loss function.

[0092] Preferably, in this embodiment, the generation of the label is represented as follows: for a certain pixel feature f in the first comprehensive reconstructed feature map and the second comprehensive reconstructed feature map... u1 and f u2 The corresponding label is y. ui =argmaxC(f ui ), i∈[1,2], and based on the same principle, we can obtain the pseudo label of a certain pixel feature in the non-overlapping region.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for defect detection of a digital tube chip, characterized in that, include: S1: Obtain the source domain digital tube chip image dataset, which includes images of various digital tube chips in different orientations; S2: Use labelImg to annotate the source domain digital tube chip image to generate corresponding defect category labels and defect location labels; S3: Train the YOLO V5 model based on the source domain digital tube chip image dataset to obtain the first defect detection model; S4: Obtain the target domain digital tube chip image dataset, and transfer the first defect detection model to the target domain based on transfer learning; S5: Use labelImg to annotate part of the target domain digital tube chip image to obtain the defect category label and defect location label of the target domain digital tube chip image; S6: Use the labeled target domain digital tube chip image to pre-train the first defect detection model to obtain the second defect detection model; S7: The second defect detection model is retrained using an unlabeled target domain digital tube chip image set based on a semi-supervised method to obtain the final defect detection model; the final defect detection model is used to detect defects in the target domain digital tube chip image to obtain the type and location of defects in the target domain chip. The training of the YOLO V5 model based on the source domain digital tube chip image dataset includes: S31: Mosaic is used for data augmentation at the input of the YOLO V5 model. Four source domain digital tube chip images are randomly sampled from the source domain digital tube chip image dataset. The four source domain digital tube chip images and their corresponding defect location labels are randomly scaled and then randomly distributed and stitched together to obtain the first source domain digital tube chip augmented image and its corresponding defect location labels. S32: Input the enhanced image of the first source domain digital tube chip into the YOLO V5 model, and obtain three feature maps of different scales through the Backbone network and Neck network; S33: Input the feature maps of three different scales into the three prediction heads of the Prediction network to predict the type and location of defects in the enhanced image of the first source domain digital tube chip; S34: Construct a binary cross-entropy loss based on the category prediction results of defects in the enhanced image of the first source domain digital tube chip; S35: Construct the CIoU-Loss loss function based on the location prediction results of defects in the enhanced image of the first source domain digital tube chip; S36: Construct the loss function of the YOLO V5 model based on the binary cross-entropy loss and CIoU-Loss loss function, and update the parameters of the YOLO V5 model to obtain the first defect detection model; The semi-supervised method for training the second defect detection model using unlabeled target domain digital tube chip images includes: S71: At the input end of the second defect detection model, Mosaic is used for data augmentation. Four target domain digital tube chip images are randomly sampled from the unlabeled target domain digital tube chip images. The four sampled target domain digital tube chip images are randomly scaled and then randomly distributed and stitched together to obtain the first target domain digital tube chip augmented image. S72: Randomly crop two image blocks with overlapping areas from the image of the first target domain digital tube chip. and ; S73: Transfer image blocks and The images are then fed into the second defect detection model and processed through the Backbone and Neck networks to obtain image patches. The corresponding first intermediate feature maps and image patches at three different scales The corresponding second intermediate feature maps at three different scales; S74: Patch the image The first comprehensive feature map is obtained by concatenating the first intermediate feature maps of the three different scales using the Concat function; the image patch is then processed. The three corresponding second intermediate feature maps at different scales are concatenated using the Concat function to obtain the second comprehensive feature map; S75: Upsample the first comprehensive feature map and the second comprehensive feature map to the same size as the image of the first target domain digital tube chip to obtain the first comprehensive reconstructed feature map and the second comprehensive reconstructed feature map; S76: Based on the feature representations of the overlapping regions in the first and second comprehensive reconstruction feature maps, a directional contrast loss function is constructed to update the parameters of the second defect detection model, thus obtaining the final defect detection model.

2. The defect detection method for a digital tube chip according to claim 1, characterized in that, The YOLO V5 model includes: an input network, a backbone network, a neck network, and a prediction network.

3. A defect detection method for a digital tube chip according to claim 1, characterized in that, The binary cross-entropy loss includes: in, Represents the binary cross-entropy loss. This indicates the category label of defects in the enhanced image of the first source domain digital tube chip. This represents the category prediction result of defects in the enhanced image of the first source domain digital tube chip.

4. A defect detection method for a digital tube chip according to claim 3, characterized in that, The CIoU-Loss loss function includes: Where A is the predicted location of the defect in the enhanced image of the first source domain digital tube chip, B is the location label of the defect in the enhanced image of the first source domain digital tube chip, and C is the minimum convex set of the predicted box and the actual box.

5. The defect detection method for a digital tube chip according to claim 4, characterized in that, The directed contrast loss function includes: in, Indicates the direction of loss. Represents a classifier. This represents the threshold for filtering low-confidence positive samples; Indicates the number of sample points in the overlapping region; , Indicates the length and width of the overlapping area; A set of pixel features representing non-overlapping regions; and Represents image blocks and The overlapping area in and Feature representation in Represents a nonlinear mapping function. and This represents the first and second comprehensive reconstructed feature maps. and The pseudo-label represents the pixel features of the overlapping region in the first and second composite reconstructed feature maps. The pseudo-labels represent the non-overlapping pixel features in the first and second composite reconstructed feature maps. Represents the similarity function. Let B represent the maximum value function, and let B represent the training batch size. This represents the directional contrast loss function.

Images