A multi-scale enhancement and selective contrast LCD defect detection method and system

Abstract

The present invention discloses a multi-scale enhancement and selective contrast LCD defect detection method and system, which is characterized by comprising the following steps: obtaining display screen image signal data information and performing image cropping and annotation processing; constructing a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and performing feature extraction processing on the image cropping and annotation processing results; performing target and feature multi-scale enhancement processing on the feature extraction processing results through an auxiliary branch network; performing selective contrast loss calculation processing on the features after multi-scale feature enhancement by the main network to improve the inter-class distance; performing feature decoupling processing on the target feature multi-scale enhancement processing results based on an adaptive decoupling module to resolve the mutual interference between classification and regression; achieving the goal of solving the problems of inter-class confusion and large scale differences in small sample defects of display screens, and having the advantages of high recognition accuracy and strong generalization.

Description

Technical Field

[0001] The present invention relates to the technical field of LCD quality detection, and in particular to a multi-scale enhancement and selective contrast LCD defect detection method and system. Background Art

[0002] Liquid crystal displays (LCDs), core components of electronic devices, are widely used in televisions, computer monitors, smartphones, and tablets. Due to their importance in numerous consumer electronics products, ensuring the quality and performance of LCDs is crucial. However, LCD defect detection faces numerous challenges in practical applications. The most notable issue is the significant imbalance in the number of defect samples, with mura and line defects being extremely rare. Furthermore, the scales of objects in LCD defect data vary significantly, resulting in a sparse distribution of defects. This, combined with the high similarity between categories, significantly interferes with accurate neural network classification.

[0003] Currently, LCD defect detection is primarily based on traditional image processing methods and deep learning approaches. Traditional image processing methods suffer from poor robustness and generalization. Conventional deep learning methods rely on large-scale sample learning, making them difficult to apply to small-sample detection scenarios. Among existing LCD small-sample detection methods, the paper "Mo Wenxing. Research on a Small-Sample LCD Screen Defect Detection System Based on YOLOv5[D]. Dongguan University of Technology, 2024. DOI: 10.44357 / d.cnki.gdgut.2024.000110" proposes a multi-scale enhancement and selective contrast LCD defect detection method based on meta-learning. This method uses six categories from a small- to medium-sized dataset (DAGM) and two categories from an LCD dataset as base classes. It then uses a meta-learning paradigm to learn new categories for the four DAGM categories and two categories from the LCD dataset. However, this method, based on a small dataset as base class learning, makes it difficult for the model to acquire broad and general feature extraction capabilities. Moreover, the model does not take into account the large scale differences and high similarity between classes in LCD defects, so the detection performance of new classes is very low, only about 20%. In the literature "[1] Ma Ling, Lu Yue, Jiang Huiqin, et al., Automatic detection method of LCD product defects based on small sample learning [J]. Journal of Intelligent Systems, 2020, 15(03): 560-567.", for small sample LCD data, a generative adversarial network and cropping of large-scale enhanced data were used, and then pre-trained with other data sets to transfer knowledge to a small amount of data for enhanced learning. However, this method is still trained under the large sample conditions after data enhancement, and is not learned and tested under the actual small sample conditions of {1, 2, 3, 5, 10, 30}. Moreover, this method can only achieve classification, but cannot locate defects. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a multi-scale enhancement and selective contrast LCD defect detection method and system, which solves the problems of inter-class confusion and large scale differences in small sample defects of display screens, and has the advantages of high recognition accuracy and strong generalization.

[0005] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0006] The first aspect of the present application provides a multi-scale enhancement and selective contrast LCD defect detection method, comprising the following steps:

[0007] S101, obtaining display screen image signal data information, and performing image cropping and annotation processing;

[0008] S102. Build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid as the baseline network, and perform feature extraction on the image cropping and annotation results;

[0009] S103, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network;

[0010] S104, performing selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance;

[0011] S105 , performing feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to resolve the mutual interference between classification and regression.

[0012] Furthermore, obtaining the display screen image signal data information and performing image cropping and annotation processing includes the following steps:

[0013] Adjust the camera's spatial position, focal length, aperture and other factors to avoid moiré and liquid crystal patterns;

[0014] Select different backgrounds in a darkroom environment and collect images of the display quality;

[0015] Selecting display screen images with different types of defects from the images, and further screening them to select more representative display screen images as a data set;

[0016] Perform image cropping on the effective feature area of ​​the display screen image;

[0017] The defective areas of the cropped display screen image are marked.

[0018] Optionally, the spatial position, focal length, aperture and other factors of the camera can be adjusted to avoid the occurrence of moiré and liquid crystal patterns. If moiré and liquid crystal patterns exist in some samples, the Fourier transform method is used to eliminate them.

[0019] Furthermore, different backgrounds include white, green, blue, black, grayscale 32, grayscale 64, grayscale 128, grayscale transition, black and white cross backgrounds, etc.

[0020] Furthermore, the display screen images with different defect types include display screen images with physical damage defects such as dots, lines, mura, columns, and scratches.

[0021] Furthermore, a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid is constructed as a baseline network, and feature extraction processing is performed on the image cropping and annotation processing results, including the following steps:

[0022] Using ResNet101 as the backbone network, we extract multi-scale feature maps of the display screen image. Then, we use BiFPN’s multi-level bidirectional feature fusion mechanism to integrate feature map information from different scales.

[0023] The candidate region generation network predicts potential target regions based on the fused feature maps and generates a set of predefined anchor points on feature maps of different scales. The anchor points have different sizes and aspect ratios to accommodate targets of different sizes and shapes.

[0024] For each candidate region, the ROI Align operation is used to extract fixed-size features from the multi-scale feature map output by BiFPN for subsequent classification and bounding box regression, and then the candidate regions are aligned by ROIAlign.

[0025] Based on the ROI detection head, each candidate region is classified and regressed into bounding boxes, and the target detection result is finally output through non-maximum suppression.

[0026] Furthermore, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network includes the following steps:

[0027] The feature extraction network in the auxiliary branch network and the main network of the feature fusion network adopt the same architecture, and the parameters are shared between the two networks;

[0028] Generate k objects of different scales through multi-scale targets;

[0029] By generating feature maps based on targets of various scales in the target pyramid, the matching is further refined through RPN and ROI detection;

[0030] The auxiliary branches are removed after training is completed so as not to affect the inference time.

[0031] Furthermore, the selective contrast loss calculation process is performed on the features after multi-scale feature enhancement of the main network to improve the inter-class distance, including the following steps:

[0032] Improve the distinction between defect classes based on the selective contrast loss function, and perform detection, classification, and regression in parallel through ROI;

[0033] For a small batch of N RoI box features First, the candidate region features Use a layer of multi-layer perceptron to encode z i =MLP(f i ), embedding vector z i and z j The distance d(z i ,z j ) to measure the similarity of sample pairs;

[0034] When the overlap IOU between the candidate region and the true annotation box is greater than the set threshold u, the sample pair will participate in the calculation of the contrast loss;

[0035] The loss is calculated by L(W,(Y,f i ,f j ) uses contrastive learning calculation method to compare high confidence vectors.

[0036] Furthermore, the adaptive decoupling module is used to perform feature decoupling on the target and feature multi-scale enhancement processing results to resolve the mutual interference between classification and regression, including the following steps:

[0037] By using different convolutions to replace the original shared convolution or full connection, the target and feature multi-scale enhancement processing results are decoupled;

[0038] The variant convolution is used to replace the conventional convolution, and the dynamic snake convolution SnakeConv irregular feature extraction method is used for classification. Enhance the model's ability to detect slender and winding structures; for regression, coordinate convolution CoordConv is used to record position coordinates while extracting features. Better target box regression;

[0039] The decoupled features of classification and regression are then processed through parallel spatial attention. With channel attention The features are enhanced and then classified and regressed using the polarization function ψ(.).

[0040] Furthermore, a multi-scale enhancement and selective contrast LCD defect detection method further includes, during the model training process, converting a large dataset into a small sample for base class training, including the following steps:

[0041] The large dataset is the VOC dataset, which is a natural dataset that includes 17,125 samples in 20 categories.

[0042] In base class training, all 20 categories in the VOC dataset are used as base classes, and random sampling without replacement is performed from the base classes in the form of {1, 2, 3, 5, 10, 30} samples.

[0043] During training, data is augmented by flipping, rotating, and cropping the data.

[0044] Furthermore, a multi-scale enhancement and selective contrast LCD defect detection method also includes fine-tuning the network parameters trained on the base class through display defect data.

[0045] Furthermore, fine-tuning the network parameters trained on the base class using display defect data includes the following steps:

[0046] In the new class fine-tuning, the five categories of point, line, mura, scratch, and cylinder in the LCD defect dataset are used as new classes, and the network is fine-tuned using the {1, 2, 3, 5, 10, 30} sample format;

[0047] During the fine-tuning process, since shallow features such as edges and textures are universal, the parameters of the first stage of the backbone network ResNet101 are frozen without adjustment.

[0048] A second aspect of the present application provides a multi-scale enhancement and selective contrast LCD defect detection system, comprising:

[0049] A data acquisition unit, used to acquire display screen image signal data information;

[0050] The first processing unit is used to obtain the display screen image signal data information and perform image cropping and annotation processing;

[0051] The model building unit is used to build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid;

[0052] The second processing unit is used to use a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and to perform feature extraction on the image cropping and annotation processing results;

[0053] The third processing unit is used to perform target and feature multi-scale enhancement processing on the feature extraction region classification processing result through the auxiliary branch network;

[0054] The fourth processing unit is used to perform selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance;

[0055] a fifth processing unit, configured to perform feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to resolve the mutual interference between classification and regression;

[0056] The sixth processing unit is used to process the large data set into small samples for base class training during the model training process;

[0057] The seventh processing unit is used to fine-tune the network parameters trained in the base class using the display screen defect data.

[0058] The beneficial effects of this application are: it solves the problems of inter-class confusion and large scale differences in small sample defects of display screens, and has the advantages of high recognition accuracy and strong generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 Schematic diagram of the steps of a multi-scale enhancement and selective contrast LCD defect detection method of the present invention;

[0061] Figure 2 It is a deep learning detection model diagram of the present invention;

[0062] Figure 3 It is the scale-diversified lifting auxiliary branch module of the present invention;

[0063] Figure 4 is a schematic diagram of the adaptive decoupling module of the present invention;

[0064] Figure 5 This is a sample diagram of small sample defect detection of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0067] Example 1:

[0068] A multi-scale enhancement and selective contrast LCD defect detection method includes the following steps:

[0069] S101, obtaining display screen image signal data information, and performing image cropping and annotation processing;

[0070] Obtain display screen image signal data information, perform image cropping and annotation processing on the display screen image signal data information, and obtain image cropping and annotation processing results.

[0071] Obtaining display screen image signal data information and performing image cropping and annotation processing includes the following steps:

[0072] Adjust the camera's spatial position, focal length, aperture and other factors to avoid moiré and liquid crystal patterns;

[0073] Select different backgrounds in a darkroom environment and collect images of the display quality;

[0074] Selecting display screen images with different types of defects from the images, and further screening them to select more representative display screen images as a data set;

[0075] Perform image cropping on the effective feature area of ​​the display screen image;

[0076] The defective areas of the cropped display screen image are marked.

[0077] By capturing images of the display's quality in a darkroom using various backgrounds, we can provide a wide range of samples for subsequent quality testing. These backgrounds include white, green, blue, black, grayscale 32, grayscale 64, grayscale 128, grayscale transitions, and black-white crossovers. By adjusting the camera's spatial position, focal length, aperture, and other factors, we can avoid moiré and liquid crystal patterning. If moiré and liquid crystal patterning are present in some samples, Fourier transforms are used to eliminate them.

[0078] Display screen images with different defect types include those with physical damage defects such as dots, lines, mura, cylinders, and scratches. By cropping the display screen images' valid feature areas, the images can be saved as 600×600 images. Defect areas in the cropped images are annotated to form a high-quality dataset.

[0079] S102. Build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid as the baseline network, and perform feature extraction on the image cropping and annotation results;

[0080] By performing image cropping and annotation processing on the display screen image signal data information, the image cropping and annotation processing results are obtained. By constructing a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid as the baseline network, and performing feature extraction region classification processing on the image cropping and annotation processing results, the feature extraction region classification processing results are obtained.

[0081] like Figure 2 As shown in the figure, ResNet101 is the backbone network, ADM is the adaptive decoupling module, BiFPN is the bidirectional feature pyramid, and ROI Align is the region of interest alignment. A two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid is constructed as the baseline network, and feature extraction is performed on the image cropping and annotation results. The following steps are included:

[0082] Using ResNet101 as the backbone network, we extract multi-scale feature maps of the display screen image. Then, we use BiFPN’s multi-level bidirectional feature fusion mechanism to integrate feature map information from different scales.

[0083] The candidate region generation network (RPN) predicts potential target regions based on the fused feature maps and generates a set of predefined anchor points on feature maps of different scales. The anchor points have different sizes and aspect ratios to accommodate targets of different sizes and shapes.

[0084] For each candidate region, the ROI Align operation is used to extract fixed-size features from the multi-scale feature map output by BiFPN for subsequent classification and bounding box regression, and then the candidate regions are aligned by ROIAlign.

[0085] Based on the ROI detection head, each candidate region is classified and regressed into bounding boxes, and the target detection result is finally output through non-maximum suppression (NMS).

[0086] S103, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network;

[0087] A two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid is constructed as the baseline network. Feature extraction region classification is performed on the cropped and annotated image results to obtain the feature extraction region classification results. An auxiliary branch network, consisting of a feature extraction network and a feature fusion network, is used to perform multi-scale target and feature enhancement on the feature extraction region classification results. This is further refined and matched using RPN and ROI detection, resulting in multi-scale target and feature enhancement.

[0088] like Figure 3 As shown in FIG, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network includes the following steps:

[0089] The feature extraction network in the auxiliary branch network and the main network of the feature fusion network adopt the same architecture, and the parameters are shared between the two networks;

[0090] Generate k objects of different scales through multi-scale targets; for example, in this example, k is set to 6, generating 32 2 , 64 2 , 128 2 , 256 2 , 512 2 , 800 2 Pixel target.

[0091] By generating feature maps based on targets of various scales in the target pyramid, the matching is further refined through RPN and ROI detection; the matching is shown in Table 1, where P2, P3, P4, P5, and P6 represent the feature maps of the first, second, third, fourth, and fifth layers of the feature fusion network, respectively.

[0092] Table 1

[0093]

[0094] The auxiliary branches are removed after training is completed so as not to affect the inference time.

[0095] S104, performing selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance;

[0096] The auxiliary branch network performs multi-scale target and feature enhancement on the feature extraction region classification results, resulting in a multi-scale target and feature enhancement result. The features after multi-scale feature enhancement in the main network are then subjected to a selective contrast loss calculation to improve inter-class distances and avoid inter-class confusion. This contrast loss is calculated for high-quality candidate boxes with an IOU greater than a threshold u to improve inter-class distances.

[0097] The selective contrast loss calculation process is performed on the features after multi-scale feature enhancement of the main network to improve the inter-class distance, including the following steps:

[0098] Improve the distinction between defect classes based on the selective contrast loss function, and perform detection, classification, and regression in parallel through ROI;

[0099] For a small batch of N RoI box features First, the candidate region features A layer of multi-layer perceptron (MLP) is used for encoding z i =MLP(f i ), embedding vector z i and z j The distance d(z i ,z j ) to measure the similarity of sample pairs;

[0100] When the overlap (Intersection over Union, IOU) between the candidate region and the true annotation box is greater than the set threshold u, the sample pair will participate in the calculation of the contrast loss. The expression of the loss function is as follows:

[0101]

[0102] in, is the loss function, u is set to 0.7 in this example, f i 、f j are all candidate region features, y j is the target real box, L(W,(Y,f i ,f j ) is the loss calculation, W is the model parameter, Y represents the similarity or dissimilarity, and IOU is the overlap of the real annotation box;

[0103] The loss calculation method in this example is L(W,(Y,f i ,f j) Use contrastive learning to perform contrastive calculations on high-confidence vectors. The expression for vector contrast calculation is as follows:

[0104]

[0105] in, and Represents the normalized feature vector, m represents the minimum distance between the same sample pairs, y = 1 means that the sample pairs are heterogeneous, thus forcing the distance between samples of different categories to be greater than m, y = 0 means that the sample pairs are homogeneous, and the actual distance between them is calculated. is the candidate region feature f i and candidate region features f j For the distance in the feature space, the expression for Euclidean distance calculation is as follows:

[0106] d(z i ,z j )=||z i -z j ||2

[0107] Where d is the Euclidean distance, z i and z j denote the i-th and j-th embedding vectors respectively.

[0108] S105, performing feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to resolve the mutual interference between classification and regression;

[0109] By performing selective contrast loss calculation on the multi-scale enhancement results of targets and features through the main network, the inter-class distance can be improved. The features output by ROIAlign are subjected to targeted variation convolution, attention, and polarization functions to decouple classification and regression.

[0110] like Figure 4 As shown, the feature decoupling process of the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to solve the mutual interference between classification and regression includes the following steps:

[0111] By using different convolutions to replace the original shared convolution or full connection, the target and feature multi-scale enhancement processing results are decoupled;

[0112] The variant convolution is used to replace the conventional convolution, and the dynamic snake convolution SnakeConv irregular feature extraction method is used for classification tasks. Enhance the model's ability to detect slender and winding structures; for regression tasks, coordinate convolution CoordConv is used to extract features while recording position coordinates to obtain To better regress the target frame;

[0113] The decoupled features of classification and regression are then processed through parallel spatial attention. With channel attention The features are enhanced and then classified and regressed using a specific polarization function ψ(.);

[0114] The characteristic expression after polarization is as follows:

[0115] M c =σ(C 1×1 (relu(C 1×1 (avgpool(f)))))

[0116] M s =σ(C 7×7 ([maxpool(F);avgpool(f)]))

[0117]

[0118] Among them, M s is spatial attention, M c is the channel attention, M f is the feature map after spatial and channel enhancement, F′ is the output feature after the polarization function, σ() is the activation function, C is the 1×1 convolution, avgpool() is the average pooling, and ψ() is the polarization function.

[0119] It should be noted that the traditional detection head uses the features processed by ROIAlign for classification and positioning, but will use full connection or convolution with the same parameters after ROIAlign. The adaptive decoupling module proposed in this application processes classification and regression separately, does not use convolution layers with the same parameters, and uses one to achieve decoupling for each, so as to achieve decoupling of classification and regression.

[0120] Example 2:

[0121] This method differs from the first embodiment in that the network parameters are trained to a level capable of extracting basic features, thereby enhancing the effectiveness of the first-stage network, and then being specifically applied in the second-stage network. A multi-scale enhancement and selective contrast LCD defect detection method also includes, during model training, processing the large dataset into small samples for base class training.

[0122] During model training, converting a large dataset into a small sample for base class training involves the following steps:

[0123] In this example, the large dataset is the VOC2007 dataset, which is a natural dataset that includes 17,125 samples in 20 categories.

[0124] In base class training, all 20 categories in the VOC dataset are used as base classes, and random sampling without replacement is performed from the base classes in the form of {1, 2, 3, 5, 10, 30} samples.

[0125] During training, data is augmented by flipping, rotating, and cropping the data.

[0126] Example 3:

[0127] The difference from the second embodiment is that the multi-scale enhancement and selective contrast LCD defect detection method further includes fine-tuning the network parameters trained on the base class using display defect data.

[0128] Fine-tuning the network parameters trained on the base class using display defect data includes the following steps:

[0129] In the new class fine-tuning, the five categories of point, line, mura, scratch, and cylinder in the LCD defect dataset are used as new classes, and the network is fine-tuned using the {1, 2, 3, 5, 10, 30} sample format;

[0130] During the fine-tuning process, since shallow features such as edges and textures are universal, the parameters of the first stage of the backbone network ResNet101 are frozen without adjustment.

[0131] The above is a multi-scale enhancement and selective contrast LCD defect detection method provided in an embodiment of the present application. The following is a multi-scale enhancement and selective contrast LCD defect detection system provided in an embodiment of the present application.

[0132] A multi-scale enhancement and selective contrast LCD defect inspection system comprising:

[0133] A data acquisition unit, used to acquire display screen image signal data information;

[0134] The first processing unit is used to obtain the display screen image signal data information and perform image cropping and annotation processing;

[0135] The model building unit is used to build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid;

[0136] The second processing unit is used to use a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and to perform feature extraction on the image cropping and annotation processing results;

[0137] The third processing unit is used to perform target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network;

[0138] The fourth processing unit is used to perform selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance;

[0139] a fifth processing unit, configured to perform feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to resolve the mutual interference between classification and regression;

[0140] The sixth processing unit is used to process the large data set into small samples for base class training during the model training process;

[0141] The seventh processing unit is used to fine-tune the network parameters trained in the base class using the display screen defect data.

[0142] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] The terms "first", "second" and "third" etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0147] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scale enhancement and selective contrast LCD defect detection method, characterized in that: The following steps are involved: S101, obtaining display screen image signal data information, and performing image cropping and annotation processing; S102. Construct a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and perform feature extraction on the image cropping and annotation processing results, including the following steps: Using ResNet101 as the backbone network, it extracts multi-scale feature maps of the display screen image. Then, through the multi-level bidirectional feature fusion mechanism of the bidirectional feature pyramid, it integrates the feature map information from different scales. The candidate region generation network predicts potential target regions based on the fused feature maps and generates a set of predefined anchor points on feature maps of different scales. The anchor points have different sizes and aspect ratios to accommodate targets of different sizes and shapes. For each candidate region, the ROI Align operation is used to extract fixed-size features from the multi-scale feature map output by BiFPN for subsequent classification and bounding box regression, and then the candidate regions are aligned by ROIAlign. Based on the ROI detection head, each candidate region is classified and regressed into bounding boxes, and the target detection result is finally output through non-maximum suppression; S103, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network, including the following steps: The feature extraction network in the auxiliary branch network and the main network of the feature fusion network adopt the same architecture, and the parameters are shared between the two networks; Generate k objects of different scales through multi-scale targets; By generating feature maps based on targets of various scales in the target pyramid, the matching is further refined through RPN and ROI detection; Remove the auxiliary branches after training to avoid affecting the inference time; S104, performing selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance, including the following steps: Improve the discrimination between defect classes based on selective contrast loss function, and perform detection, classification and regression in parallel through regions of interest; For a small batch of N RoI box features First, the candidate region features Use a layer of multi-layer perceptron to encode z i =MLP(f i ), embedding vector z i and z j The distance d(z i ,z j ) to measure the similarity of sample pairs; When the overlap between the candidate region and the true annotation box is greater than the set threshold u, the sample pair will participate in the calculation of the contrast loss. The expression of the loss function is as follows: in, is the loss function, u is set to 0.7, f i 、f j are all candidate region features, y j is the target real box, L(W,(Y,f i ,f j ) is the loss calculation, W is the model parameter, Y represents the similarity or dissimilarity, and IOU is the overlap of the real annotation box; The loss is calculated by L(W,(Y,f i ,f j ) Use contrastive learning to perform contrastive calculations on high-confidence vectors. The expression for vector contrast calculation is as follows: in, and Represents the normalized feature vector, m represents the minimum distance between the same sample pairs, y = 1 means that the sample pairs are heterogeneous, thus forcing the distance between samples of different categories to be greater than m, y = 0 means that the sample pairs are homogeneous, and the actual distance between them is calculated. is the candidate region feature f i and candidate region features f j For the distance in the feature space, the expression for Euclidean distance calculation is as follows: d(z i ,z j )=||z i -z j ||2 Where d is the Euclidean distance, z i and z j denote the i-th and j-th embedding vectors respectively; S105: performing feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to resolve the mutual interference between classification and regression, including the following steps: By using different convolutions to replace the original shared convolution or full connection, the target and feature multi-scale enhancement processing results are decoupled; The variant convolution is used to replace the conventional convolution, and the dynamic snake convolution SnakeConv irregular feature extraction method is used for classification. Enhance the model's ability to detect slender and winding structures; for regression, coordinate convolution CoordConv is used to record position coordinates while extracting features. Better target box regression; The decoupled features of classification and regression are then processed through parallel spatial attention. With channel attention The features are enhanced and then classified and regressed using the polarization function ψ(.); The characteristic expression after polarization is as follows: M c =σ(C 1×1 (relu(C 1×1 (avgpool(f))))) M s =σ(C 7×7 ([maxpool(F);avgpool(f)])) Among them, M s is spatial attention, M c is the channel attention, M f is the feature map after spatial and channel enhancement, F′ is the output feature after the polarization function, σ() is the activation function, C is the 1×1 convolution, avgpool() is the average pooling, and ψ() is the polarization function.

2. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1, wherein: The method of obtaining the display screen image signal data information and performing image cropping and annotation processing includes the following steps: Adjust the camera's spatial position, focal length, and aperture to avoid moiré and liquid crystal patterns; Select different backgrounds in a darkroom environment and collect images of the display quality; Selecting display screen images with different types of defects from the images, and further screening them to select more representative display screen images as a data set; Perform image cropping on the effective feature area of ​​the display screen image; The defective areas of the cropped display screen image are marked.

3. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1 or 2, characterized in that: Also includes: During model training, converting a large dataset into small samples for base class training involves the following steps: The large dataset is the VOC dataset, which is a natural dataset that includes 17,125 samples in 20 categories. In base class training, all 20 categories in the VOC dataset are used as base classes, and random sampling without replacement is performed from the base classes in the form of {1, 2, 3, 5, 10, 30} samples. During training, data is flipped, rotated, and cropped to achieve data enhancement.

4. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 3, wherein: It also includes fine-tuning the network parameters trained on the base class using display defect data.

5. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 4, characterized in that: The fine-tuning of the network parameters trained in the base class using the display screen defect data includes the following steps: In the new class fine-tuning, the five categories of point, line, mura, scratch, and cylinder in the LCD defect dataset are used as new classes, and the network is fine-tuned using the {1, 2, 3, 5, 10, 30} sample format; During the fine-tuning process, since shallow features such as edges and textures are universal, the parameters of the first stage of the backbone network ResNet101 are frozen without adjustment.

6. A multi-scale enhancement and selective contrast LCD defect detection system, configured to execute the multi-scale enhancement and selective contrast LCD defect detection method according to any one of claims 1 to 5, characterized in that: include: A data acquisition unit, used to acquire display screen image signal data information; The first processing unit is used to obtain the display screen image signal data information and perform image cropping and annotation processing; The model building unit is used to build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid; The second processing unit is used to use a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and to perform feature extraction on the image cropping and annotation processing results; The third processing unit is used to perform target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network; The fourth processing unit is used to perform selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance; a fifth processing unit, configured to perform feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to resolve the mutual interference between classification and regression; The sixth processing unit is used to process the large data set into small samples for base class training during the model training process; The seventh processing unit is used to fine-tune the network parameters trained in the base class using the display screen defect data.

Images