Multi-scale enhancement and selective comparison LCD defect detection method and system

Abstract

The invention discloses a multi-scale enhancement and selective comparison LCD defect detection method and system, and the method is characterized in that the method comprises the following steps: obtaining display screen image signal data information, and carrying out the image cutting and marking processing; constructing a two-stage target detection model based on a Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and carrying out feature extraction processing on an image cutting and labeling processing result; performing target and feature multi-scale enhancement processing on a feature extraction processing result through an auxiliary branch network; selective comparison loss calculation processing is carried out on the features after multi-scale feature enhancement of the main network, so that the inter-class distance is increased; performing feature decoupling processing on a target feature multi-scale enhancement processing result based on a self-adaptive decoupling module so as to solve mutual interference between classification and regression; the problems of inter-class confusion and large scale difference in small sample defects of the display screen are solved, and the method has the advantages of high recognition precision and high generalization.

Description

Technical Field

[0001] The present invention relates to the technical field of LCD quality detection, and in particular to a multi-scale enhancement and selective contrast LCD defect detection method and system. Background Art

[0002] Liquid crystal displays (LCDs), as core components of electronic devices, have been widely used in fields such as televisions, computer monitors, smartphones, and tablets. Due to its importance in many consumer electronic products, ensuring the quality and performance of LCDs is particularly critical. However, LCD defect detection faces many challenges in practical applications. The most notable problem is the serious imbalance in the number of defect samples, with mura and line defects being extremely rare. In addition, the scale differences of targets in LCD defect data are extremely large, and the defect distribution is sparse. The high similarity between categories also seriously interferes with the accurate classification of neural networks.

[0003] At present, LCD defects are mainly divided into traditional image processing methods and deep learning methods. Traditional image methods have poor robustness and generalization ability. Conventional deep learning methods rely on large-scale sample learning and are difficult to apply to small sample scene detection. Among the existing LCD small sample detection research methods, the literature "Mo Wenxing. Research on small sample LCD screen defect detection system based on YOLOv5 [D]. Dongguan University of Technology, 2024. DOI: 10.44357 / d.cnki.gdgut.2024.000110." proposed a multi-scale enhancement and selective contrast LCD defect detection method based on meta-learning method. This method uses 6 categories in a small and medium-sized dataset DAGM and 2 categories in the LCD dataset as base classes for learning, and then uses the meta-learning paradigm to learn new categories for 4 categories in DAGM and 2 categories in LCD data. However, this method uses a small dataset as the base class for learning, which makes it difficult for the model to learn broad and general feature extraction capabilities. Moreover, the model does not take into account the large scale differences and high similarity between classes in LCD defects, so the detection performance of new classes is very low, only about 20%. In the literature "[1] Ma Ling, Lu Yue, Jiang Huiqin, et al. Automatic detection method of LCD product defects based on small sample learning [J]. Journal of Intelligent Systems, 2020, 15(03): 560-567.", for small sample LCD data, generative adversarial networks and cropping are used to greatly enhance data, and then pre-train with other data sets to transfer knowledge to a small amount of data for enhanced learning. However, this method is still trained under the condition of large samples after data enhancement, and has not been learned and tested under the actual small sample conditions of {1, 2, 3, 5, 10, 30}. Moreover, this method can only achieve classification, but cannot locate defects. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a multi-scale enhancement and selective contrast LCD defect detection method and system, which solves the problems of inter-class confusion and large scale differences in small sample defects of display screens, and has the advantages of high recognition accuracy and strong generalization.

[0005] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0006] The first aspect of the present application provides a multi-scale enhancement and selective contrast LCD defect detection method, comprising the following steps:

[0007] S101, obtaining display screen image signal data information, and performing image cropping and annotation processing;

[0008] S102. Build a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and perform feature extraction on the image cropping and annotation processing results;

[0009] S103, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network;

[0010] S104, performing selective contrast loss calculation processing on the features enhanced by multi-scale features of the main network to improve the distance between classes;

[0011] S105. Perform feature decoupling processing on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to solve the mutual interference between classification and regression.

[0012] Furthermore, obtaining the display screen image signal data information and performing image cropping and annotation processing includes the following steps:

[0013] Adjust the camera's spatial position, focal length, aperture and other factors to avoid moiré and liquid crystal patterns;

[0014] Select different backgrounds in a darkroom environment and collect images of the display quality of the display screen;

[0015] Selecting display screen images with different types of defects on the display screen, and further screening to select more representative display screen images as a data set;

[0016] Perform image cropping processing on the effective feature area of ​​the display screen image;

[0017] The defective area of ​​the cropped display screen image is marked.

[0018] Optionally, the spatial position, focal length, aperture and other factors of the camera may be adjusted to avoid the occurrence of moiré and liquid crystal patterns. If moiré and liquid crystal patterns exist in some samples, the Fourier transform method may be used to eliminate them.

[0019] Furthermore, different backgrounds include white, green, blue, black, grayscale 32, grayscale 64, grayscale 128, grayscale transition, black and white cross and other backgrounds.

[0020] Furthermore, display screen images with different defect types include display screen images with physical damage defect types such as dots, lines, mura, columns, scratches, etc.

[0021] Furthermore, a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid is constructed as a baseline network, and feature extraction processing is performed on the image cropping and annotation processing results, including the following steps:

[0022] ResNet101 is used as the backbone network to extract multi-scale feature maps of display screen images, and then the feature map information from different scales is integrated through the multi-level bidirectional feature fusion mechanism of BiFPN;

[0023] The candidate region generation network predicts potential target regions based on the fused feature maps and generates a set of predefined anchors on feature maps of different scales. The anchors have different sizes and aspect ratios to accommodate targets of different sizes and shapes.

[0024] For each candidate region, the ROIAlign operation is used to extract fixed-size features from the multi-scale feature map output by BiFPN for subsequent classification and bounding box regression, and then the candidate regions are aligned by ROI Align.

[0025] Based on the ROI detection head, each candidate region is classified and regressed into a bounding box, and finally the target detection result is output through non-maximum suppression.

[0026] Furthermore, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network includes the following steps:

[0027] The main network of the feature extraction network and the feature fusion network in the auxiliary branch network adopt the same architecture, and the parameters are shared between the two networks;

[0028] Generate k objects of different scales through multi-scale targets;

[0029] By generating feature maps based on the targets of each scale in the target pyramid, the matching is further refined through RPN and ROI detection;

[0030] The auxiliary branches are removed after training is completed so as not to affect the inference time.

[0031] Furthermore, the selective contrast loss calculation process is performed on the features after multi-scale feature enhancement of the main network to improve the inter-class distance, including the following steps:

[0032] Improve the distinction between defect classes based on the selective contrast loss function, and perform detection, classification, and regression in parallel through ROI;

[0033] For a small batch of N RoI box features First, the candidate region features Use a layer of multi-layer perceptron for encoding z i =MLP(f i ), embedding vector z i and z j The distance d(z i ,z j ) to measure the similarity of sample pairs;

[0034] When the overlap IOU between the candidate region and the true annotation box is greater than the set threshold u, the sample pair will participate in the calculation of the contrast loss;

[0035] The loss calculation method L(W,(Y,f i ,f j ) uses contrastive learning calculation method to compare and calculate high confidence vectors.

[0036] Further, the feature decoupling process is performed on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to solve the mutual interference between classification and regression, including the following steps:

[0037] By using different convolutions to replace the original shared convolutions or full connections, the multi-scale enhancement processing results of the target features are decoupled.

[0038] The variant convolution is used to replace the conventional convolution, and the dynamic snake convolution SnakeConv irregular feature extraction method is used for classification. Enhance the model's ability to detect slender and circuitous structures; for regression, coordinate convolution CoordConv is used to extract features while recording position coordinates to obtain Better target box regression;

[0039] The decoupled features of classification and regression are then processed through parallel spatial attention. With channel attention The features are enhanced and then classified and regressed through the polarization function ψ(.).

[0040] Furthermore, a multi-scale enhancement and selective contrast LCD defect detection method further includes, during the model training process, processing the large data set into a small sample form for base class training, including the following steps:

[0041] The large dataset is the VOC dataset, which is a natural dataset that includes 17,125 samples in 20 categories.

[0042] In base class training, all 20 categories in the VOC dataset are used as base classes, and random sampling without replacement is performed from the base classes in the form of {1, 2, 3, 5, 10, 30} samples for training;

[0043] During training, data is augmented by flipping, rotating, and cropping the data.

[0044] Furthermore, a multi-scale enhancement and selective contrast LCD defect detection method also includes fine-tuning the network parameters trained on the base class through display defect data.

[0045] Furthermore, fine-tuning the network parameters trained in the base class using the display defect data includes the following steps:

[0046] In the new class fine-tuning, the five categories of point, line, mura, scratch, and cylinder in the LCD defect dataset are taken as new classes, and the network is fine-tuned using the {1, 2, 3, 5, 10, 30} sample format;

[0047] During the fine-tuning process, since shallow features such as edges and textures are universal, the parameters of the first stage of the backbone network ResNet101 are frozen without adjustment.

[0048] The second aspect of the present application provides a multi-scale enhancement and selective contrast LCD defect detection system, comprising:

[0049] A data acquisition unit, used to acquire display screen image signal data information;

[0050] The first processing unit is used to obtain the display screen image signal data information and perform image cropping and annotation processing;

[0051] The model building unit is used to build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid;

[0052] The second processing unit is used to use a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and perform feature extraction processing on the image cropping and annotation processing results;

[0053] The third processing unit is used to perform target feature multi-scale enhancement processing on the feature extraction region classification processing result through the auxiliary branch network;

[0054] A fourth processing unit is used to perform selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance;

[0055] A fifth processing unit, configured to perform feature decoupling processing on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to resolve the mutual interference between classification and regression;

[0056] The sixth processing unit is used to process the large data set into a small sample form for base class training during the model training process;

[0057] The seventh processing unit is used to fine-tune the network parameters trained by the base class through the display screen defect data.

[0058] The beneficial effects of the present application are as follows: it solves the problems of inter-class confusion and large scale differences in small sample defects of display screens, and has the advantages of high recognition accuracy and strong generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a schematic diagram of the steps of a multi-scale enhancement and selective contrast LCD defect detection method of the present invention;

[0061] Figure 2 It is a deep learning detection model diagram of the present invention;

[0062] Figure 3 It is the scale-diversified lifting auxiliary branch module of the present invention;

[0063] Figure 4 is a schematic diagram of an adaptive decoupling module of the present invention;

[0064] Figure 5 This is a sample diagram of small sample defect detection of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0067] Embodiment 1:

[0068] A multi-scale enhancement and selective contrast LCD defect detection method, comprising the following steps:

[0069] S101, obtaining display screen image signal data information, and performing image cropping and annotation processing;

[0070] Obtain display screen image signal data information, perform image cropping and annotation processing on the display screen image signal data information, and obtain image cropping and annotation processing results.

[0071] Obtaining display screen image signal data information and performing image cropping and annotation processing includes the following steps:

[0072] Adjust the camera's spatial position, focal length, aperture and other factors to avoid moiré and liquid crystal patterns;

[0073] Select different backgrounds in a darkroom environment and collect images of the display quality of the display screen;

[0074] Selecting display screen images with different types of defects on the display screen, and further screening to select more representative display screen images as a data set;

[0075] Perform image cropping processing on the effective feature area of ​​the display screen image;

[0076] The defective area of ​​the cropped display screen image is marked.

[0077] By selecting different backgrounds in a darkroom environment and collecting images of the display quality, a wide range of samples can be provided for subsequent quality inspection. Different backgrounds include white, green, blue, black, grayscale 32, grayscale 64, grayscale 128, grayscale transition, black and white cross, etc. By adjusting the spatial position, focal length, aperture and other factors of the camera, moiré and liquid crystal patterns can be avoided. If some samples exist, the Fourier transform method is used to eliminate them.

[0078] Display screen images with different defect types include those with physical damage defects such as dots, lines, mura, cylinders, scratches, etc. The effective feature areas of the display screen images are cropped and saved as 600×600 images. The cropped display screen images are labeled with defect areas to form a high-quality data set.

[0079] S102. Build a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and perform feature extraction on the image cropping and annotation processing results;

[0080] By performing image cropping and annotation processing on the display screen image signal data information, the image cropping and annotation processing results are obtained. By constructing a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and performing feature extraction region classification processing on the image cropping and annotation processing results, the feature extraction region classification processing results are obtained.

[0081] like Figure 2 As shown in the figure, ResNet101 is the backbone network, ADM is the adaptive decoupling module, BiFPN is the bidirectional feature pyramid, and ROIAlign is the region of interest alignment. A two-stage target detection model based on FasterR-CNN combined with a bidirectional feature pyramid is constructed as a baseline network, and feature extraction processing is performed on the image cropping and annotation processing results, including the following steps:

[0082] ResNet101 is used as the backbone network to extract multi-scale feature maps of display screen images, and then the feature map information from different scales is integrated through the multi-level bidirectional feature fusion mechanism of BiFPN;

[0083] The candidate region generation network (RPN) predicts the potential target region based on the fused feature map and generates a set of predefined anchor points on the feature maps of different scales. The anchor points have different sizes and aspect ratios to accommodate targets of different sizes and shapes.

[0084] For each candidate region, the ROIAlign operation is used to extract fixed-size features from the multi-scale feature map output by BiFPN for subsequent classification and bounding box regression, and then the candidate regions are aligned by ROI Align.

[0085] Based on the ROI detection head, each candidate region is classified and regressed into a bounding box, and finally the target detection result is output through non-maximum suppression (NMS).

[0086] S103, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network;

[0087] By constructing a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as the baseline network, and performing feature extraction region classification processing on the image cropping and annotation processing results, the feature extraction region classification processing results are obtained. The auxiliary branch network includes a feature extraction network and a feature fusion network. The auxiliary branch network performs target and feature multi-scale enhancement processing on the feature extraction region classification processing results, and further refines the matching through RPN and ROI detection to obtain target and feature multi-scale enhancement processing.

[0088] like Figure 3 As shown, the target and feature multi-scale enhancement processing of the feature extraction processing result by the auxiliary branch network includes the following steps:

[0089] The main network of the feature extraction network and the feature fusion network in the auxiliary branch network adopt the same architecture, and the parameters are shared between the two networks;

[0090] Generate k objects of different scales through multi-scale targets; for example, in this example, k is set to 6, generating 32 2 , 64 2 , 128 2 , 256 2 ,512 2 , 800 2 Pixel target.

[0091] By generating feature maps according to the targets of each scale in the target pyramid, the matching is further refined through RPN and ROI detection; the matching is shown in Table 1, where P2, P3, P4, P5, and P6 represent the first, second, third, fourth, and fifth layer feature maps of the feature fusion network, respectively.

[0092] Table 1

[0093]

[0094] The auxiliary branches are removed after training is completed so as not to affect the inference time.

[0095] S104, performing selective contrast loss calculation processing on the features enhanced by multi-scale features of the main network to improve the distance between classes;

[0096] The auxiliary branch network performs multi-scale enhancement of target features on the classification results of feature extraction regions to obtain multi-scale enhancement results of targets and features. The features after multi-scale feature enhancement by the main network are subjected to selective contrast loss calculation to improve the distance between classes and avoid confusion between classes. The inter-class distance is improved by calculating contrast loss for high-quality candidate boxes whose IOU is greater than the threshold u.

[0097] The selective contrast loss calculation process is performed on the features after multi-scale feature enhancement of the main network to improve the inter-class distance, including the following steps:

[0098] Improve the distinction between defect classes based on the selective contrast loss function, and perform detection, classification, and regression in parallel through ROI;

[0099] For a small batch of N RoI box features First, the candidate region features A layer of multi-layer perceptron (MLP) is used for encoding z i =MLP(f i ), embedding vector z i and z j The distance d(z i ,z j ) to measure the similarity of sample pairs;

[0100] When the overlap (Intersection over Union, IOU) between the candidate region and the true annotation box is greater than the set threshold u, the sample pair will participate in the calculation of the contrast loss. The expression of the loss function is as follows:

[0101]

[0102] in, is the loss function, u is set to 0.7 in this example, and f j is the candidate region feature, y j is the target real box, L(W,(Y,f i ,f j ) is the loss calculation, W is the model parameter, Y represents the similarity or dissimilarity, and IOU is the overlap of the real annotation box;

[0103] The loss calculation method in this example is L(W,(Y,f i ,f j) The contrastive learning calculation method is used to compare and calculate the high confidence vector. The expression of vector comparison calculation is as follows:

[0104]

[0105] in, and represents the normalized feature vector, m represents the minimum distance between the same sample pairs, y = 1 represents that the sample pairs are heterogeneous, thus forcing the distance between samples of different categories to be greater than m, and y = 0 represents that the sample pairs are homogeneous sample pairs, then the actual distance between them is calculated. is the distance between candidate region f1 and candidate region f2 in the feature space. The expression for calculating the Euclidean distance is as follows:

[0106] d(z i ,z j )=||z i -z j ||2

[0107] Where d is the Euclidean distance, z i and z j denote the i-th and j-th embedding vectors respectively.

[0108] S105, performing feature decoupling processing on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to solve the mutual interference between classification and regression;

[0109] By performing selective contrast loss calculation on the target and feature multi-scale enhancement results through the main network, the inter-class distance can be improved. The features output by ROIAlign are subjected to targeted variation convolution, attention, and polarization functions to decouple classification and regression.

[0110] like Figure 4 As shown, the feature decoupling process of the target feature multi-scale enhancement processing result based on the adaptive decoupling module to solve the mutual interference between classification and regression includes the following steps:

[0111] By using different convolutions to replace the original shared convolutions or full connections, the multi-scale enhancement processing results of the target features are decoupled.

[0112] The variant convolution is used to replace the conventional convolution. For the classification task, the dynamic snake convolution SnakeConv irregular feature extraction method is used to obtain Enhance the model's ability to detect slender and circuitous structures; for regression tasks, coordinate convolution CoordConv is used to extract features while recording position coordinates to obtain To better regress the target frame;

[0113] The decoupled features of classification and regression are then processed through parallel spatial attention. With channel attention The features are enhanced, and then classified and regressed through a specific polarization function ψ(.);

[0114] The characteristic expression after polarization is as follows:

[0115] M c =σ(C 1×1 (relu(C 1×1 (avgpool(f)))))

[0116] M s =σ(C 7×7 ([maxpool(F);avgpool(f)]))

[0117]

[0118] Among them, M s is the spatial attention, M c is the channel attention, M f is the feature map after spatial and channel enhancement, F' is the output feature after the polarization function, σ() is the activation function, C is a 1×1 convolution, avgpool() is the average pooling, and ψ() is the polarization function.

[0119] It should be noted that the traditional detection head uses the features processed by ROIAlign for classification and positioning, but will use full connection or convolution with the same parameters after ROIAlign. The adaptive decoupling module proposed in this application processes classification and regression separately, does not use convolution layers with the same parameters, and uses one to achieve decoupling for each, so as to achieve decoupling of classification and regression.

[0120] Embodiment 2:

[0121] The difference from the first embodiment is that in order to train the network parameters to a level that can extract basic features to improve the effect of the first-stage network, the second-stage network is then specifically applied. A multi-scale enhancement and selective contrast LCD defect detection method also includes, during the model training process, processing the large data set in the form of small samples for base class training.

[0122] During the model training process, the large dataset is converted into small samples for base class training, which includes the following steps:

[0123] In this example, the large data set is the VOC2007 data set, which is a natural data set. The natural data set includes 17,125 samples in 20 categories.

[0124] In base class training, all 20 categories in the VOC dataset are used as base classes, and random sampling without replacement is performed from the base classes in the form of {1, 2, 3, 5, 10, 30} samples for training;

[0125] During training, data is augmented by flipping, rotating, and cropping the data.

[0126] Embodiment three:

[0127] The difference from the second embodiment is that the multi-scale enhancement and selective contrast LCD defect detection method also includes fine-tuning the network parameters trained on the base class through the display defect data.

[0128] Fine-tuning the network parameters trained on the base class using display defect data includes the following steps:

[0129] In the new class fine-tuning, the five categories of point, line, mura, scratch, and cylinder in the LCD defect dataset are taken as new classes, and the network is fine-tuned using the {1, 2, 3, 5, 10, 30} sample format;

[0130] During the fine-tuning process, since shallow features such as edges and textures are universal, the parameters of the first stage of the backbone network ResNet101 are frozen without adjustment.

[0131] The above is a multi-scale enhancement and selective contrast LCD defect detection method provided in an embodiment of the present application, and the following is a multi-scale enhancement and selective contrast LCD defect detection system provided in an embodiment of the present application.

[0132] A multi-scale enhancement and selective contrast LCD defect inspection system comprising:

[0133] A data acquisition unit, used to acquire display screen image signal data information;

[0134] The first processing unit is used to obtain the display screen image signal data information and perform image cropping and annotation processing;

[0135] The model building unit is used to build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid;

[0136] The second processing unit is used to use a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and perform feature extraction processing on the image cropping and annotation processing results;

[0137] The third processing unit is used to perform target feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network;

[0138] A fourth processing unit is used to perform selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance;

[0139] A fifth processing unit, configured to perform feature decoupling processing on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to resolve the mutual interference between classification and regression;

[0140] The sixth processing unit is used to process the large data set into a small sample form for base class training during the model training process;

[0141] The seventh processing unit is used to fine-tune the network parameters trained by the base class through the display screen defect data.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] The terms "first", "second" and "third" etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scale enhancement and selective contrast LCD defect detection method, characterized in that: The following steps are involved: S101, obtaining display screen image signal data information, and performing image cropping and annotation processing; S102. Build a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as the baseline network, and perform feature extraction on the image cropping and annotation processing results; S103, performing target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network; S104, performing selective contrast loss calculation processing on the features enhanced by multi-scale features of the main network to improve the distance between classes; S105. Perform feature decoupling processing on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to solve the mutual interference between classification and regression.

2. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1, characterized in that: The method of obtaining display screen image signal data information and performing image cropping and annotation processing includes the following steps: Adjust the camera's spatial position, focal length, aperture and other factors to avoid moiré and liquid crystal patterns; Select different backgrounds in a darkroom environment and collect images of the display quality of the display screen; Selecting display screen images with different types of defects on the display screen, and further screening to select more representative display screen images as a data set; Perform image cropping processing on the effective feature area of ​​the display screen image; The defective area of ​​the cropped display screen image is marked.

3. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1, characterized in that: The method of constructing a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network and performing feature extraction processing on the image cropping and annotation processing results includes the following steps: ResNet101 is used as the backbone network to extract multi-scale feature maps of display screen images, and then the feature map information from different scales is integrated through the multi-level bidirectional feature fusion mechanism of the bidirectional feature pyramid; The candidate region generation network predicts potential target regions based on the fused feature maps and generates a set of predefined anchors on feature maps of different scales. The anchors have different sizes and aspect ratios to accommodate targets of different sizes and shapes. For each candidate region, the ROIAlign operation is used to extract fixed-size features from the multi-scale feature map output by BiFPN for subsequent classification and bounding box regression, and then the candidate regions are aligned by ROI Align. Based on the ROI detection head, each candidate region is classified and regressed into a bounding box, and finally the target detection result is output through non-maximum suppression.

4. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1, characterized in that: The target and feature multi-scale enhancement processing of the feature extraction processing result by the auxiliary branch network comprises the following steps: The main network of the feature extraction network and the feature fusion network in the auxiliary branch network adopts the same architecture, and the parameters are shared between the two networks; Generate k objects of different scales through multi-scale targets; By generating feature maps based on the targets of each scale in the target pyramid, the matching is further refined through RPN and ROI detection; The auxiliary branches are removed after training is completed so as not to affect the inference time.

5. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1, characterized in that: The selective contrast loss calculation process of the features enhanced by the multi-scale features of the main network to improve the inter-class distance includes the following steps: Improve the discrimination between defect classes based on selective contrast loss function, and perform detection, classification and regression in parallel through regions of interest; For a small batch of N RoI box features First, the candidate region features Use a layer of multi-layer perceptron for encoding z i =MLP(f i ), embedding vector z i and z j The distance d(z i ,z j ) to measure the similarity of sample pairs; When the overlap between the candidate region and the true annotation box is greater than the set threshold u, the sample pair will participate in the calculation of the contrast loss. The expression of the loss function is as follows: in, is the loss function, u is set to 0.7 in this example, and f j is the candidate region feature, y j is the target real box, L(W,(Y,f i ,f j ) is the loss calculation, W is the model parameter, Y represents the similarity or dissimilarity, and IOU is the overlap of the real annotation box; The loss calculation method L(W,(Y,f i ,f j ) The contrastive learning calculation method is used to compare and calculate the high confidence vector. The expression of vector comparison calculation is as follows: in, and represents the normalized feature vector, m represents the minimum distance between the same sample pairs, y = 1 represents that the sample pairs are heterogeneous, thus forcing the distance between samples of different categories to be greater than m, and y = 0 represents that the sample pairs are homogeneous sample pairs, then the actual distance between them is calculated. is the distance between candidate region f1 and candidate region f2 in the feature space. The expression for calculating the Euclidean distance is as follows: d(z i ,z j )=||z i -z j ||2 Where d is the Euclidean distance, z i and z j denote the i-th and j-th embedding vectors respectively.

6. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 1, characterized in that: The method of performing feature decoupling processing on the target and feature multi-scale enhancement processing results based on the adaptive decoupling module to solve the mutual interference between classification and regression includes the following steps: By using different convolutions to replace the original shared convolutions or full connections, the multi-scale enhancement processing results of the target features are decoupled. The variant convolution is used to replace the conventional convolution, and the dynamic snake convolution SnakeConv irregular feature extraction method is used for classification. Enhance the model's ability to detect slender and circuitous structures; for regression, coordinate convolution CoordConv is used to extract features while recording position coordinates to obtain Better target box regression; The decoupled features of classification and regression are then processed through parallel spatial attention. With channel attention The features are enhanced, and then classified and regressed through the polarization function ψ(.); The characteristic expression after polarization is as follows: M c =σ(C 1×1 (relu(C 1×1 (avgpool(f))))) M s =σ(C 7×7 {[maxpool(F);avgpool(f)])) Among them, M s is the spatial attention, M c is the channel attention, M f is the feature map after spatial and channel enhancement, F' is the output feature after the polarization function, σ() is the activation function, C is a 1×1 convolution, avgpool() is the average pooling, and ψ() is the polarization function.

7. The multi-scale enhancement and selective contrast LCD defect detection method according to any one of claims 1 to 6, characterized in that: Also includes: During the model training process, the following steps are involved in converting a large dataset into small samples for base class training: The large dataset is the VOC dataset, which is a natural dataset that includes 17,125 samples in 20 categories. In base class training, all 20 categories in the VOC dataset are used as base classes, and random sampling without replacement is performed from the base classes in the form of {1, 2, 3, 5, 10, 30} samples for training; During training, data is augmented by flipping, rotating, and cropping the data.

8. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 7, characterized in that: It also includes fine-tuning the network parameters trained on the base class through display defect data.

9. The multi-scale enhancement and selective contrast LCD defect detection method according to claim 8, characterized in that: The fine-tuning of the network parameters trained in the base class by using the display screen defect data includes the following steps: In the new class fine-tuning, the five categories of point, line, mura, scratch, and cylinder in the LCD defect dataset are taken as new classes, and the network is fine-tuned using the {1, 2, 3, 5, 10, 30} sample format; During the fine-tuning process, since shallow features such as edges and textures are universal, the parameters of the first stage of the backbone network ResNet101 are frozen without adjustment.

10. A multi-scale enhancement and selective contrast LCD defect detection system, used to execute the multi-scale enhancement and selective contrast LCD defect detection method according to any one of claims 1 to 9, characterized in that: include: A data acquisition unit, used to acquire display screen image signal data information; The first processing unit is used to obtain the display screen image signal data information and perform image cropping and annotation processing; The model building unit is used to build a two-stage object detection model based on Faster R-CNN combined with a bidirectional feature pyramid; The second processing unit is used to use a two-stage target detection model based on Faster R-CNN combined with a bidirectional feature pyramid as a baseline network, and perform feature extraction processing on the image cropping and annotation processing results; The third processing unit is used to perform target and feature multi-scale enhancement processing on the feature extraction processing result through the auxiliary branch network; A fourth processing unit is used to perform selective contrast loss calculation processing on the features enhanced by the multi-scale features of the main network to improve the inter-class distance; A fifth processing unit, configured to perform feature decoupling processing on the target feature multi-scale enhancement processing result based on the adaptive decoupling module to resolve the mutual interference between classification and regression; The sixth processing unit is used to process the large data set into a small sample form for base class training during the model training process; The seventh processing unit is used to fine-tune the network parameters trained by the base class through the display screen defect data.

Images