Defect detection method and device and computer storage medium

Abstract

The invention discloses a defect detection method and device and a computer storage medium, and belongs to the technical field of visual detection. The method comprises the following steps: inputting a to-be-detected image into a target detection model; and obtaining defect information output by the target detection model. The obtaining process of the target detection model comprises the steps of establishing a pre-training model comprising a pre-training feature extraction network, training the pre-training model by adopting an unsupervised comparative learning strategy to obtain pre-training parameters, then loading the pre-training parameters into a first adjustment model, training the first adjustment model, and obtaining a target detection result. And training the first student network in the first adjustment model twice in sequence to obtain a target detection model. In the pre-training process, the model can learn general characteristics in a large amount of data, and in the target detection task, the model can utilize the general characteristics to improve the performance of the model, so that the effect of improving the accuracy of the defect detection method can be realized.

Description

Defect detection methods, devices, and computer storage media Technical Field

[0001] This application relates to the field of visual inspection technology, and in particular to a defect detection method, apparatus, and computer storage medium. Background Technology

[0002] Defect detection technology refers to the technology of detecting defects on the surface of an object using machine vision inspection technology.

[0003] Current defect detection methods first train a target detection model using a large number of labeled samples to obtain a trained target detection model. Then, an image of the target object to be detected is acquired, the image is input into the target detection model, and the defect detection result output by the target detection model is obtained.

[0004] However, the accuracy of the above-mentioned defect detection methods may be low.

[0005] Summary of the Invention

[0006] This application provides a defect detection method, apparatus, and computer storage medium. The technical solution is as follows:

[0007] According to one aspect of the embodiments of this application, a defect detection method is provided, the method comprising:

[0008] Acquire an image to be detected, wherein the image to be detected includes an image acquired by an image acquisition component that includes at least one target object to be detected;

[0009] Input the image to be detected into the target detection model;

[0010] Obtain the defect detection information output by the target detection model;

[0011] The process of acquiring the target detection model includes:

[0012] Based on the structure of the target detection model, a pre-trained model including a pre-trained feature extraction network is established;

[0013] The pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the first pre-trained parameters of the pre-trained feature extraction network;

[0014] A first adjustment model is established, which includes a first student network, a teacher network, and a loss module.

[0015] The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network;

[0016] Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network;

[0017] The first adjustment model is trained, and the first student network becomes the second student network after training.

[0018] Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0019] Optionally, the step of training the pre-trained model using an unsupervised contrastive learning strategy to obtain the first pre-trained parameters of the pre-trained feature extraction network includes:

[0020] Obtain an unlabeled dataset, which includes multiple unlabeled samples, each of which includes a first image of a target object;

[0021] Multiple batches of training data were obtained based on the unlabeled dataset;

[0022] The pre-trained model is trained based on the multiple batches of training data;

[0023] After the training cutoff condition is met, the target pre-trained model is obtained;

[0024] Obtain the first pre-training parameters of the pre-trained feature extraction network in the target pre-trained model;

[0025] The process of obtaining training data from one batch of the multiple batches of training data includes:

[0026] Obtain n unlabeled samples from the unlabeled dataset, where n is a positive integer greater than 1;

[0027] One target sample from the n unlabeled samples is augmented to obtain an augmented sample;

[0028] Obtain the similarity between the target sample and the n-1 unlabeled samples (excluding the target sample) among the n unlabeled samples;

[0029] The enhanced samples and the unlabeled samples with a similarity greater than or equal to a preset threshold are identified as positive samples;

[0030] Unlabeled samples with similarity less than the preset threshold are identified as negative samples. The training data in a batch includes the augmented samples, positive samples from the n unlabeled samples, and negative samples.

[0031] Optionally, the pre-trained model includes a pre-trained loss module, a main path network, and a momentum path network. The main path network and the momentum path network have the same structure. The main path network includes the pre-trained feature extraction network and a linear classifier that are interconnected. The linear classifier of the main path network and the linear classifier of the momentum path network are respectively connected to the pre-trained loss module.

[0032] Optionally, obtaining the unlabeled dataset includes:

[0033] Acquire images of multiple target objects;

[0034] The images of the multiple target objects are segmented using a sliding window to obtain the multiple unlabeled samples.

[0035] Optionally, before training the second student network, the method further includes:

[0036] A labeled dataset is obtained, which includes multiple labeled samples. Each labeled sample includes a second image of a target object and label information, wherein the label information is used to identify defects of the target object.

[0037] The training of the second student network includes:

[0038] The hyperparameters of the training are preset and adjusted to obtain the adjusted hyperparameters. The preset adjustment includes at least one of adjusting the learning rate decrease rate to a first rate and adjusting the learning rate to a first learning rate. The first rate is a learning rate decrease rate that is less than a second rate. The first learning rate is a learning rate that is less than a second learning rate. The learning rate decrease rate of the second rate and the second learning rate are the hyperparameters used to train the first adjusted model.

[0039] The second student network is trained using the labeled dataset and the adjusted hyperparameters.

[0040] Optionally, training the first adjustment model includes:

[0041] Obtain multiple second images from multiple labeled samples in the labeled dataset;

[0042] The multiple second images are processed into multiple pseudo-label samples using a selective search algorithm. The pseudo-label samples include the second images and pseudo-labels of the second images that contain region resolution capabilities.

[0043] The first adjustment model is trained based on the multiple pseudo-label samples.

[0044] Optionally, the first student network and the teacher network have the same structure as the target detection model and are respectively connected to the loss module. The establishment of the first adjustment model includes:

[0045] A data upscaling module is provided, which includes an input end and an output end. The data upscaling module is used to process the image to be detected input by the input end into an image with the same resolution as the second image, and output it from the output end.

[0046] A first object detection network is established, which includes a feature extraction network to be trained, wherein the first layer of the feature extraction network to be trained is a first convolutional layer.

[0047] Adjust the first convolutional layer to match the data dimensionality enhancement module;

[0048] The output of the data upscaling module is connected to the first convolutional layer to form the first student network;

[0049] The first adjustment model is obtained based on the first student network and the loss module, and the number of regression categories of the teacher network and the first student network is set to 1.

[0050] Optionally, before pre-adjusting the hyperparameters during training to obtain the adjusted hyperparameters, the method further includes:

[0051] Obtain the number of target regression categories;

[0052] Adjust the number of regression categories in the second student network to the target number of regression categories.

[0053] Optionally, the establishment of a pre-trained model based on the structure of the target detection model, including a pre-trained feature extraction network, includes:

[0054] A pre-training data upscaling module is provided, which includes an input end and an output end. The pre-training data upscaling module is used to process the image to be detected input by the input end into an image with the same resolution as the second image, and output it from the output end.

[0055] Obtain the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network;

[0056] The pre-trained model is obtained based on the pre-trained data dimensionality enhancement module, the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network. The pre-trained model includes the pre-trained loss module, the main path network, and the momentum path network. The main path network and the momentum path network have the same structure. The main path network includes the pre-trained data dimensionality enhancement module, the pre-trained feature extraction network, and the linear classifier connected in sequence. The linear classifier of the main path network and the linear classifier of the momentum path network are respectively connected to the pre-trained loss module.

[0057] The unsupervised contrastive learning strategy is used to train the pre-trained model to obtain the first pre-trained parameters of the pre-trained feature extraction network, including:

[0058] The pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the first pre-training parameters of the pre-trained feature extraction network and the second pre-training parameters of the pre-trained data dimensionality enhancement module.

[0059] The step of loading the pre-trained parameters into the feature extraction networks of the first student network and the teacher network includes:

[0060] The first pre-training parameters and the second pre-training parameters are loaded into the first student network and the teacher network.

[0061] Optionally, the first student network includes the data dimensionality enhancement module, the feature extraction network, the feature pyramid network, the candidate box extraction network, and the prediction network, which are connected in sequence.

[0062] Optionally, freezing the gradients of the parameters of the feature extraction network in the first student network and the gradients of the parameters of the teacher network includes:

[0063] Set the gradients of the parameters of the feature extraction network in the first student network and the parameters of the teacher network to false;

[0064] The step of unfreezing the parameters of the feature extraction network in the second student network includes:

[0065] Set the gradient values ​​of the parameters of the feature extraction network in the second student network to true.

[0066] Optionally, the parameters in the teacher network are iterated by determining the parameters in the teacher network based on a preset formula, which includes: tn=sn*x+tn'*(1-x);

[0067] Wherein, tn is the parameter n of the teacher network corresponding to the current iteration process, sn is the parameter n of the first student network corresponding to the current iteration process, x is the preset smoothing coefficient, and tn' is the parameter n of the teacher network corresponding to the previous iteration process of the current iteration process. The parameter n is one of the parameters of the teacher network and the first student network.

[0068] Optionally, the establishment of a pre-trained model based on the structure of the target detection model, including a pre-trained feature extraction network, includes:

[0069] Obtain the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network;

[0070] The pre-trained model is obtained based on the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network. The pre-trained model includes the pre-trained loss module, the main path network, and the momentum path network. The main path network and the momentum path network have the same structure. The main path network includes the pre-trained feature extraction network and the linear classifier that are interconnected. The linear classifier of the main path network and the linear classifier of the momentum path network are respectively connected to the pre-trained loss module.

[0071] According to another aspect of the embodiments of this application, a defect detection device is provided, the defect detection device comprising:

[0072] The acquisition module is used to acquire an image to be detected, wherein the image to be detected includes an image containing at least one target object acquired by the image acquisition component;

[0073] The input module is used to input the image to be detected into the target detection model;

[0074] The acquisition module is used to acquire the defect detection information output by the target detection model;

[0075] The process of acquiring the target detection model includes:

[0076] Based on the structure of the target detection model, a pre-trained model including a pre-trained feature extraction network is established; the pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the pre-trained parameters of the pre-trained feature extraction network.

[0077] A first adjustment model is established, which includes a first student network, a teacher network, and a loss module.

[0078] The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network;

[0079] Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network;

[0080] The first adjustment model is trained, and the first student network becomes the second student network after training.

[0081] Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0082] According to another aspect of the embodiments of this application, a defect detection device is provided, the defect detection device including a control component and an image acquisition component;

[0083] The control component is used to acquire an image of at least one target object collected by the image acquisition component;

[0084] The control component is equipped with a target detection model, and the control component is also used to input the image to be detected into the target detection model and obtain the defect detection information output by the target detection model;

[0085] The process of acquiring the target detection model includes:

[0086] Based on the structure of the target detection model, a pre-trained model including a pre-trained feature extraction network is established; the pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the pre-trained parameters of the pre-trained feature extraction network.

[0087] A first adjustment model is established, which includes a first student network, a teacher network, and a loss module.

[0088] The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network;

[0089] Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network;

[0090] The first adjustment model is trained, and the first student network becomes the second student network after training.

[0091] Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0092] According to another aspect of the embodiments of this application, a defect detection device is provided, the defect detection device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the defect detection method as described above.

[0093] According to another aspect of the embodiments of this application, a non-volatile computer storage medium is provided, wherein the non-volatile computer storage medium stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the defect detection method as described above.

[0094] The beneficial effects of the technical solutions provided in this application include at least the following:

[0095] A pre-trained model, including a pre-trained feature extraction network, is established based on the structure of the object detection model. An unsupervised contrastive learning strategy is used to train the pre-trained model to obtain pre-training parameters. These pre-training parameters are then loaded into a first adjustment model, which is trained. The first student network in the first adjustment model is then trained twice to obtain the object detection model. This model can then be used to detect defects in the image. Because the model learns a large number of general features from the data during pre-training, these general features can be used to improve the model's performance in the object detection task, thus improving the accuracy of the defect detection method. Attached Figure Description

[0096] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0097] Figure 1 is a schematic diagram of a defect detection device provided in an embodiment of this application;

[0098] Figure 2 is a flowchart illustrating a defect detection method according to an embodiment of this application;

[0099] Figure 3 is at least a partial flowchart of another defect detection method provided in an embodiment of this application;

[0100] Figure 4 is at least a partial flowchart of another defect detection method provided in an embodiment of this application;

[0101] Figure 5 is a schematic diagram of the structure of a pre-trained model in an embodiment of this application;

[0102] Figure 6 is a schematic diagram of obtaining training data in an embodiment of this application;

[0103] Figure 7 is a schematic diagram of the structure of a first adjustment model provided in an embodiment of this application;

[0104] Figure 8 is a flowchart of training a first adjustment model according to an embodiment of this application;

[0105] Figure 9 is a flowchart of training a second student network according to an embodiment of this application;

[0106] Figure 10 is at least a partial flowchart of another defect detection method provided in an embodiment of this application;

[0107] Figure 11 is a schematic diagram of a pre-trained model provided in an embodiment of this application;

[0108] Figure 12 is a block diagram of a defect detection device provided in an embodiment of this application.

[0109] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0110] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0111] The following describes the application scenarios of the defect detection method provided in the embodiments of this application.

[0112] The defect detection method provided in this application can be applied to various scenarios to detect defects in various target objects, including various electronic devices, mechanical devices, parts, and components. In an exemplary embodiment, the target object can be a display panel, and the defect detection method provided in this application can be applied to detect defects on the display surface of the display panel to determine whether defects exist on the display surface and their location. These defects can be scratches, foreign objects, or other defects on the display surface, which seriously affect the display effect of the display panel. For example, the defect detection method provided in this application can perform defect detection on the display panel before it leaves the factory, or it can perform defect detection on the display device including the display panel before it leaves the factory, or it can perform defect testing on the display device after it leaves the factory, so as to determine whether defects exist on the display surface and, if so, the location of the defects.

[0113] When applied in industrial scenarios, the method provided in this application embodiment can perform defect detection on the display panel before it leaves the factory, and can also perform defect detection on the display motherboard including multiple uncut display panels, thus improving the speed of defect detection.

[0114] The display device involved in this application embodiment includes a display panel and some devices that cooperate with the display panel (such as control components, power supply, and housing). The display panel includes a display surface and a back surface, and the display surface has a display area for displaying images. The display surface of the display panel is the display surface of the display device. During the manufacturing process of the display panel and the display device, defects may occur in the display surface (e.g., the display area and the surrounding area outside the display area) due to various reasons (such as the yield rate of the manufacturing process and some unexpected reasons).

[0115] Figure 1 is a schematic diagram of a defect detection device provided in an embodiment of this application. The defect detection device may include a control component 11.

[0116] The control component 11 can be a terminal, or it can be a server or a server cluster. The control component 11 may be equipped with a target detection model, which can be used to perform defect inspection on images including at least one display panel.

[0117] Optionally, the defect detection device may further include an image acquisition component 12, which is capable of establishing a wired or wireless connection with the control component 11. The image acquisition component 12 may include a camera for capturing images including the target object to be detected.

[0118] Figure 2 is a flowchart illustrating a defect detection method according to an embodiment of this application. This embodiment uses the application of this method in the control component shown in Figure 1 as an example. The defect detection method may include the following steps:

[0119] Step 201: Acquire the image to be detected. The image to be detected includes an image acquired by the image acquisition component that includes at least one target object to be detected.

[0120] Step 202: Input the image to be detected into the target detection model.

[0121] Step 203: Obtain the defect detection information output by the target detection model.

[0122] The defect detection information may include defect information of at least one defect of the target object to be detected (the defect information may include the location of the defect and the detection frame surrounding the defect, for example, the location of at least one defect of the display surface of the display panel and the detection frame surrounding the defect), or it may be information about no defects detected.

[0123] The process of acquiring the target detection model includes:

[0124] 1) Establish the first adjustment model, which includes the first student network, the teacher network, and the loss module;

[0125] 2) Load the first pre-trained parameters into the feature extraction networks of the first student network and the teacher network;

[0126] 3) Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network;

[0127] 4) The first adjustment model is trained, and the first student network becomes the second student network after training;

[0128] 5) Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0129] The control component can obtain the target detection model through steps 1) to 5) above, before step 201.

[0130] In summary, the defect detection method provided in this application establishes a pre-trained model, including a pre-trained feature extraction network, based on the structure of the target detection model. An unsupervised contrastive learning strategy is then used to train the pre-trained model to obtain pre-training parameters. These pre-training parameters are then loaded into a first adjustment model, which is trained. The first student network in the first adjustment model is then trained twice to obtain the target detection model. This target detection model can then be used to detect defects in the image to be detected. Because the model can learn a large number of general features from the data during pre-training, these general features can be used to improve the model's performance in the target detection task, thus improving the accuracy of the defect detection method.

[0131] Figure 3 is at least a partial flowchart of another defect detection method provided in an embodiment of this application. Before step 201 of the defect detection method shown in Figure 2, the defect detection method may further include the following steps:

[0132] Step 301: Based on the structure of the object detection model, establish a pre-trained model including a pre-trained feature extraction network.

[0133] Step 302: Train the pre-trained model using an unsupervised contrastive learning strategy to obtain the first pre-trained parameters of the pre-trained feature extraction network.

[0134] Step 303: Establish the first adjustment model, which includes the first student network, the teacher network, and the loss module.

[0135] Optionally, the first student network and the teacher network have the same structure as the object detection model and are respectively connected to the loss module.

[0136] Step 304: Load the first pre-trained parameters into the feature extraction networks of the first student network and the teacher network.

[0137] Step 305: Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network.

[0138] Optionally, the parameters in the teacher network are iterated by determining the corresponding parameters in the teacher network based on the parameters in the first student network.

[0139] Step 306: Train the first adjustment model. After training, the first student network becomes the second student network.

[0140] Step 307: Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0141] Steps 301 to 307 describe a method for obtaining a target detection model in a defect detection method.

[0142] In summary, the defect detection method provided in this application establishes a pre-trained model, including a pre-trained feature extraction network, based on the structure of the target detection model. An unsupervised contrastive learning strategy is then used to train the pre-trained model to obtain pre-training parameters. These pre-training parameters are then loaded into a first adjustment model, which is trained. The first student network in the first adjustment model is then trained twice to obtain the target detection model. This target detection model can then be used to detect defects in the image to be detected. Because the model can learn a large number of general features from the data during pre-training, these general features can be used to improve the model's performance in the target detection task, thus improving the accuracy of the defect detection method.

[0143] Figure 4 is at least a partial flowchart of another defect detection method provided in an embodiment of this application. Before step 201 of the defect detection method shown in Figure 2, the defect detection method may further include the following steps:

[0144] Step 401: Obtain the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network.

[0145] When applying the method provided in the embodiments of this application, the control component can acquire a linear classifier, a pre-trained loss module, and a pre-trained feature extraction network.

[0146] The linear classifier may include a neck network, which may consist of two cascaded linear layers (or more cascaded linear layers, which are not limited in this embodiment). The output of the neck network is a feature vector of a preset target feature dimension. The pre-trained feature extraction network may include a backbone network, which may be a ResNet50 network. The output of the backbone network is a feature matrix (the dimension of the feature matrix may be a preset dimension, such as 2048 dimensions). The pre-trained loss module, also known as the pre-trained loss calculation module, is used to calculate the loss value. The pre-trained loss module may include a head network. The head network can iterate the network parameters by calculating the cross-entropy loss.

[0147] Step 402: Obtain the pre-trained model based on the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network.

[0148] The control component can construct a pre-trained model based on the linear classifier, pre-trained loss module and pre-trained feature extraction network obtained in step 401. This pre-trained model can be used to pre-train some parameters. Pre-training has the effects of accelerating the training speed of the object detection model and improving the performance of the object detection model.

[0149] Please refer to Figure 5, which is a schematic diagram of the structure of a pre-trained model in an embodiment of this application. The pre-trained model includes a pre-training loss module 51, a main path network 52, and a momentum path network 53. The main path network 52 and the momentum path network 53 have identical structures. The main path network 52 includes an interconnected pre-trained feature extraction network and a linear classifier. The linear classifiers of the main path network 52 and the momentum path network 53 are respectively connected to the pre-training loss module 51. During the training of this pre-trained model, the parameters of the main path network 52 and the momentum path network 53 can be updated through momentum updates.

[0150] The pre-trained model is structured as a Momentum Contrast v2 (MoCo-v2) model, which can be applied to self-supervised learning, a type of unsupervised learning. For details on MoCo-v2, please refer to relevant technologies; these will not be elaborated upon here. Alternatively, model structures based on Simple Contrast Learning Representation (simclr) (an unsupervised contrastive learning method that maximizes the feature similarity between two different augmented versions of a positive sample and minimizes the feature similarity between augmented versions from all different samples) or other unsupervised learning models can be established, and the pre-trained model can be trained using the corresponding algorithms. This application does not impose any limitations on this approach.

[0151] In this system, the pre-trained feature extraction network in the main path network 52 is used to receive images from the training samples, while the pre-trained feature extraction network in the momentum path network 53 is used to receive enhanced images of the training samples. The enhanced image is the image obtained after performing enhancement processing on the original image.

[0152] When the defect detection method provided in this application is applied to an industrial scenario, the enhancement processing may include at least one of the following processing methods: random cropping, random contrast enhancement, random brightness enhancement, Gaussian blurring, random erasure, random grayscale conversion, and scale scaling.

[0153] Step 403: Obtain the unlabeled dataset.

[0154] The unlabeled dataset includes multiple unlabeled samples. Each unlabeled sample includes a first image of a target object. The target object and the target object to be detected are of the same type. For example, if the target object to be detected is a display panel, then the target object is also a display panel. In this case, the first image can be an image that includes at least a portion of the display surface of the display panel.

[0155] The process of obtaining an unlabeled dataset may include:

[0156] 1. Acquire images of multiple target objects.

[0157] When the target object is a display panel, the images of multiple display panels can be composed of images of multiple display master panels. Each display master panel image includes images of multiple display panels, and each display panel image is an image including the display surface of the display panel. Of course, when the target object is other structures, such as various plates, lenses, and films, the control component can also acquire corresponding images. This application embodiment does not impose any limitations on this.

[0158] The control component can acquire images of the display panel by taking pictures of the display panel with the image acquisition component. For example, the image acquisition component can take pictures of the display motherboard to acquire an image including the display motherboard. The image of the display motherboard can include images of the display areas of multiple display panels and some lines around the display areas, so that defects can be detected in these areas.

[0159] In addition, the control component can also acquire display panel images (such as images of display master boards) from existing databases of the same type to expand the multiple display panel images, thereby improving the efficiency of image acquisition in the embodiments of this application.

[0160] 2. Segment the images of multiple target objects using a sliding window to obtain multiple unlabeled samples.

[0161] The control component can segment images of multiple target objects using a sliding image slicing method to obtain multiple unlabeled samples. The specific process of sliding image slicing can include:

[0162] 2.1 Set parameters such as target window size and window overlap, and calculate the horizontal and vertical step lengths of the window based on the window overlap.

[0163] 2.2 Start the target window from the top left corner of the target object's image and slice the image based on the area where the target window is located to obtain an unlabeled sample;

[0164] 2.3 Move the target window to the next position according to the horizontal step size, and slice the image to obtain an unlabeled sample;

[0165] 2.4 Repeat step 2.3 until the window reaches the upper right corner of the target object's image;

[0166] 2.5 Move the target window vertically by the step size, reset the horizontal position of the target window to the left side of the image and crop the image to obtain an unlabeled sample;

[0167] 2.6 Repeat steps 2.3 to 2.5 until the window reaches the lower right corner of the image, thus completing the image traversal of the entire target object.

[0168] The control component can construct an unlabeled dataset from multiple unlabeled samples obtained through a sliding tiling method. In a specific example, the image captured by the terminal device can be a 3840*3840 display master image. After segmenting the display master image through a sliding window, multiple 640*640 resolution images can be obtained, which can constitute an unlabeled dataset.

[0169] The unlabeled dataset obtained in this way is a dataset with a small number of samples.

[0170] Step 404: Obtain training data in multiple batches based on the unlabeled dataset.

[0171] After obtaining the unlabeled dataset, the control component can then acquire multiple batches of training data based on the unlabeled dataset.

[0172] In one exemplary embodiment, the control component may employ conditional contrastive learning to train the pre-trained model. In this case, please refer to Figure 6, which is a schematic diagram illustrating the acquisition of training data in an embodiment of this application. The process by which the control component acquires one batch of training data from multiple batches includes:

[0173] Sub-step 4041: Obtain n unlabeled samples from the unlabeled dataset.

[0174] The control component can first select n unlabeled samples from a plurality of unlabeled samples in the unlabeled dataset, where n is a positive integer greater than 1. The control component can randomly select n unlabeled samples, select n unlabeled samples sequentially according to the order of unlabeled samples in the unlabeled dataset, or select n unlabeled samples according to other rules; this embodiment does not impose any restrictions on this. Furthermore, since the unlabeled samples are selected from the unlabeled dataset, n is less than the total number of unlabeled samples in the unlabeled dataset.

[0175] Sub-step 4042: Enhance one of the target samples from the n unlabeled samples to obtain the enhanced sample.

[0176] The control component can select a target sample from n unlabeled samples (randomly or according to preset rules) and perform enhancement processing on the target sample to obtain an enhanced sample of the target sample. The enhancement processing methods may include at least one of the following: random cropping, random contrast enhancement, random brightness enhancement, Gaussian blur, random erasure, random grayscale conversion, and scale scaling.

[0177] Sub-step 4043: Obtain the similarity between the target sample and the n-1 unlabeled samples (excluding the target sample) among the n unlabeled samples.

[0178] The control component can determine the similarity between the target sample and the n-1 unlabeled samples (excluding the target sample) out of n unlabeled samples. This similarity is the similarity of images; for example, this similarity can be the Structural Similarity Index Measure (SSIM), which is a metric used to measure the similarity between two images.

[0179] Sub-step 4044: Define the enhanced samples and unlabeled samples with similarity greater than or equal to a preset threshold as positive samples, and define unlabeled samples with similarity less than a preset threshold as negative samples.

[0180] The control component can determine whether each of the n unlabeled samples is a positive or negative sample based on the similarity determined in the above steps. In this step, the control component identifies augmented samples and unlabeled samples with a similarity greater than or equal to a preset threshold as positive samples, and identifies unlabeled samples with a similarity less than the preset threshold as negative samples. This preset threshold can be an empirical value; for example, when the similarity is the SSIM index, the preset threshold can range from 0.5 to 1.

[0181] When the target object is a display panel, a batch of training data includes augmented samples (which are also positive samples), positive samples from n unlabeled samples, and negative samples. By obtaining positive and negative samples through whether the similarity is greater than a preset threshold, the effect of conditional contrastive learning can be achieved when training the pre-trained model. This can alleviate the problem of feature bias caused by contrastive learning mistakenly classifying similar display panel images as negative samples.

[0182] Currently, in industrial scenarios, the characteristics of industrial scene images are somewhat incompatible with contrastive learning. In unsupervised contrastive learning, the image of the display master is processed through random cropping to obtain multiple samples. Unsupervised contrastive learning guides the pre-trained model to learn effective feature extraction capabilities based on the similarity between samples and local views or enhanced views within the samples, as well as the differences between positive and negative samples. However, samples generated based on random cropping or their enhanced images reflect, to a certain extent, only the main body of the image (the main body can be understood as the display area). The image of the display master contains a small amount of boundary information and a large amount of information similar to display areas. Therefore, algorithms such as MoCo-v2 and Simclr cause the contrastive learning process to focus more on the noise information of the display area in the image of the display master, thus failing to learn effective feature extraction capabilities. The conditional contrastive learning method provided in this application avoids this problem, enabling the pre-trained model to learn effective feature extraction capabilities.

[0183] Step 405: Train the pre-trained model based on multiple batches of training data to obtain the first pre-trained parameters of the pre-trained feature extraction network.

[0184] After acquiring the training data from multiple batches, the control component can train the pre-trained model based on this data to obtain the first pre-trained parameters of the pre-trained feature extraction network. When the pre-trained feature extraction network is a backbone network, the first pre-trained parameters include the parameters of that backbone network. The deadline for training the pre-trained model can include reaching a preset number of training epochs (e.g., 200 to 800). One training epoch can refer to inputting multiple batches of training data obtained from an unlabeled dataset into the pre-trained model and iterating the parameters within the pre-trained model. Through pre-training, the general feature representations learned on large-scale data can be used as initialization parameters, accelerating the model's training process on feature tasks. This is because the pre-trained parameters are already close to optimal and have captured some general patterns in the input data, making the optimization process on the target task easier to converge.

[0185] In one exemplary embodiment, the control component uses the training data acquisition process shown in Figure 5 to acquire training data. Then, in step 405, the control component can perform conditional contrastive learning on the pre-trained model based on this training data to improve the training effect of the pre-trained model. In a specific example, some hyperparameters used in training the pre-trained model can be set as follows: feature dimension is 128, temporary queue length = batch size * integer multiple of feature dimension, and temporary queue length is no greater than 65536.

[0186] Steps 401 to 405 can be the pre-training stage performed in the embodiments of this application, and the first stage of coarse adjustment can be continued thereafter.

[0187] Step 406: Obtain the data upgrade module.

[0188] The data upscaling module includes an input end and an output end. It processes the input image to be detected into an image with the same resolution as the second image and outputs it from the output end. In an exemplary embodiment, the data upscaling module includes multiple cascaded convolutional layers, which can convert a large input image into a collection of high-dimensional, small-sized data. For example, one data upscaling module includes two convolutional layers: one with a 3-channel input and a 64-channel output, a 3*3 kernel size, and a stride of 2; the other with a 64-channel input and output, a 3*3 kernel size, and a stride of 34. The downsampling factor of this data upscaling module is 6. Of course, in this embodiment, the data upscaling module can have other structures, and this embodiment does not further limit this.

[0189] Step 407: Establish a first object detection network, which includes a feature extraction network to be trained.

[0190] The first object detection network is an object detection network that includes, in sequence, a feature extraction network to be trained, a feature pyramid network (FPN), a region proposal network (RPN) for extracting candidate boxes, and a prediction network. The prediction network may include a head network; for example, the prediction network may include a category prediction head network (ResNet head) and a ROI head network for classifying candidate boxes and regressing bounding boxes. The category prediction head and the ROI head are respectively connected to the RPN. This type of first object detection network is a faster region convolutional neural network (faster-RCN).

[0191] In addition, the sequential connection of multiple networks involved in the embodiments of this application can mean that these multiple networks are connected sequentially according to the order in which they appear in the statement. For example, the sequential connection of network A, network B, and network C means that network A is connected to network B, and network B is connected to network C.

[0192] Step 408: Adjust the first convolutional layer of the feature extraction network to be trained so that the first convolutional layer matches the data dimensionality enhancement module.

[0193] The control component can adjust the number of input channels of the first convolutional layer of the feature extraction network to be trained (the number of input channels can be adjusted by adjusting the number of convolutional kernels) so that the number of input channels of the first convolutional layer is equal to the number of output channels of the data upscaling module (the number of output channels can be the number of output channels of the last convolutional layer of the data upscaling module), so that the first convolutional layer can be matched with the data upscaling module.

[0194] Step 409: Connect the output of the data upscaling module to the first convolutional layer to form the first student network.

[0195] After the data dimensionality enhancement module is matched with the first convolutional layer of the feature extraction network to be trained, the control component can connect the output of the data dimensionality enhancement module to the first convolutional layer to form a first student network. This first student network includes, in sequence, the data dimensionality enhancement module, the feature extraction network, the feature pyramid network, the candidate box extraction network, and the prediction network.

[0196] Step 410: Obtain the first adjustment model based on the first student network and the loss module.

[0197] The first adjustment model includes a first student network, a teacher network, and a loss module. The structures of the first student network and the teacher network are the same as those of the object detection model and are respectively connected to the loss module.

[0198] In this step, the process by which the control component obtains the first adjusted model includes:

[0199] 1. The control component establishes a teacher network with a structure consistent with the first student network.

[0200] The first student network is also called the student branch, and the teacher network is also called the teacher branch.

[0201] 2. The control component connects the teacher network and the first student network to the loss module, and sets the number of regression categories for the teacher network and the first student network to 1 to obtain the first adjusted model. The coarse adjustment in the first stage does not involve categories, so the number of regression categories for the teacher network and the first student network can be set to 1.

[0202] Please refer to Figure 7, which is a schematic diagram of the structure of a first adjustment model provided in an embodiment of this application. The first adjustment model includes a first student network 71, a teacher network 72, and a loss module 73. Both the first student network 71 and the teacher network 72 include, in sequence, a data dimensionality enhancement module, a feature extraction network, a feature pyramid network, a candidate box extraction network, and a prediction network. The prediction networks of the first student network 71 and the teacher network 72 are respectively connected to the loss module 73. The loss module 73 is used to calculate the loss value based on the outputs of the first student network 71 and the teacher network 72.

[0203] Step 411: Load the first pre-trained parameters into the feature extraction networks of the first student network and the teacher network.

[0204] The control component can load the first pre-training parameters obtained during the pre-training phase into the feature extraction networks in the first student network and the teacher network. These first pre-training parameters include the parameters of the pre-trained feature extraction network. For example, the control component can obtain a list of pre-trained weight parameters, which includes the parameters of the backbone network.

[0205] Step 412: Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network. The iteration method of the parameters in the teacher network is to determine the corresponding parameters in the teacher network based on the parameters in the first student network.

[0206] This step involves setting up the first adjusted model before training. During this setup, the control component can freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network, and set the iteration method of the parameters in the teacher network to determine the corresponding parameters in the teacher network based on the parameters in the first student network.

[0207] In one exemplary embodiment, the control component can freeze the gradients of the feature extraction network parameters in the first student network and the gradients of all parameters in the teacher network by setting them to false. During subsequent training, the unfrozen parameters in the first student network can be iteratively trained using the Faster R-CNN loss algorithm.

[0208] In one exemplary embodiment, the parameters in the teacher network are iterated by determining the parameters in the teacher network based on a preset formula, which includes:

[0209] tn = sn*x + tn'*(1-x);

[0210] Where tn is the parameter n of the teacher network corresponding to the current iteration, sn is the parameter n of the first student network corresponding to the current iteration, x is the preset smoothing coefficient, and tn' is the parameter n of the teacher network corresponding to the previous iteration of the current iteration. Parameter n is one of the parameters in the teacher network and the first student network. The smoothing coefficient x is an empirical value, ranging from 0.80 to 0.99, for example, smoothing coefficient x = 0.9.

[0211] Step 413: Train the first adjustment model to obtain the second student network.

[0212] After the control component completes some settings for model training, it can begin training the first adjusted model. The first student network becomes the second student network after training.

[0213] Please refer to Figure 8, which is a flowchart of training a first adjustment model according to an embodiment of this application. The process may include the following steps:

[0214] Sub-step 4131: Obtain the labeled dataset.

[0215] The labeled dataset includes multiple labeled samples, each containing a second image of the target object and label information used to identify defects in the target object.

[0216] In one exemplary embodiment, the target object is a display panel, and the tagged sample includes a second image of at least a portion of the display surface of the display panel and tag information, the tag information being used to identify defects in at least a portion of the display surface.

[0217] The labeled samples in the labeled dataset can be obtained in advance by adding labels to the images of the target objects, or they can be obtained from existing labeled samples in the database. This application embodiment does not limit this.

[0218] In this embodiment of the application, the tagged sample can be a large-size sample, and for example, the second image included in the tagged sample can be an image of a display motherboard including multiple display panels.

[0219] Sub-step 4132: Obtain multiple second images from multiple labeled samples in the labeled dataset.

[0220] The second image refers to the image in the labeled sample. In this step, the control component can acquire multiple second images from multiple labeled samples in the labeled dataset; that is, for each labeled sample, only the second image is acquired, without acquiring the label.

[0221] Sub-step 4133: Process multiple second images into multiple pseudo-label samples using a selective search algorithm. The pseudo-label samples include the second images and pseudo-labels of the second images that contain region resolution capabilities.

[0222] The control component can process multiple second images into multiple pseudo-label samples using a selective search algorithm. The pseudo-label samples include the second images and pseudo-labels for the second images that contain region-resolution capabilities. There are two types of pseudo-labels: those containing region-resolution capabilities and those containing category-resolution capabilities. This application uses pseudo-labels that contain region-resolution capabilities.

[0223] Sub-step 4134: Train the first adjustment model based on multiple pseudo-label samples. After training, the first student network becomes the second student network.

[0224] The control component can train the first adjustment model based on the multiple pseudo-label samples. After a preset cutoff condition is met, the training of the first adjustment model is complete, and the first student network in the first adjustment model is transformed into a second student network. The preset cutoff condition may include that the similarity loss between the candidate box features output by the first student network and the candidate box features output by the teacher network is less than a preset loss threshold, and that the output of the student network fits the pseudo-labels. Of course, the preset cutoff condition may also include others, such as the number of iterations reaching a specified value, etc., which are not limited in this embodiment.

[0225] Steps 406 to 413 constitute the first-stage coarse adjustment process in this embodiment of the application. The method used in this process can be the AlignDet pre-training method, which is a general pre-training method. Of course, other methods can also be used to train the first adjustment model, and this embodiment of the application does not limit this.

[0226] Steps 406 to 413 constitute the first stage of coarse adjustment, which allows for preliminary fitting of the parameters of the data dimensionality upscaling module and the target prediction network of the target detection model. Subsequent fine-tuning can then be performed.

[0227] Step 414: Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0228] The control component can unfreeze the parameters of the feature extraction network in the second student network by setting the gradient value (requires_grad) of the parameters to true. Then, the control component trains the second student network using a labeled dataset to obtain an object detection model. Please refer to Figure 9, which is a flowchart of training the second student network according to an embodiment of this application. This process may include the following steps:

[0229] Sub-step 4141: Obtain the labeled dataset.

[0230] If a labeled dataset has already been obtained in step 413, it is not necessary to obtain a labeled dataset in this step. If a labeled dataset was not obtained in step 413, it can be obtained in this step. This step can refer to sub-step 4131.

[0231] Sub-step 4142: Obtain the number of target regression categories.

[0232] The number of regression categories for this target is a value set based on different industrial scenarios, and this value can be set by the operator.

[0233] Sub-step 4143: Adjust the number of regression categories in the second student network to the target number of regression categories.

[0234] The number of regression categories can be a parameter of the category prediction head network in the second student network. The control component can adjust this parameter to the target number of regression categories and initialize other parameters in the category prediction head network to avoid mismatches.

[0235] Sub-step 4144: Preset and adjust the hyperparameters of the training to obtain the adjusted hyperparameters.

[0236] Hyperparameters are parameters set before the training process begins. The control component can preset and adjust the training hyperparameters to obtain the adjusted hyperparameters. The preset adjustment includes at least one of adjusting the learning rate descent rate to a first rate and adjusting the learning rate to a first learning rate. The first rate is a learning rate descent rate lower than a second rate, the first learning rate is a learning rate lower than the second learning rate, and the second learning rate and the second learning rate are the hyperparameters used to train the first adjusted model. That is, compared to the coarse adjustment in steps 406 to 413 above, this embodiment of the application reduces the learning rate descent rate and the learning rate during fine adjustment. This allows the second student network to avoid oscillations when converging to the vicinity of the optimal solution, thus enabling better convergence to the optimal solution.

[0237] Sub-step 4145: Train the second student network with the adjusted hyperparameters using the labeled dataset to obtain the object detection model.

[0238] The control component can train the second student network using adjusted hyperparameters based on a labeled dataset. Once the training cutoff condition is met, the second student network can be trained into an object detection model. Subsequently, the control component can acquire the image to be detected and input it into the object detection model. This model can then perform defect detection on the image and extract the defect information output by the model. The training cutoff condition for the second student network can include fitting the output of the second student network to the labels of the labeled samples.

[0239] Fine-tuning the model parameters in this stage can yield model parameters that generalize well to specific application scenarios, thereby improving the performance of the resulting target detection model.

[0240] In summary, the defect detection method provided in this application establishes a pre-trained model, including a pre-trained feature extraction network, based on the structure of the target detection model. An unsupervised contrastive learning strategy is then used to train the pre-trained model to obtain pre-training parameters. These pre-training parameters are then loaded into a first adjustment model, which is trained. The first student network in the first adjustment model is then trained twice to obtain the target detection model. This target detection model can then be used to detect defects in the image to be detected. Because the model can learn a large number of general features from the data during pre-training, these general features can be used to improve the model's performance in the target detection task, thus improving the accuracy of the defect detection method.

[0241] In the method of the embodiment shown in Figure 4, small-sized samples are used to pre-train the feature extraction network of the target detection network. However, large-sized samples can also be used to train the first student network and the target detection network. For example, please refer to Figure 10. Figure 10 is at least a partial flowchart of another defect detection method provided in this application embodiment. Before step 201 of the defect detection method shown in Figure 2, the defect detection method may also include the following steps:

[0242] Step 901: Obtain the pre-trained data dimensionality upgrade module.

[0243] This step can refer to step 406 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0244] Step 902: Obtain the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network.

[0245] This step can refer to step 401 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0246] Step 903: Obtain the pre-trained model based on the pre-trained data dimensionality enhancement module, linear classifier, pre-trained loss module, and pre-trained feature extraction network.

[0247] The control component can obtain a pre-trained model based on a pre-trained data dimensionality enhancement module, a linear classifier, a pre-trained loss module, and a pre-trained feature extraction network. This pre-trained model includes a pre-trained loss module, a main path network, and a momentum path network. The main path network and momentum path network have identical structures. The main path network includes the pre-trained data dimensionality enhancement module, the pre-trained feature extraction network, and the linear classifier, which are connected sequentially. The linear classifiers of the main path network and the momentum path network are respectively connected to the pre-trained loss module.

[0248] Please refer to Figure 11, which is a schematic diagram of a pre-trained model provided in an embodiment of this application. The pre-trained model includes a pre-training loss module 91, a main path network 92, and a momentum path network 93. The main path network 92 includes a pre-training data dimensionality enhancement module, a pre-training feature extraction network, and a linear classifier connected in sequence. The linear classifiers of the main path network and the momentum path network are respectively connected to the pre-training loss module. During the training of this pre-trained model, the parameters of the main path network 92 and the momentum path network 93 can be updated through momentum updates.

[0249] Compared with the embodiment using small-sized samples shown in Figure 4, the embodiment of this application adds a data dimensionality enhancement module to the model during the pre-training stage and pre-trains the data dimensionality enhancement module together.

[0250] Step 904: Obtain the unlabeled dataset.

[0251] The unlabeled dataset includes multiple unlabeled samples. In one exemplary embodiment, the target object is a display panel, and each unlabeled sample includes images of multiple display surfaces of the display panel. This unlabeled sample is also referred to as a large-size sample. Unlike the embodiment shown in Figure 4 above, this embodiment does not require slicing large-size samples using a sliding window to obtain small-size samples.

[0252] Step 905: Train the pre-trained model using an unsupervised contrastive learning strategy to obtain the first pre-training parameters of the pre-trained feature extraction network and the second pre-training parameters of the pre-trained data dimensionality enhancement module.

[0253] The control component can train the pre-trained model using an unsupervised contrastive learning strategy, such as using the contrastive file strategy of MoCo-v2.

[0254] Step 906: Obtain the data upgrade module.

[0255] This step can refer to step 406 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0256] It should be noted that, unlike step 901, this step obtains the data dimensionality upgrade module in the subsequently established first student network.

[0257] Step 907: Establish a first object detection network, which includes a feature extraction network to be trained.

[0258] This step can refer to step 407 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0259] Step 908: Adjust the first convolutional layer of the feature extraction network to be trained so that the first convolutional layer matches the data dimensionality enhancement module.

[0260] This step can refer to step 408 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0261] Step 909: Connect the output of the data upscaling module to the first convolutional layer to form the first student network.

[0262] This step can refer to step 409 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0263] Step 910: Obtain the first adjustment model based on the first student network and the loss module.

[0264] This step can refer to step 410 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0265] Step 911: Load the first pre-training parameters and the second pre-training parameters into the first student network and the teacher network.

[0266] Unlike the embodiment shown in Figure 4 above, since the data dimensionality enhancement module was trained in the pre-training stage and the second pre-training parameters were obtained, the first pre-training parameters can be loaded into the feature extraction network of the first student network and the feature extraction network of the teacher network in this step, and the second pre-training parameters can be loaded into the data dimensionality enhancement module of the first student network and the data dimensionality enhancement module of the teacher network.

[0267] Step 912: Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network. The iteration method of the parameters in the teacher network is to determine the corresponding parameters in the teacher network based on the parameters in the first student network.

[0268] This step can refer to step 412 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0269] Step 913: Train the first adjustment model to obtain the second student network.

[0270] This step can refer to step 413 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0271] Step 914: Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0272] This step can refer to step 414 in the above embodiment, and will not be repeated here in the embodiments of this application.

[0273] In summary, the defect detection method provided in this application establishes a pre-trained model, including a pre-trained feature extraction network, based on the structure of the target detection model. An unsupervised contrastive learning strategy is then used to train the pre-trained model to obtain pre-training parameters. These pre-training parameters are then loaded into a first adjustment model, which is trained. The first student network in the first adjustment model is then trained twice to obtain the target detection model. This target detection model can then be used to detect defects in the image to be detected. Because the model can learn a large number of general features from the data during pre-training, these general features can be used to improve the model's performance in the target detection task, thus improving the accuracy of the defect detection method.

[0274] This patent verifies the improvement of target detection accuracy in refined application scenarios through comparative experiments:

[0275] Option 1: Train the object detection model by randomly initializing network parameters.

[0276] Option 2: Load the pre-trained ResNet50 weight parameters based on the publicly available COCO dataset (a publicly available dataset that can be used for image detection) and train the object detection model.

[0277] Option 3: Use the method provided in Figure 4 or Figure 10 of this application to obtain the target detection model.

[0278] (1) Compared with Scheme 1, Scheme 3 provided in this application: Since the network parameters of Scheme 1 are set randomly, on the one hand, the initial loss of the network may be large, which makes it difficult for the network to converge and the network may underfit. On the other hand, when the training dataset is relatively limited, the network may overfit, that is, the generalization is poor.

[0279] (2) Compared with Scheme 2, Scheme 3 provided in this application has a large difference in the adaptation scenarios of the two pre-trained model parameters. The COCO dataset is mostly data from complex visual scenes in the real world, while the industrial scene applied in this application has a high degree of repetition (the similarity of images of different display motherboards is high). The focus of the models required by Scheme 2 and Scheme 3 is quite different. Therefore, the difference in pre-trained parameters based on the two different datasets may be large. In the case of limited fine-tuning training data, the pre-training method provided in this application can make the network more generalized to feature extraction in industrial scenes and improve the accuracy of the model.

[0280] Figure 12 is a block diagram of a defect detection device provided in an embodiment of this application. The defect detection device 1200 includes:

[0281] The acquisition module 1210 acquires the image to be detected, which includes an image of at least one target object to be detected collected by the image acquisition component.

[0282] The input module 1220 is used to input the image to be detected into the target detection model.

[0283] The acquisition module 1230 is used to acquire defect information of at least one defect in the display surface output by the target detection model.

[0284] The process of acquiring the target detection model includes:

[0285] A first adjustment model is established, which includes a first student network, a teacher network, and a loss module.

[0286] The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network;

[0287] Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network;

[0288] The first adjustment model is trained, and the first student network becomes the second student network after training.

[0289] Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

[0290] Furthermore, this application embodiment also provides a defect detection device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the defect detection method provided in the above embodiments.

[0291] Furthermore, this application embodiment also provides a non-volatile computer storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the defect detection method provided in the above embodiments.

[0292] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0293] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0294] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0295] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0296] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A defect detection method characterized by, The method includes: Acquire an image to be detected, wherein the image to be detected includes an image acquired by an image acquisition component that includes at least one target object to be detected; Input the image to be detected into the target detection model; Obtain the defect detection information output by the target detection model; The process of acquiring the target detection model includes: Based on the structure of the target detection model, a pre-trained model including a pre-trained feature extraction network is established; The pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the first pre-trained parameters of the pre-trained feature extraction network; A first adjustment model is established, which includes a first student network, a teacher network, and a loss module. The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network; Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network; The first adjustment model is trained, and the first student network becomes the second student network after training. Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

2. The method according to claim 1, characterized in that, The unsupervised contrastive learning strategy is used to train the pre-trained model to obtain the first pre-trained parameters of the pre-trained feature extraction network, including: Obtain an unlabeled dataset, which includes multiple unlabeled samples, each of which includes a first image of a target object; Multiple batches of training data were obtained based on the unlabeled dataset; The pre-trained model is trained based on the multiple batches of training data; After the training cutoff condition is met, the target pre-trained model is obtained; Obtain the first pre-training parameters of the pre-trained feature extraction network in the target pre-trained model; The process of obtaining training data from one batch of the multiple batches of training data includes: Obtain n unlabeled samples from the unlabeled dataset, where n is a positive integer greater than 1; One target sample from the n unlabeled samples is augmented to obtain an augmented sample; Obtain the similarity between the target sample and the n-1 unlabeled samples (excluding the target sample) among the n unlabeled samples; The enhanced samples and the unlabeled samples with a similarity greater than or equal to a preset threshold are identified as positive samples; Unlabeled samples with similarity less than the preset threshold are identified as negative samples. The training data in a batch includes the augmented samples, positive samples from the n unlabeled samples, and negative samples.

3. The method according to claim 2, characterized in that, The pre-trained model includes a pre-trained loss module, a main path network, and a momentum path network. The main path network and the momentum path network have the same structure. The main path network includes the pre-trained feature extraction network and a linear classifier that are interconnected. The linear classifier of the main path network and the linear classifier of the momentum path network are respectively connected to the pre-trained loss module.

4. The method according to claim 2, characterized in that, The process of obtaining the unlabeled dataset includes: Acquire images of multiple target objects; The images of the multiple target objects are segmented using a sliding window to obtain the multiple unlabeled samples.

5. The method according to claim 1, characterized in that, Before training the second student network, the method further includes: A labeled dataset is obtained, which includes multiple labeled samples. Each labeled sample includes a second image of a target object and label information, wherein the label information is used to identify defects of the target object. The training of the second student network includes: The hyperparameters of the training are preset and adjusted to obtain the adjusted hyperparameters. The preset adjustment includes at least one of adjusting the learning rate decrease rate to a first rate and adjusting the learning rate to a first learning rate, wherein the first rate is a learning rate decrease rate that is less than a second rate, and the first learning rate is a small... The learning rate of the second learning rate, the learning rate decrease rate of the second speed, and the second learning rate are hyperparameters for training the first adjusted model; The second student network is trained using the labeled dataset and the adjusted hyperparameters.

6. The method according to claim 1, characterized in that, The training of the first adjustment model includes: Obtain multiple second images from multiple labeled samples in the labeled dataset; The multiple second images are processed into multiple pseudo-label samples using a selective search algorithm. The pseudo-label samples include the second images and pseudo-labels of the second images that contain region resolution capabilities. The first adjustment model is trained based on the multiple pseudo-label samples.

7. The method according to claim 5, characterized in that, The first student network and the teacher network have the same structure as the object detection model and are respectively connected to the loss module. The establishment of the first adjustment model includes: A data upscaling module is provided, which includes an input end and an output end. The data upscaling module is used to process the image to be detected input by the input end into an image with the same resolution as the second image, and output it from the output end. A first object detection network is established, which includes a feature extraction network to be trained, wherein the first layer of the feature extraction network to be trained is a first convolutional layer. Adjust the first convolutional layer to match the data dimensionality enhancement module; The output of the data upscaling module is connected to the first convolutional layer to form the first student network; The first adjustment model is obtained based on the first student network and the loss module, and the number of regression categories of the teacher network and the first student network is set to 1.

8. The method according to claim 7, characterized in that, Before the pre-adjustment of the hyperparameters during training to obtain the adjusted hyperparameters, the method further includes: Obtain the number of target regression categories; Adjust the number of regression categories in the second student network to the target number of regression categories.

9. The method according to claim 7, characterized in that, The establishment of a pre-trained model based on the structure of the target detection model, including a pre-trained feature extraction network, includes: A pre-training data upscaling module is provided, which includes an input end and an output end. The pre-training data upscaling module is used to process the image to be detected input by the input end into an image with the same resolution as the second image, and output it from the output end. Obtain the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network; The pre-trained model is obtained based on the pre-trained data dimensionality enhancement module, the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network. The pre-trained model includes the pre-trained loss module, the main path network, and the momentum path network. The main path network and the momentum path network have the same structure. The main path network includes the pre-trained data dimensionality enhancement module, the pre-trained feature extraction network, and the linear classifier connected in sequence. The linear classifier of the main path network and the linear classifier of the momentum path network are respectively connected to the pre-trained loss module. The unsupervised contrastive learning strategy is used to train the pre-trained model to obtain the first pre-trained parameters of the pre-trained feature extraction network, including: The pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the first pre-training parameters of the pre-trained feature extraction network and the second pre-training parameters of the pre-trained data dimensionality enhancement module. The step of loading the pre-trained parameters into the feature extraction networks of the first student network and the teacher network includes: The first pre-training parameters and the second pre-training parameters are loaded into the first student network and the teacher network.

10. The method according to claim 7, characterized in that, The first student network includes the data dimensionality enhancement module, the feature extraction network, the feature pyramid network, the candidate box extraction network, and the prediction network, which are connected in sequence.

11. The method according to claim 5, characterized in that, Freezing the gradients of the parameters of the feature extraction network in the first student network and the gradients of the parameters of the teacher network includes: Set the gradients of the parameters of the feature extraction network in the first student network and all parameters of the teacher network to false; The step of unfreezing the parameters of the feature extraction network in the second student network includes: Set the gradient values ​​of the parameters of the feature extraction network in the second student network to true.

12. The method according to any one of claims 1 to 11, characterized in that, The parameters in the teacher network are iterated using a preset formula to determine the parameters, which includes: tn = sn*x + tn'*(1-x); Wherein, tn is the parameter n of the teacher network corresponding to the current iteration process, sn is the parameter n of the first student network corresponding to the current iteration process, x is the preset smoothing coefficient, and tn' is the parameter n of the teacher network corresponding to the previous iteration process of the current iteration process. The parameter n is one of the parameters of the teacher network and the first student network.

13. The method according to any one of claims 1 to 8, characterized in that, The establishment of a pre-trained model based on the structure of the target detection model, including a pre-trained feature extraction network, includes: Obtain the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network; The pre-trained model is obtained based on the linear classifier, the pre-trained loss module, and the pre-trained feature extraction network. The pre-trained model includes the pre-trained loss module, the main path network, and the momentum path network. The main path network and the momentum path network have the same structure. The main path network includes the pre-trained feature extraction network and the linear classifier that are interconnected. The linear classifier of the main path network and the linear classifier of the momentum path network are respectively connected to the pre-trained loss module.

14. A defect detection device, characterized in that, The defect detection device includes: The acquisition module is used to acquire the image to be detected, the image to be detected including an image of at least one target object to be detected acquired by the image acquisition component; The input module is used to input the image to be detected into the target detection model; The acquisition module is used to acquire the defect detection information output by the target detection model; The process of acquiring the target detection model includes: Based on the structure of the target detection model, a pre-trained model including a pre-trained feature extraction network is established; the pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the pre-trained parameters of the pre-trained feature extraction network. A first adjustment model is established, which includes a first student network, a teacher network, and a loss module. The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network; Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network; The first adjustment model is trained, and the first student network becomes the second student network after training. Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

15. A defect detection device, characterized in that, The defect detection device includes a control component and an image acquisition component; The control component is used to acquire an image of at least one target object collected by the image acquisition component; The control component is equipped with a target detection model, and the control component is also used to input the image to be detected into the target detection model and obtain the defect detection information output by the target detection model; The process of acquiring the target detection model includes: Based on the structure of the target detection model, a pre-trained model including a pre-trained feature extraction network is established; the pre-trained model is trained using an unsupervised contrastive learning strategy to obtain the pre-trained parameters of the pre-trained feature extraction network. A first adjustment model is established, which includes a first student network, a teacher network, and a loss module. The first pre-training parameters are loaded into the feature extraction networks in the first student network and the teacher network; Freeze the gradients of the parameters of the feature extraction network in the first student network and the gradients of all parameters of the teacher network; The first adjustment model is trained, and the first student network becomes the second student network after training. Unfreeze the parameters of the feature extraction network in the second student network and train the second student network to obtain the target detection model.

16. A defect detection device, characterized in that, The defect detection device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the defect detection method as described in any one of claims 1 to 13.

17. A non-volatile computer storage medium, characterized in that, The non-volatile computer storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the defect detection method as described in any one of claims 1 to 13.

Images