A defect detection model training method, defect detection method and device

Abstract

The embodiments of the present invention provide a method for training a defect detection model, a defect detection method, and an apparatus, the method comprising: obtaining a training image; determining a pseudo-label of the training image using an unsupervised training method; and training the defect detection model using a weakly supervised training method and / or a supervised training method based on the training image and its corresponding pseudo-label. Since the scheme of the embodiment of the present invention adopts an unsupervised training method to obtain the pseudo-label of the training image, and trains the defect detection model based on the training image and its corresponding pseudo-label, the problem of low detection accuracy of the trained model due to the inability to obtain a large number of defect image samples is solved, thereby improving the detection accuracy of the defect detection model. The scheme of the embodiment of the present invention can also choose to train the defect detection model using a weakly supervised and / or supervised training method according to the needs of different application scenarios, which has the advantage of flexible deployment and minimized labor costs.

Description

Technical Field

[0001] The present invention relates to the field of defect detection technology, and in particular to a defect detection model training method, a defect detection method and a defect detection device. Background Art

[0002] During the industrial production process, various defects may occur in products due to production reasons such as raw materials, accessories, processes, procedures, or design reasons such as product structure and formula. Therefore, after the product is produced, it is necessary to conduct defect detection on the product.

[0003] In current defect detection technology, it is first necessary to obtain a large number of defect image samples to construct a training sample data set for the defect detection model. However, in the early stages of industrial production, it is impossible to produce a large number of defective products, and there are not enough defect image samples. Therefore, the detection accuracy of the trained defect detection model is not high and cannot meet the needs of practical applications. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a defect detection model training method, defect detection method and device to improve the detection accuracy of the defect detection model. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for training a defect detection model, the method comprising:

[0006] Get training images;

[0007] Determining pseudo labels for the training images using an unsupervised training approach;

[0008] Based on the training images and their corresponding pseudo labels, a defect detection model is trained using a weakly supervised training method and / or a supervised training method.

[0009] Optionally, the method further includes:

[0010] Obtaining a detection result output by the defect detection model;

[0011] If the detection result does not meet the preset detection condition, acquiring the target image;

[0012] The target image is added to the training image, and the step of training the defect detection model based on the training image and its corresponding pseudo-label using a weakly supervised training method and / or a supervised training method is returned until the detection result output by the defect detection model meets the preset detection condition.

[0013] Optionally, the unsupervised training method includes:

[0014] Acquire a plurality of sample images and an initial feature reconstruction network, wherein the sample images include defect-free objects;

[0015] Perform feature extraction on each sample image to obtain multiple sample feature maps of different resolutions corresponding to each sample image;

[0016] Using the initial feature reconstruction network, feature reconstruction is performed based on each sample feature map to obtain a reconstructed sample feature map corresponding to the sample feature map;

[0017] Based on the reconstruction error between each sample feature map and its corresponding reconstructed sample feature map, the parameters of the initial feature reconstruction network are adjusted until the initial feature reconstruction network converges to obtain a feature reconstruction network.

[0018] Optionally, the step of determining the pseudo labels of the training images using an unsupervised training method includes:

[0019] Inputting the training image into a residual network, performing feature extraction on the training image, and obtaining a plurality of target feature maps with different resolutions;

[0020] Inputting the plurality of target feature maps into the feature reconstruction network, performing feature reconstruction on the plurality of target feature maps respectively, and obtaining a reconstructed feature map corresponding to each target feature map;

[0021] Based on a reconstruction error between each target feature map and its corresponding reconstructed feature map, a pseudo label of the training image is determined, and the pseudo label is output.

[0022] Optionally, the step of determining the pseudo label of the training image based on the reconstruction error between each target feature map and its corresponding reconstructed feature map includes:

[0023] Calculating a reconstruction error between each target feature map and its corresponding reconstructed feature map;

[0024] According to the reconstruction error, according to the formula Calculate the fusion error d;

[0025] Determining a pseudo label of the training image based on a magnitude relationship between the fusion error and a preset threshold;

[0026] Among them, f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of the target feature maps.

[0027] Optionally, the target feature map is a feature map corresponding to pixels in the training image;

[0028] The step of determining the pseudo label of the training image based on the magnitude relationship between the fusion error and a preset threshold comprises:

[0029] When the fusion error is not less than a preset threshold, the pixel is determined to be a defective pixel in the training image, and a defective area in the training image is determined based on all defective pixels as a pseudo label of the training image.

[0030] Optionally, the weakly supervised training method includes:

[0031] Get the initial segmentation network;

[0032] Inputting each training image into the initial segmentation network to obtain a prediction score map corresponding to the training image, wherein the pixel value of a pixel point in the prediction score map represents the predicted probability that the corresponding pixel point in the training image is a defective pixel point;

[0033] Based on the predicted score map and annotated area corresponding to each training image, calculate the sampling loss corresponding to the training image;

[0034] Calculating the consistency loss corresponding to the training image based on the pixel values ​​of each pixel point within the marked area of ​​each training image;

[0035] The weighted value of the sampling loss and the consistency loss is taken as the total loss, and the parameters of the initial segmentation network are adjusted based on the total loss until the initial segmentation network converges, thereby obtaining a segmentation network as the defect detection model.

[0036] Optionally, the step of calculating the sampling loss corresponding to each training image based on the predicted score map and the annotated area corresponding to the training image includes:

[0037] For each pixel outside the marked area in each training image, calculate the first loss function value corresponding to each pixel based on the predicted score map;

[0038] Calculate the average value of pixel values ​​of a first preset number of pixel points with the largest corresponding first loss function value as the first loss;

[0039] For each pixel in the marked area of ​​each training image, each pixel is regarded as a defective pixel, and a second loss function value corresponding to each pixel is calculated based on the predicted score map;

[0040] Calculate the average value of pixel values ​​of a second preset number of pixel points whose corresponding second loss function values ​​are minimum as the second loss;

[0041] For each pixel in the marked area of ​​each training image, each pixel is regarded as a non-defective pixel, and a third loss function value corresponding to each pixel is calculated based on the prediction score map;

[0042] Calculate the average of the pixel values ​​of a third preset number of pixels whose corresponding third loss function values ​​are minimum as the third loss;

[0043] The sum of the first loss, the second loss, and the third loss is used as the sampling loss.

[0044] Optionally, the step of calculating the consistency loss corresponding to each training image based on the pixel value of each pixel point within the marked area in each training image includes:

[0045] Clustering each pixel point in the marked area of ​​each training image according to the pixel value to obtain multiple clusters;

[0046] For each cluster, the mean of the variance of the pixel values ​​corresponding to each pixel point included in the cluster in the prediction score map is calculated as the consistency loss.

[0047] In a second aspect, an embodiment of the present invention provides a defect detection method, the method comprising:

[0048] Obtain the image to be detected;

[0049] The image to be detected is input into a pre-trained defect detection model for defect detection to obtain a defect detection result, wherein the defect detection model is based on the training image and its corresponding pseudo-label, and is trained using a weakly supervised training method and / or a supervised training method, and the pseudo-label is a pseudo-label of the training image determined using an unsupervised training method.

[0050] In a third aspect, an embodiment of the present invention provides a training device for a defect detection model, the device comprising:

[0051] A sample acquisition module is used to acquire training images;

[0052] a pseudo-label determination module, configured to determine the pseudo-labels of the training images using an unsupervised training method;

[0053] The model training module is used to train the defect detection model based on the training images and their corresponding pseudo labels using a weakly supervised training method and / or a supervised training method.

[0054] In a fourth aspect, an embodiment of the present invention provides a defect detection device, comprising:

[0055] An image acquisition module, used to acquire an image to be detected;

[0056] An image detection module is used to input the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result, wherein the defect detection model is based on the training image and its corresponding pseudo-label, and is trained using a weakly supervised training method and / or a supervised training method, and the pseudo-label is a pseudo-label of the training image determined using an unsupervised training method.

[0057] In a fifth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0058] Memory for storing computer programs;

[0059] The processor is used to implement the method steps described in any one of the first aspect or the second aspect when executing the program stored in the memory.

[0060] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first or second aspects are implemented.

[0061] Beneficial effects of the embodiments of the present invention:

[0062] In the solution provided by the embodiment of the present invention, the electronic device can obtain a training image; determine the pseudo-label of the training image using an unsupervised training method; and train the defect detection model based on the training image and its corresponding pseudo-label using a weakly supervised training method and / or a supervised training method. Since in the solution of the embodiment of the present invention, the pseudo-label of the training image can be obtained by an unsupervised training method first, and then the defect detection model is trained based on the training image and its corresponding pseudo-label, there is no need to obtain defect image samples during the model training process, which solves the problem of low detection accuracy of the trained defect detection model due to the inability to obtain a large number of defect image samples, thereby improving the detection accuracy of the defect detection model. At the same time, in the solution of the embodiment of the present invention, it is also possible to choose to train the defect detection model using a weakly supervised and / or supervised training method based on the requirements for detection accuracy in different application scenarios, which has the advantage of flexible deployment and minimization of labor costs. Of course, it is not necessary to achieve all of the advantages described above at the same time when implementing any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0064] Figure 1 A flowchart of a defect detection model training method provided by an embodiment of the present invention;

[0065] Figure 2 Based on Figure 1 A schematic diagram of a flow chart of a model training method in the illustrated embodiment;

[0066] Figure 3 Based on Figure 1 A flow chart of the iterative model training method of the illustrated embodiment;

[0067] Figure 4 Based on Figure 1 A flow chart of the unsupervised training method of the illustrated embodiment;

[0068] Figure 5 Based on Figure 4 A schematic diagram of a flow chart of an unsupervised training method according to the illustrated embodiment;

[0069] Figure 6 Based on Figure 4 A flow chart of a method for determining a pseudo label according to the illustrated embodiment;

[0070] Figure 7 Based on Figure 1 A flow chart of the weakly supervised training method of the illustrated embodiment;

[0071] Figure 8 Based on Figure 7 A schematic diagram of a flow chart of a weakly supervised training method according to the illustrated embodiment;

[0072] Figure 9 A flowchart of a defect detection method provided by an embodiment of the present invention;

[0073] Figure 10 A schematic structural diagram of a defect detection model training device provided by an embodiment of the present invention;

[0074] Figure 11 A schematic structural diagram of a defect detection device provided by an embodiment of the present invention;

[0075] Figure 12 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0076] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0077] To improve the detection accuracy of a defect detection model, embodiments of the present invention provide a defect detection model training method and apparatus, a defect detection method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following first introduces a defect detection model training method provided by an embodiment of the present invention.

[0078] The defect detection model training method provided in the embodiment of the present invention can be applied to any electronic device that needs to train a defect detection model, for example, a server, a processing device, etc., without specific limitation herein. For the sake of clarity, it is hereinafter referred to as an electronic device.

[0079] like Figure 1 As shown, a method for training a defect detection model includes:

[0080] S101, obtaining a training image;

[0081] S102, determining a pseudo label of the training image using an unsupervised training method;

[0082] S103: Based on the training images and their corresponding pseudo labels, a defect detection model is trained using a weakly supervised training method and / or a supervised training method.

[0083] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can obtain training images; use an unsupervised training method to determine the pseudo labels of the training images; and use a weakly supervised training method and / or a supervised training method to train the defect detection model based on the training images and their corresponding pseudo labels. Since in the solution of the embodiment of the present invention, the pseudo labels of the training images can be obtained by an unsupervised training method first, and then the defect detection model is trained based on the training images and their corresponding pseudo labels, there is no need to obtain defect image samples during the model training process, which solves the problem of low detection accuracy of the trained defect detection model due to the inability to obtain a large number of defect image samples, thereby improving the detection accuracy of the defect detection model. At the same time, in the solution of the embodiment of the present invention, it is also possible to choose to use weakly supervised and / or supervised training methods to train the defect detection model according to the requirements for detection accuracy in different application scenarios, which has the advantage of flexible deployment and minimization of labor costs.

[0084] During the training process of a defect detection model, since it may not be possible to obtain a large number of defective products in the early stages of industrial production, there may not be enough training images with defects. Therefore, the detection accuracy of defect detection models trained using weakly supervised training and / or supervised training methods is not high and cannot meet the needs of practical applications. Therefore, in the defect detection model training method provided in the embodiment of the present invention, after obtaining the training images, the above-mentioned step S102 can be performed, that is, using an unsupervised training method to determine the pseudo labels of the training images.

[0085] For example, Figure 2 As shown, an unsupervised training method can be first used as a cold start, that is, after obtaining the training image, the pseudo label of the training image is determined through the unsupervised training method. The pseudo label can be used to represent the defective area in the training image or to identify whether the training image includes a defective object. For example, the defective area in the training image can be marked out by a labeling box as a pseudo label.

[0086] However, the accuracy of the pseudo-labels determined by only using unsupervised training methods is low and may not meet the needs of industrial production, that is, it cannot meet the actual implementation scenarios. Therefore, after using unsupervised training methods to determine the pseudo-labels of the training images, you can choose to use weakly supervised training methods and / or supervised training methods to train the defect detection model based on the training images and their corresponding pseudo-labels, according to the needs of the actual application scenarios.

[0087] In one embodiment, after the electronic device determines the pseudo label of the training image through unsupervised training, a small amount of manual work can be performed to roughly calibrate and modify it to obtain the rough label of the training image. Then, the electronic device can use weakly supervised training to train the defect detection model based on the training image and its corresponding rough label.

[0088] In another embodiment, after the electronic device determines the pseudo label of the training image through unsupervised training, it can be manually refined and calibrated and modified to obtain the fine label of the training image. Then, the electronic device can use supervised training to train the defect detection model based on the training image and its corresponding fine label.

[0089] In another embodiment, after the defect detection model is trained using a weakly supervised training method based on the training image and its corresponding coarse label, in order to further improve the accuracy of the defect detection model, the defect detection model can also be further optimized and trained using a supervised training method based on the training image and its corresponding fine label. This is reasonable.

[0090] In the solution provided by the embodiment of the present invention, the pseudo-labels of the training images can be determined by an unsupervised training method first, and then the defect detection model can be trained by a weakly supervised training method and / or a supervised training method based on the training images and their corresponding pseudo-labels. Since the unsupervised training method can provide a large number of pseudo-labels for complex defect detection problems, it can solve the cold start problem of industrial production lines, has obvious advantages in inference time and space occupancy, and is easy to deploy. The weakly supervised training method can use the pseudo-labels obtained by the unsupervised training method to train the detection model. Therefore, it solves the problem that the detection accuracy of the trained defect detection model is not high due to the inability to obtain a large number of defect image samples, and improves the detection accuracy of the defect detection model. At the same time, in the solution of the embodiment of the present invention, it is also possible to choose to use weakly supervised and / or supervised training methods to train the defect detection model according to the requirements for detection accuracy in different application scenarios, so it also has the advantages of flexible deployment while minimizing labor costs.

[0091] As an implementation method of the embodiment of the present invention, Figure 3 As shown, the above method also includes:

[0092] S301, obtaining the detection result output by the defect detection model;

[0093] After training, the defect detection model can be deployed on the production line to perform real-time defect detection on products. Product images can be input into the defect detection model, which then outputs a detection result, which can be a defect area in the input product image.

[0094] S302, if the detection result does not meet the preset detection condition, acquiring the target image;

[0095] After obtaining the detection results of the defect detection model, it can be determined whether the detection results meet preset detection conditions, where the preset detection conditions are used to indicate whether the accuracy of the detection results of the defect detection model meets actual application requirements. If it is determined that the detection results do not meet the preset detection conditions, a target image can be obtained, where the target image can be an input image of the product.

[0096] In one embodiment, the electronic device can determine whether the error between the defect area output by the defect detection model and the actual defect area of ​​the product is less than a preset threshold. If it is not less than the preset threshold, it can be determined that the detection result does not meet the preset detection conditions.

[0097] S303, adding the target image to the training image, and returning to the step of training the defect detection model based on the training image and its corresponding pseudo-label using a weakly supervised training method and / or a supervised training method, until the detection result output by the defect detection model meets the preset detection condition.

[0098] Since the detection results do not meet the preset detection conditions, it means that the current defect detection model cannot meet the accuracy requirements for defect detection on the actual production line. Therefore, after acquiring the target image, the electronic device can add the target image to the training image and continue to iteratively optimize the defect detection model based on the training image using weak supervision and / or supervised training methods. In other words, return to the above step of training the defect detection model based on the training image and its corresponding pseudo-label using weak supervision training and / or supervised training methods until the detection results output by the defect detection model meet the preset detection conditions, at which point iterative training can be stopped.

[0099] It can be seen that in the solution provided by the embodiment of the present invention, after the defect detection model is trained, it can be deployed to the production line to detect the product in real time and determine whether the detection results meet the preset detection conditions. If the preset detection conditions are not met, the model can be iteratively trained to achieve optimization of the defect detection model, so that the defect detection model has excellent defect detection performance, ensuring that defect detection can meet the accuracy requirements of actual applications.

[0100] As an implementation method of the embodiment of the present invention, Figure 4 As shown, the above unsupervised training method may include:

[0101] S401, obtaining multiple sample images and an initial feature reconstruction network;

[0102] When the detection model is trained using an unsupervised training method, multiple sample images and an initial feature reconstruction network can be obtained, where the sample image can be an image of a defect-free object. The initial feature reconstruction network is used to reconstruct a reconstructed sample feature map corresponding to the feature map extracted from the image, and specifically can be a network model such as an autoencoder that can perform feature reconstruction.

[0103] S402, performing feature extraction on each sample image to obtain a plurality of sample feature maps of different resolutions corresponding to each sample image;

[0104] After acquiring the sample images, feature extraction can be performed on each sample image to obtain multiple sample feature maps of different resolutions corresponding to each sample image. As an embodiment, the electronic device can use a feature extractor to extract features from the sample images, where the feature extractor can be a residual network (ResNet) pre-trained using a large public dataset.

[0105] When using this pre-trained residual network to extract features from sample images, multi-scale, or resolution-varying, regional feature information can be obtained. This feature information not only describes the local characteristics of the corresponding region but also encodes multiple spatial contexts, making it highly discriminative and facilitating defect detection. Of course, the feature extractor can also be other network models capable of feature extraction, and this is not specifically limited here.

[0106] When performing feature extraction, the electronic device can extract features from the sample image at different resolutions, thereby obtaining a plurality of sample feature maps of different resolutions corresponding to the sample image, for example, Figure 5 As shown, Resnet can include three network layers, namely layer1, layer2 and layer3. The sample image is processed by feature extraction of layer1, layer2 and layer3 respectively, and three sample feature maps f with different resolutions can be obtained. For example, the sample feature maps f with resolutions of resolution A, resolution B and resolution C are obtained. i , f2 and f3.

[0107] S403, using the initial feature reconstruction network, performing feature reconstruction based on each sample feature map to obtain a reconstructed sample feature map corresponding to the sample feature map;

[0108] After obtaining the above sample feature map, the sample feature map can be reconstructed using the pre-acquired initial feature reconstruction network to obtain a reconstructed sample feature map corresponding to the sample feature map. Figure 5 As shown in the figure, the sample feature map output by Resnet can be input into the corresponding Auto Encoder (feature reconstruction self-encoder), and the Auto Encoder can reconstruct the features of the sample image to obtain the reconstructed sample feature map f′ corresponding to the sample feature map.

[0109] S404: Based on the reconstruction error between each sample feature map and its corresponding reconstructed sample feature map, adjust the parameters of the initial feature reconstruction network until the initial feature reconstruction network converges to obtain a feature reconstruction network.

[0110] If the object in the sample image has defects, the feature reconstruction network will have a poor feature reconstruction effect, and the difference between the reconstructed sample feature map and the sample feature map will be large. If the object in the sample image does not have defects, the feature reconstruction network will have a good feature reconstruction effect, and the difference between the reconstructed sample feature map and the sample feature map will be small. Therefore, the initial feature reconstruction network can be trained based on this.

[0111] In one embodiment, after determining the sample feature graph and its corresponding reconstructed sample feature graph, a reconstruction error between each sample feature graph and its corresponding reconstructed sample feature graph can be calculated, and the parameters of the initial feature reconstruction network can be adjusted based on the reconstruction error to make the parameters of the initial feature reconstruction network increasingly appropriate. The reconstruction error can be obtained by calculating the cosine distance, L1 distance, L2 distance, or other calculation method that can measure the difference between two vectors, which is not specifically limited here.

[0112] As the parameters of the initial feature reconstruction network are continuously adjusted during iterative training, the performance of the initial feature reconstruction network becomes better and better, and the output results become more and more accurate. When the number of sample image iterations reaches a preset number or the accuracy of the output results reaches a preset threshold, the initial feature reconstruction network converges and training can be stopped. At this time, the training of the initial feature reconstruction network is completed, and a feature reconstruction network is obtained. The preset number of times can be set based on factors such as model accuracy. For example, it can be 10,000 times, 20,000 times, 50,000 times, etc., and is not specifically limited here. The preset threshold can also be set based on factors such as model accuracy. For example, it can be 90%, 95%, 98%, etc., and is not specifically limited here.

[0113] It can be seen that in the solution provided by the embodiment of the present invention, when performing unsupervised training, multiple sample images and an initial feature reconstruction network can be obtained, and features can be extracted for each sample image to obtain multiple sample feature maps of different resolutions corresponding to each sample image. Then, the initial feature reconstruction network is used to perform feature reconstruction based on each sample feature map to obtain a reconstructed sample feature map corresponding to the sample feature map. Based on the reconstruction error between each sample feature map and its corresponding reconstructed sample feature map, the parameters of the initial feature reconstruction network are adjusted until the initial feature reconstruction network converges to obtain a feature reconstruction network. By using the reconstruction error corresponding to the feature reconstruction network for model training, a detection model capable of defect detection can be obtained by training with positive samples without manual labeling, thus saving labor costs.

[0114] As an implementation method of an embodiment of the present invention, in the case where the above-mentioned detection model is trained based on sample images using an unsupervised training method, the detection model may include the above-mentioned residual network and the above-mentioned feature reconstruction network.

[0115] Correspondingly, such as Figure 6 As shown, the step of determining the pseudo labels of the training images using the unsupervised training method may include:

[0116] S601, inputting the training image into the residual network, performing feature extraction on the training image, and obtaining a plurality of target feature maps with different resolutions;

[0117] S602, inputting the plurality of target feature maps into the feature reconstruction network, performing feature reconstruction on the plurality of target feature maps respectively, and obtaining a reconstructed feature map corresponding to each target feature map;

[0118] After obtaining the training image, the electronic device can input the training image into the detection model trained by the above-mentioned unsupervised training method. The detection model obtains the training image, and the residual network included in it can extract features of the training image, thereby obtaining multiple target feature maps of different resolutions corresponding to the training image.

[0119] The residual network can input the extracted target feature map into the feature reconstruction network, and then the feature reconstruction network can reconstruct each target feature map separately to obtain a reconstructed feature map corresponding to each target feature map.

[0120] S603: Determine a pseudo label of the training image based on a reconstruction error between each target feature map and its corresponding reconstructed feature map, and output the pseudo label.

[0121] Because the difference between the reconstructed feature map obtained by the feature reconstruction network and the target feature map increases when the target object in the training image contains defects, and decreases when the target object in the training image does not contain defects, the pseudo-label of the training image can be determined by the reconstruction error between the target feature map and its corresponding reconstructed feature map. After determining the pseudo-label of the training image, the detection model can output the pseudo-label.

[0122] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can input the training image into the residual network, perform feature extraction on the training image, obtain multiple target feature maps of different resolutions, input the multiple target feature maps into the feature reconstruction network, perform feature reconstruction on the multiple target feature maps respectively, obtain the reconstructed feature map corresponding to each target feature map, and then determine the pseudo label of the training image based on the reconstruction error between each target feature map and its corresponding reconstructed feature map. In this way, due to the use of the residual network, multi-scale regional feature information, namely the target feature map, can be obtained. The target feature map not only describes the local features of the corresponding area in the training image, but also encodes its multiple spatial context information, so that it has strong recognition power, which is very beneficial for defect detection. By introducing more prior information through the residual network pre-trained on Imagenet (a large public data set), the target feature map is more accurate, thereby further improving the accuracy of the input pseudo label.

[0123] As an implementation manner of an embodiment of the present invention, the step of determining the pseudo label of the training image based on the reconstruction error between each target feature map and its corresponding reconstructed feature map may include:

[0124] Calculate the reconstruction error between each target feature map and its corresponding reconstructed feature map; according to the reconstruction error, according to the formula A fusion error d is calculated; and a pseudo label of the training image is determined based on a magnitude relationship between the fusion error and a preset threshold.

[0125] Since multiple target feature maps of different resolutions are obtained for the training image during feature extraction, a reconstruction error will be obtained for each target feature map when calculating the reconstruction error. After obtaining the reconstruction error between each target feature map and its corresponding reconstructed feature map, the electronic device can fuse the reconstruction errors of different resolutions corresponding to the training image to obtain a fusion error.

[0126] In one embodiment, the fusion error d can be calculated according to the following formula:

[0127]

[0128] Among them, f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of target feature maps.

[0129] For example, for Figure 5 The residual network and feature reconstruction network shown in the figure have RGB images as input training images and K is 3. Among them, f1, f2 and f3 are the target feature maps of different resolutions output by the residual network, and f′1, f′2 and f′3 are the reconstructed feature maps corresponding to f1, f2 and f3 respectively.

[0130] After obtaining the fusion error, the electronic device can determine the pseudo label of the training image based on the magnitude relationship between the fusion error and a preset threshold. Specifically, the electronic device can compare the fusion error with the preset threshold. If the fusion error is not less than the preset threshold, it indicates that the difference between the reconstructed feature map and the target feature map is significant. In this case, it indicates that there is a high probability that the target object in the training image has a defect.

[0131] When the fusion error is less than a preset threshold, the difference between the reconstructed feature map and the target feature map is small, indicating a high probability that the target object in the training image does not have defects. Therefore, after obtaining the fusion error, the electronic device can determine a pseudo label for the training image based on the relationship between the fusion error and the preset threshold.

[0132] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can calculate the reconstruction error between each target feature map and its corresponding reconstructed feature map, calculate the fusion error based on the above formula, and then determine the pseudo label of the training image based on the size relationship between the fusion error and the preset threshold. In this way, the pseudo label of the training image can be accurately determined based on the size relationship between the fusion error and the preset threshold.

[0133] As an implementation manner of an embodiment of the present invention, the target feature map is a feature map corresponding to pixels in the training image, and the step of determining a pseudo label of the training image based on a magnitude relationship between a fusion error and a preset threshold may include:

[0134] When the fusion error is not less than a preset threshold, the pixel is determined to be a defective pixel in the training image, and a defective area in the training image is determined based on all defective pixels as a pseudo label of the training image.

[0135] For pixel-level defect detection, when using the residual network to extract features from training images, pixel-level feature extraction can be performed on the training images. The resulting target feature map is the feature map corresponding to each pixel in the training image, reflecting the characteristics of each pixel. Furthermore, when reconstructing features using the feature reconstruction network, the resulting reconstructed feature map is also corresponding to each pixel. In this way, the fusion error reflects whether a pixel in the training image is a defective pixel.

[0136] Therefore, when the fusion error is not less than the preset threshold, it means that the pixel point corresponding to the fusion error is a pixel point in the defect area; when the fusion error is less than the preset threshold, it means that the pixel point corresponding to the fusion error is not a pixel point in the defect area.

[0137] In this way, after determining the detection result corresponding to each pixel in the training image, the electronic device can determine the defect area in the training image based on the position of the pixel in the defect area as a pseudo label of the training image, for example, Figure 5 The defect detection result diagram shown in the figure can mark the defect area.

[0138] As an implementation method, the electronic device may determine the minimum bounding rectangle of all pixel points in the defect area as the defect area in the training image, and mark the defect area in a marking frame.

[0139] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can determine the defective pixel points in the training image based on the relationship between the fusion error and the preset threshold, and then determine the defective area in the training image based on the defective pixel points, and obtain a pseudo label that can identify the defective area in the training image.

[0140] As an implementation method of the embodiment of the present invention, Figure 7 As shown, the above weakly supervised training method can include:

[0141] S701, obtaining an initial segmentation network;

[0142] S702, inputting each training image into the initial segmentation network to obtain a prediction score map corresponding to the training image;

[0143] After acquiring training images and an initial segmentation network, the electronic device can input each training image into the initial segmentation network. The initial segmentation network performs pixel-level segmentation on the training image based on the current network parameters, determining which pixels in the training image belong to the defect area and which pixels do not belong to the defect area, and then obtains a prediction score map corresponding to the training image. The pixel value of the pixel in the prediction score map is used to represent the predicted probability that the corresponding pixel in the training image is a defective pixel.

[0144] For example, a pixel value of 0 can be used to indicate that the corresponding pixel point in the training image is a pixel point in the defect area, a pixel value of 255 can be used to indicate that the corresponding pixel point in the training image is not a pixel point in the defect area, and a pixel value of 1-254 can be used to indicate the probability that the corresponding pixel point in the training image is a pixel point in the defect area. The closer to 0, the greater the probability that the corresponding pixel point in the training image is a pixel point in the defect area, and the closer to 255, the smaller the probability that the corresponding pixel point in the training image is a pixel point in the defect area.

[0145] S703, calculating the sampling loss corresponding to each training image based on the prediction score map and the annotated area corresponding to the training image;

[0146] After determining the predicted score map corresponding to each training image, the sampling loss corresponding to the training image can be calculated based on the difference between the predicted score map and the annotated area in the corresponding training image. In order to ensure the accuracy of defect detection of the trained segmentation network, in the weakly supervised training method, the area outside the annotated area of ​​each training image can be strictly background, and both foreground and background exist in the annotated area. A small number of high-confidence foreground / background pixels in the annotated area are used to calculate the sampling loss to ensure the segmentation ability of the segmentation network. Based on this principle, the sampling loss can specifically include the first loss, the second loss, and the third loss. Specifically:

[0147] like Figure 8 As shown in the figure, after the training image is input into the segmentation network, the corresponding prediction score map (Scoremap) can be obtained. Based on the Scoremap and the labeled area (weak label), the first loss (OHEM loss), second loss (OEEM loss) and third loss (Consist loss) corresponding to the training image can be calculated.

[0148] With respect to the aforementioned first loss, for pixels outside the annotated region in each training image, the electronic device can calculate the first loss function value corresponding to each pixel based on the predicted score map. After obtaining the predicted score map corresponding to each training image, since the predicted score map identifies the probability that each pixel in the training image belongs to a defect region, and the annotated region in the training image is the defect region in the calibrated training image, the electronic device can calculate the first loss function value corresponding to each pixel outside the annotated region in the training image based on the difference between the predicted score map and the annotated region and the loss function of the initial segmentation network.

[0149] Next, the electronic device may determine a first preset number of pixel points with the largest first loss function value, and then calculate the average value of the pixel values ​​of these pixel points as the first loss, which can be recorded as L ohem =Topk(L背景 ), that is Figure 8 The OHEM loss in the training image is used. The first preset number can be determined based on factors such as the total number of pixels, and can be, for example, 1%, 2%, or 40% of the total number, etc., without specific limitation. This first loss can reflect the loss of segmentation capability of the initial segmentation network for background areas outside the annotated area in the training image.

[0150] For the second loss, the electronic device can treat each pixel within the annotated area of ​​each training image as a defective pixel and then calculate the corresponding second loss function value for each pixel based on the predicted score map. Because the annotated area in the training image is a rough result obtained by manual coarse annotation, there may be pixels within the annotated area that are not defective.

[0151] In order to enable the trained segmentation network to accurately segment the defect area, all pixels in the labeled area can be first regarded as foreground, that is, all pixels in the labeled area can be regarded as defective pixels. Then, based on the difference between the predicted score map and the labeled area and the loss function of the initial segmentation network, the second loss function value corresponding to each pixel in the labeled area in the training image is calculated.

[0152] Next, the electronic device may determine a second preset number of pixel points with the smallest second loss function value, and then calculate the average value of the pixel values ​​of these pixel points as the second loss, which can be recorded as L oeem-前景 =Bottomk(L 前景 ), that is Figure 8 The second preset number can be determined based on factors such as the total number of pixels, and can be the same as or different from the first preset number. For example, it can be 1%, 2%, or 50 of the total number, and is not specifically limited here.

[0153] For the above-mentioned third loss, for each pixel point in the marked area in each training image, the electronic device can treat each pixel point as a non-defective pixel point, that is, treat all pixels in the marked area as the background, and then calculate the third loss function value corresponding to each pixel point in the marked area in the training image based on the difference between the predicted score map and the marked area and the loss function of the initial segmentation network.

[0154] Next, the electronic device may determine a third preset number of pixels with the smallest third loss function value, and then calculate the average value of the pixel values ​​of these pixels as the third loss, which can be recorded as L oeem-背景 =Bottomk(L 背景 ), that is Figure 7The third preset number can be determined based on factors such as the total number of pixels, and can be the same as or different from the first preset number and the second preset number. For example, it can be 1%, 2%, or 100 of the total number, and is not specifically limited here.

[0155] After obtaining the first, second, and third losses, the sum of the first, second, and third losses can be used as the sampling loss. Alternatively, corresponding weights can be set for the first, second, and third losses, and the weighted sum of the first, second, and third losses can be calculated as the sampling loss. This is reasonable and is not specifically limited here.

[0156] S704, calculating the consistency loss corresponding to the training image based on the pixel values ​​of each pixel point within the marked area of ​​each training image;

[0157] In order to constrain the consistency of the unsampled areas according to the original pixel values ​​of the training image, and to ensure that the output results of the segmentation network corresponding to the areas with similar original pixel values ​​will not differ too much, the consistency loss corresponding to the training image can be calculated based on the pixel values ​​of each pixel point in the marked area of ​​each training image.

[0158] In one embodiment, the electronic device may cluster each pixel within the annotated region of each training image according to its pixel value to obtain a plurality of clusters. Clustering of each pixel within the annotated region according to its pixel value may be performed using methods such as K-means, mean-shift clustering, and Gaussian mixture model clustering, which are not specifically limited herein.

[0159] Since the pixel values ​​of the pixels included in each cluster are similar, when the output results of the segmentation network corresponding to the areas with similar pixel values ​​are consistent, the scores of the features of each cluster after the cross entropy loss should be consistent. Therefore, based on this principle, for each cluster, the electronic device can calculate the mean of the variance of the pixel values ​​corresponding to each pixel point included in it in the predicted score map as the consistency loss, which identifies the difference in the output results of the segmentation network corresponding to the areas with similar pixel values ​​in the second image sample.

[0160] S705 , taking the weighted value of the sampling loss and the consistency loss as the total loss, and adjusting the parameters of the initial segmentation network based on the total loss until the initial segmentation network converges, thereby obtaining a segmentation network as the defect detection model.

[0161] After obtaining the above-mentioned sampling loss and consistency loss, the electronic device can use the weighted value of the sampling loss and consistency loss as the total loss. The weights corresponding to the sampling loss and consistency loss can be set according to actual application scenarios and empirical values, and are not specifically limited here. For example, the weight corresponding to the sampling loss can be 0.7, and the weight corresponding to the consistency loss can be 0.3, etc.

[0162] In one embodiment, in order to ensure that the defect area in each training image can be sampled, the sampling loss and consistency loss can be calculated separately for each second image sample in a batch, and then their respective means are calculated. The means of the sampling loss and the consistency loss are weightedly summed to obtain the total loss.

[0163] After obtaining the total loss, the parameters of the initial segmentation network can be adjusted based on the total loss to make the parameters of the initial segmentation network more and more appropriate and the segmentation results more and more accurate. Training can be stopped until the number of iterations of the training image reaches a preset number or the accuracy of the segmentation network reaches a preset accuracy, and the initial segmentation network converges, resulting in a segmentation network.

[0164] During the training of the above-mentioned feature reconstruction network and segmentation network, the specific method of adjusting the network parameters can be a gradient descent algorithm, a stochastic gradient descent algorithm, etc., which is not specifically limited here.

[0165] The training image is input into the segmentation network, which can determine and output a score map based on the image features of the training image. The pixel value of each pixel in the score map can accurately identify the probability that the pixel at the same position in the training image is a defective pixel, thereby achieving accurate pixel-level defect detection for the training image.

[0166] It can be seen that in the solution provided by the embodiment of the present invention, in the weakly supervised training method, the area outside the marked area is strictly set as the background, and the area inside the marked area is set to have both foreground and background. The pixel values ​​of a small number of foreground / background pixels with high confidence are used in the marked box to calculate the sampling loss to ensure the segmentation ability of the segmentation network. At the same time, the unsampled areas are constrained by consistency according to the original pixel values ​​of the training image to ensure that the output results of the segmentation network corresponding to the areas with similar original pixel values ​​of the training image will not differ too much. Thereby ensuring the detection accuracy of the detection model. At the same time, in the weakly supervised training method, a weakly supervised algorithm of sampling loss-consistency loss matching is proposed, which can obtain good segmentation performance for most industrial scenarios and can be modified based on the existing segmentation network for easy deployment.

[0167] Corresponding to the above-mentioned defect detection model training method, an embodiment of the present invention further provides a defect detection method. The following is an introduction to a defect detection method provided by an embodiment of the present invention.

[0168] like Figure 9 As shown, a defect detection method, the method comprising:

[0169] S901, obtaining an image to be detected;

[0170] S902, inputting the image to be detected into a pre-trained defect detection model to perform defect detection and obtain a defect detection result;

[0171] The defect detection model is trained based on training images and their corresponding pseudo labels using a weakly supervised training method and / or a supervised training method, and the pseudo labels are pseudo labels of the training images determined using an unsupervised training method.

[0172] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can obtain an image to be detected, input the image to be detected into a pre-trained defect detection model to perform defect detection, and obtain a defect detection result; wherein the defect detection model is based on the training image and its corresponding pseudo-label, and is trained using a weakly supervised training method and / or a supervised training method, and the pseudo-label is a pseudo-label of the training image determined using an unsupervised training method. Since in the solution of the embodiment of the present invention, the defect detection model is based on the training image and its corresponding pseudo-label, and is trained using a weakly supervised training method and / or a supervised training method, the accuracy of the defect detection model is high, and therefore the accuracy of the defect detection result is also high.

[0173] When it is necessary to perform defect detection on the target object, the electronic device may execute the above-mentioned step S901, i.e., obtain an image to be detected, wherein the image to be detected may include the target object. The target object may be any product that needs to be inspected for defects. In one embodiment, the electronic device may be integrated with an image acquisition device, and the image acquisition device may be used to acquire an image of the target object as the image to be detected. For example, when the user wants to perform defect detection on the target object, the image acquisition device in the electronic device may be controlled to acquire an image of the target object, and the acquired image may be used as the image to be detected. Alternatively, when a preset defect detection cycle is reached, the electronic device automatically controls the image acquisition device to acquire an image of the target object, and the acquired image may be used as the image to be detected.

[0174] In another embodiment, the electronic device may also obtain an image including a target object from other devices as the image to be detected. For example, the image acquisition device is arranged in the detection link of the industrial production process and is responsible for acquiring the image of the target object. After acquiring the image of the target object, the image acquisition device may send the image to be detected to the electronic device.

[0175] The electronic device can perform defect detection on the image to be detected in real time, or it can store the image to be detected. When a preset defect detection period is reached, the electronic device can retrieve the stored image to be detected and then perform defect detection. Of course, the electronic device can also receive an image of the target object input by the user and use it as the image to be detected. This is reasonable and not specifically limited here.

[0176] After acquiring an image to be inspected, the electronic device can input it into a pre-trained defect detection model. The defect detection model is trained using weakly supervised and / or supervised training methods based on training images and their corresponding pseudo-labels. The pseudo-labels are the pseudo-labels of the training images determined using unsupervised training methods. In other words, the defect detection model is trained using the aforementioned defect detection model training method and can therefore output accurate detection results for the image to be inspected.

[0177] Corresponding to the above-mentioned defect detection model training method, an embodiment of the present invention further provides a defect detection model training device. The following introduces a defect detection model training device provided by an embodiment of the present invention.

[0178] like Figure 10 As shown, a training device for a defect detection model, the device comprising:

[0179] The sample acquisition module 1010 is used to acquire training images;

[0180] a pseudo-label determination module 1020 for determining pseudo-labels of the training images using an unsupervised training method;

[0181] The model training module 1030 is configured to train the defect detection model based on the training images and their corresponding pseudo labels using a weakly supervised training method and / or a supervised training method.

[0182] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can obtain training images; use an unsupervised training method to determine the pseudo labels of the training images; and use a weakly supervised training method and / or a supervised training method to train the defect detection model based on the training images and their corresponding pseudo labels. Since in the solution of the embodiment of the present invention, the pseudo labels of the training images can be obtained by an unsupervised training method first, and then the defect detection model is trained based on the training images and their corresponding pseudo labels, there is no need to obtain defect image samples during the model training process, which solves the problem of low detection accuracy of the trained defect detection model due to the inability to obtain a large number of defect image samples, thereby improving the detection accuracy of the defect detection model. At the same time, in the solution of the embodiment of the present invention, it is also possible to choose to use weakly supervised and / or supervised training methods to train the defect detection model according to the requirements for detection accuracy in different application scenarios, which has the advantage of flexible deployment and minimization of labor costs.

[0183] As an implementation of an embodiment of the present invention, the above-mentioned defect detection model training device may further include:

[0184] A detection result acquisition module, used to obtain the detection result output by the defect detection model;

[0185] A target image acquisition module, configured to acquire a target image when the detection result does not meet a preset detection condition;

[0186] The iterative training module is used to add the target image to the training image and trigger the model training module 1030 until the detection result output by the defect detection model meets the preset detection condition.

[0187] As an implementation of an embodiment of the present invention, the pseudo-label determination module 1020 may include:

[0188] A first data acquisition unit is used to acquire a plurality of sample images and an initial feature reconstruction network;

[0189] The sample image includes a defect-free object.

[0190] A sample feature extraction unit is used to extract features from each sample image to obtain a plurality of sample feature maps of different resolutions corresponding to each sample image;

[0191] A sample feature reconstruction unit, configured to use the initial feature reconstruction network to perform feature reconstruction based on each sample feature map to obtain a reconstructed sample feature map corresponding to the sample feature map;

[0192] The first parameter adjustment unit is used to adjust the parameters of the initial feature reconstruction network based on the reconstruction error between each sample feature map and its corresponding reconstructed sample feature map until the initial feature reconstruction network converges to obtain a feature reconstruction network.

[0193] As an implementation of an embodiment of the present invention, the pseudo-label determination module 1020 may include:

[0194] The feature extraction unit is further used to input the training image into the residual network, perform feature extraction on the training image, and obtain a plurality of target feature maps with different resolutions;

[0195] The feature reconstruction unit is further configured to input the plurality of target feature maps into the feature reconstruction network, and perform feature reconstruction on the plurality of target feature maps respectively to obtain a reconstructed feature map corresponding to each target feature map;

[0196] A pseudo label determination unit is used to determine a pseudo label of the training image based on a reconstruction error between each target feature map and its corresponding reconstructed feature map, and output the pseudo label.

[0197] As an implementation manner of an embodiment of the present invention, the pseudo-label determination unit may include:

[0198] A reconstruction error calculation subunit, configured to calculate a reconstruction error between each target feature map and its corresponding reconstructed feature map;

[0199] The fusion error calculation subunit is used to calculate the reconstruction error according to the formula Calculate the fusion error d;

[0200] a pseudo label determination subunit, which determines a pseudo label of the training image based on a magnitude relationship between the fusion error and a preset threshold;

[0201] Among them, f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of the target feature maps.

[0202] As an implementation manner of an embodiment of the present invention, the target feature map is a feature map corresponding to pixels in the training image;

[0203] The above-mentioned pseudo-label determination subunit is specifically used to determine that the pixel point is a defective pixel point in the training image when the fusion error is not less than a preset threshold, and determine the defective area in the training image based on all defective pixels as the detection result of the target object in the training image.

[0204] As an implementation of an embodiment of the present invention, the model training module 1030 may include:

[0205] A second data acquisition unit, configured to acquire an initial segmentation network;

[0206] a score map determining unit, configured to input each training image into the initial segmentation network to obtain a predicted score map corresponding to the training image;

[0207] The pixel value of a pixel point in the prediction score map represents the predicted probability that the corresponding pixel point in the training image is a defective pixel point.

[0208] A sampling loss determination unit, configured to calculate the sampling loss corresponding to each training image based on the prediction score map and the annotated area corresponding to the training image;

[0209] a consistency loss determining unit, configured to calculate the consistency loss corresponding to the training image based on the pixel values ​​of each pixel point within the marked area of ​​each training image;

[0210] The second parameter adjustment unit is used to take the weighted value of the sampling loss and the consistency loss as the total loss, and adjust the parameters of the initial segmentation network based on the total loss until the initial segmentation network converges, thereby obtaining a segmentation network as the defect detection model.

[0211] As an implementation manner of an embodiment of the present invention, the sampling loss determination unit may include:

[0212] A first loss function value calculation subunit is configured to calculate, for each pixel outside the marked area in each training image, a first loss function value corresponding to each pixel based on the prediction score map;

[0213] A first loss calculation subunit is configured to calculate an average of pixel values ​​of a first preset number of pixel points having the largest corresponding first loss function value as a first loss;

[0214] A second loss function value calculation subunit is configured to calculate, for each pixel in the marked area of ​​each training image, a second loss function value corresponding to each pixel as a defective pixel based on the predicted score map;

[0215] A second loss function value calculation subunit is used to calculate the average value of pixel values ​​of a second preset number of pixel points with the minimum corresponding second loss function value as the second loss;

[0216] A third loss function value calculation subunit is configured to calculate, for each pixel in the marked area of ​​each training image, a third loss function value corresponding to each pixel as a non-defective pixel based on the predicted score map;

[0217] A third loss function value calculation subunit is configured to calculate an average value of pixel values ​​of a third preset number of pixel points having the smallest corresponding third loss function value as a third loss;

[0218] The sampling loss determining subunit is configured to take the sum of the first loss, the second loss and the third loss as the sampling loss.

[0219] As an implementation manner of an embodiment of the present invention, the consistency loss determination unit may include:

[0220] A clustering subunit, configured to cluster each pixel point in the marked area of ​​each training image according to its pixel value to obtain a plurality of clusters;

[0221] The consistency loss determination subunit is used to calculate, for each cluster, the mean of the variances of the pixel values ​​corresponding to the pixel points included in the cluster in the prediction score map as the consistency loss.

[0222] Corresponding to the above-mentioned defect detection method, an embodiment of the present invention further provides a defect detection device. The following is an introduction to a defect detection device provided by an embodiment of the present invention.

[0223] like Figure 11 As shown, a defect detection device, the device comprising:

[0224] An image acquisition module 1110 is used to acquire an image to be detected;

[0225] The image detection module 1120 is used to input the image to be detected into a pre-trained defect detection model to perform defect detection and obtain a defect detection result.

[0226] The defect detection model is trained based on training images and their corresponding pseudo labels using a weakly supervised training method and / or a supervised training method, and the pseudo labels are pseudo labels of the training images determined using an unsupervised training method.

[0227] It can be seen that in the solution provided by the embodiment of the present invention, the electronic device can obtain an image to be detected, input the image to be detected into a pre-trained defect detection model to perform defect detection, and obtain a defect detection result; wherein the defect detection model is based on the training image and its corresponding pseudo-label, and is trained using a weakly supervised training method and / or a supervised training method, and the pseudo-label is a pseudo-label of the training image determined using an unsupervised training method. Since in the solution of the embodiment of the present invention, the defect detection model is based on the training image and its corresponding pseudo-label, and is trained using a weakly supervised training method and / or a supervised training method, the accuracy of the defect detection model is high, and therefore the accuracy of the defect detection result is also high.

[0228] The embodiment of the present invention further provides an electronic device, such as Figure 12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.

[0229] Memory 1203, used for storing computer programs;

[0230] The processor 1201 is configured to implement the defect detection model training method or defect detection method steps described in any of the above embodiments when executing the program stored in the memory 1203 .

[0231] It can be seen that in the solution provided by the embodiment of the present invention, when the electronic device executes the training method of the defect detection model described in any of the above embodiments, it can obtain training images and use unsupervised training to determine the pseudo labels of the training images; based on the training images and their corresponding pseudo labels, the defect detection model can be trained using weakly supervised training and / or supervised training. The pseudo labels of the training images can be obtained by unsupervised training first, and then the defect detection model can be trained based on the training images and their corresponding pseudo labels. Therefore, there is no need to obtain defect image samples during the model training process, which solves the problem of low detection accuracy of the trained defect detection model due to the inability to obtain a large number of defect image samples, thereby improving the detection accuracy of the defect detection model. At the same time, in the solution of the embodiment of the present invention, it is also possible to choose to use weakly supervised and / or supervised training methods to train the defect detection model according to the requirements for detection accuracy in different application scenarios, which has the advantage of flexible deployment and minimization of labor costs. When an electronic device executes the defect detection method described in any of the above embodiments, it can obtain an image to be detected, input the image to be detected into a pre-trained defect detection model for defect detection, and obtain a defect detection result; wherein the defect detection model is trained based on the training images and their corresponding pseudo-labels using a weakly supervised training method and / or a supervised training method, and the pseudo-labels are pseudo-labels of the training images determined using an unsupervised training method. Because in the embodiments of the present invention, the defect detection model is trained based on the training images and their corresponding pseudo-labels using a weakly supervised training method and / or a supervised training method, the defect detection model has high accuracy, and therefore the defect detection result obtained is also highly accurate.

[0232] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0233] The communication interface is used for communication between the above electronic device and other devices.

[0234] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0235] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0236] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the defect detection method steps described in any of the above embodiments are implemented.

[0237] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the defect detection method described in any of the above embodiments.

[0238] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0239] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0240] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between other embodiments. In particular, the device, electronic device, computer-readable storage medium, and computer program product embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0241] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for training a defect detection model, characterized in that: The method comprises: Get training images; Determining pseudo labels of the training images using an unsupervised training method; wherein the pseudo labels are used to represent defect areas in the training images; Based on the training images and their corresponding pseudo labels, a defect detection model is trained using a weakly supervised training method and / or a supervised training method; The step of determining the pseudo labels of the training images using an unsupervised training method includes: Inputting the training image into a residual network, performing feature extraction on the training image, and obtaining a plurality of target feature maps of different resolutions, wherein the target feature map is a feature map corresponding to a pixel point in the training image; Inputting the plurality of target feature maps into a feature reconstruction network, performing feature reconstruction on the plurality of target feature maps respectively, and obtaining a reconstructed feature map corresponding to each target feature map; Calculate the reconstruction error between each target feature map and its corresponding reconstructed feature map; according to the reconstruction error, according to the formula Calculate the fusion error d; when the fusion error is not less than a preset threshold, determine the pixel as a defective pixel in the training image, and determine the defective area in the training image based on all defective pixels as the pseudo label of the training image, where f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of the target feature maps.

2. The method according to claim 1, wherein: The method further comprises: Obtaining a detection result output by the defect detection model; If the detection result does not meet the preset detection condition, acquiring the target image; The target image is added to the training image, and the step of training the defect detection model based on the training image and its corresponding pseudo-label using a weakly supervised training method and / or a supervised training method is returned until the detection result output by the defect detection model meets the preset detection condition.

3. The method according to claim 1, wherein: The unsupervised training method includes: Acquire a plurality of sample images and an initial feature reconstruction network, wherein the sample images include defect-free objects; Perform feature extraction on each sample image to obtain multiple sample feature maps of different resolutions corresponding to each sample image; Using the initial feature reconstruction network, feature reconstruction is performed based on each sample feature map to obtain a reconstructed sample feature map corresponding to the sample feature map; Based on the reconstruction error between each sample feature map and its corresponding reconstructed sample feature map, the parameters of the initial feature reconstruction network are adjusted until the initial feature reconstruction network converges to obtain a feature reconstruction network.

4. The method according to claim 1, wherein: The weakly supervised training method includes: Get the initial segmentation network; Inputting each training image into the initial segmentation network to obtain a prediction score map corresponding to the training image, wherein the pixel value of a pixel point in the prediction score map represents the predicted probability that the corresponding pixel point in the training image is a defective pixel point; Based on the predicted score map and annotated area corresponding to each training image, calculate the sampling loss corresponding to the training image; Calculating the consistency loss corresponding to the training image based on the pixel values ​​of each pixel point within the marked area of ​​each training image; The weighted value of the sampling loss and the consistency loss is taken as the total loss, and the parameters of the initial segmentation network are adjusted based on the total loss until the initial segmentation network converges, thereby obtaining a segmentation network as the defect detection model.

5. The method according to claim 4, wherein: The step of calculating the sampling loss corresponding to each training image based on the predicted score map and the annotated area corresponding to the training image includes: For each pixel outside the marked area in each training image, calculate the first loss function value corresponding to each pixel based on the predicted score map; Calculate the average value of pixel values ​​of a first preset number of pixel points with the largest corresponding first loss function value as the first loss; For each pixel in the marked area of ​​each training image, each pixel is regarded as a defective pixel, and a second loss function value corresponding to each pixel is calculated based on the prediction score map; Calculate the average value of pixel values ​​of a second preset number of pixel points whose corresponding second loss function values ​​are minimum as the second loss; For each pixel in the marked area of ​​each training image, each pixel is regarded as a non-defective pixel, and a third loss function value corresponding to each pixel is calculated based on the prediction score map; Calculate the average of the pixel values ​​of a third preset number of pixels whose corresponding third loss function values ​​are minimum as the third loss; The sum of the first loss, the second loss, and the third loss is used as the sampling loss.

6. The method according to claim 4, wherein: The step of calculating the consistency loss corresponding to each training image based on the pixel values ​​of each pixel point within the marked area of ​​each training image includes: Clustering each pixel point in the marked area of ​​each training image according to the pixel value to obtain multiple clusters; For each cluster, the mean of the variance of the pixel values ​​corresponding to each pixel point included in the cluster in the prediction score map is calculated as the consistency loss.

7. A defect detection method, characterized in that: The method comprises: Obtain the image to be detected; The image to be detected is input into a pre-trained defect detection model for defect detection to obtain a defect detection result, wherein the defect detection model is based on a training image and its corresponding pseudo-label, the pseudo-label is used to represent the defect area in the training image, and is trained using a weakly supervised training method and / or a supervised training method, the pseudo-label is a pseudo-label of the training image determined using an unsupervised training method, and the pseudo-label is determined by inputting the training image into a residual network, performing feature extraction on the training image, and obtaining a plurality of target feature maps of different resolutions, the target feature map being a feature map corresponding to a pixel point in the training image; inputting the plurality of target feature maps into a feature reconstruction network, performing feature reconstruction on the plurality of target feature maps respectively, and obtaining a reconstructed feature map corresponding to each target feature map; calculating the reconstruction error between each target feature map and its corresponding reconstructed feature map; according to the reconstruction error, according to the formula Calculate the fusion error d; when the fusion error is not less than a preset threshold, determine the pixel as a defective pixel in the training image, and determine the defective area in the training image based on all defective pixels as the pseudo label of the training image, f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of the target feature maps.

8. A training device for a defect detection model, characterized in that: The device comprises: A sample acquisition module is used to acquire training images; a pseudo-label determination module, configured to determine a pseudo-label of the training image using an unsupervised training method; wherein the pseudo-label is used to represent a defect area in the training image; A model training module, configured to train a defect detection model based on the training images and their corresponding pseudo labels using a weakly supervised training method and / or a supervised training method; The pseudo-label determination module includes: The feature extraction unit is further configured to input the training image into a residual network, perform feature extraction on the training image, and obtain a plurality of target feature maps of different resolutions, wherein the target feature map is a feature map corresponding to a pixel point in the training image; The feature reconstruction unit is further configured to input the plurality of target feature maps into a feature reconstruction network, perform feature reconstruction on the plurality of target feature maps respectively, and obtain a reconstructed feature map corresponding to each target feature map; The pseudo label determination unit is used to calculate the reconstruction error between each target feature map and its corresponding reconstructed feature map; according to the reconstruction error, according to the formula Calculate the fusion error d; when the fusion error is not less than a preset threshold, determine the pixel as a defective pixel in the training image, and determine the defective area in the training image based on all defective pixels as the pseudo label of the training image, where f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of the target feature maps.

9. A defect detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be detected; An image detection module is used to input the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result, wherein the defect detection model is based on a training image and its corresponding pseudo-label, the pseudo-label is used to represent the defect area in the training image, and is trained using a weakly supervised training method and / or a supervised training method, and the pseudo-label is a pseudo-label of the training image determined using an unsupervised training method; a method for determining the pseudo-label includes inputting the training image into a residual network, performing feature extraction on the training image, and obtaining a plurality of target feature maps of different resolutions, the target feature map being a feature map corresponding to a pixel point in the training image; inputting the plurality of target feature maps into a feature reconstruction network, and performing feature reconstruction on the plurality of target feature maps respectively to obtain a reconstructed feature map corresponding to each target feature map; calculating a reconstruction error between each target feature map and its corresponding reconstructed feature map; according to the reconstruction error, according to the formula Calculate the fusion error d; when the fusion error is not less than a preset threshold, determine the pixel as a defective pixel in the training image, and determine the defective area in the training image based on all defective pixels as the pseudo label of the training image, where f i represents the i-th target feature map, f i ′ represents the reconstructed feature map corresponding to the i-th target feature map, and K represents the number of the multiple target feature maps.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 or 7 when executing a program stored in a memory.

Images