Defect detection method, electronic device and computer readable storage medium

Abstract

The present application discloses a defect detection method, an electronic device and a computer-readable storage medium. The method comprises: determining a candidate defect region in a target object to be detected image; classifying the candidate defect region to obtain a defect classification result; judging whether the defect classification result can be used to determine the defect category of the target object; in response to the defect classification result not being able to be used to determine the defect category of the target object, comparing the features of the candidate defect region with the template features of each candidate defect category to determine the defect category of the target object. In the above manner, the accuracy of the defect category of the target object can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a defect detection method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the industrial field, it is often necessary to detect defects on the surface of materials and products made of materials. Defect detection aims to determine the defect type of the detection object. One method of defect detection is through manual detection, and another method is through neural network detection. Among them, the neural network detection method is widely used because of its high efficiency and can greatly reduce labor costs.

[0003] However, the existing methods of defect detection using neural networks do not provide high accuracy in the defect classification. Summary of the invention

[0004] The present application provides a defect detection method, an electronic device and a computer-readable storage medium, which can solve the problem that the defect classification obtained by the existing defect detection method is not accurate enough.

[0005] In order to solve the above technical problems, a technical solution adopted by the present application is to provide a defect detection method. The method includes: determining a candidate defect area in a target object to be detected image; classifying the candidate defect area to obtain a defect classification result; judging whether the defect classification result can be used to determine the defect category of the target object; in response to the defect classification result not being able to be used to determine the defect category of the target object, comparing the features of the candidate defect area with the template features of each candidate defect category to determine the defect category of the target object.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium, which stores program instructions, and the program instructions can be executed by a processor to implement the above method when executed.

[0008] Through the above-mentioned method, when the defect classification result cannot be used to determine the defect category of the target object, that is, when the defect classification result has a low accuracy, the present application regards the template features of each candidate defect category as the standard for the features of each candidate defect region, and compares the features of the candidate defect region with the template features of each candidate defect category to determine the defect category of the target object. Therefore, even if the classification algorithm used as the basis for classification is not accurate and cannot be used to determine the category of the target object, the defect category of the target object can be determined by comparing it with the template features, thereby improving the accuracy of the defect category of the determined target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flow chart of an embodiment of a defect detection method of the present application;

[0010] Figure 2 yes Figure 1 Specific process diagram of S14;

[0011] Figure 3 is a flow chart of another embodiment of the defect detection method of the present application;

[0012] Figure 4 yes Figure 3 The specific process diagram of S23;

[0013] Figure 5 is a flow chart of an embodiment of a training method for a region determination network;

[0014] Figure 6 It is a schematic diagram of the process of obtaining contrast loss;

[0015] Figure 7 It is a structural diagram of a regional determination network;

[0016] Figure 8 It is a flow chart of an embodiment of a training method for a classification network;

[0017] Fig. 9 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0018] Fig.10 It is a structural schematic diagram of an embodiment of a computer-readable storage medium of the present application; DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "first", "second", and "third" in this application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first", "second", and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments without conflict.

[0022] Before introducing the defect detection method provided by this application, the defect detection method in the related art and its defects are first described:

[0023] In the related art, an image of an object is obtained and defects of the image are directly classified using a classification network; or the defect area is determined from the image and then classified to obtain the defect category of the object.

[0024] The inventor of this application has found through long-term research that the defect classification results obtained by the defect detection method in the related art are not accurate. The reasons for the low accuracy may include at least the following two:

[0025] One of the reasons is that the classification accuracy of the classification network used for classification is not high, resulting in low accuracy of defect classification results obtained by the classification network. The low classification accuracy of the classification network may be caused by defects in the classification network itself; it may also be due to the difficulty in obtaining training samples for a certain defect category, resulting in a small number of training samples for the defect category used to train the classification network during the training phase, making the classification network insufficient in its ability to classify the defect category.

[0026] Another reason is that the defect category cannot be completely defined. When a defect category appears for the first time and has not been defined, and is different from the defined defect categories, the undefined defect category cannot be accurately classified.

[0027] To this end, the defect detection method provided by this application is as follows:

[0028] Figure 1 It is a flow chart of an embodiment of the defect detection method of the present application. It should be noted that if there are substantially the same results, this embodiment does not Figure 1The process sequence shown is limited. Figure 1 As shown, this embodiment may include:

[0029] S11: Determine a candidate defect area in the image to be detected of the target object.

[0030] The defect detection method of the present application is directed to visual objects, which can be materials (such as metals, silicon wafers, plastics), or products obtained by processing materials. For ease of understanding, the present application will be described below using silicon wafers as an example.

[0031] The image to be detected can be the original image of the target object itself, or it can be an image block obtained by segmenting the original image (such as an image block with a size of 512*512). For example, if the defect of the target object is subtle, the image to be detected is an image block of the original image to better locate the defect of the target object; if the defect of the target object is significant, the image to be detected is the original image.

[0032] In some embodiments, defect detection may be performed on the image to be detected to obtain a defect detection frame, and the area corresponding to the defect detection frame in the image to be detected is used as a candidate defect area.

[0033] In some embodiments, defect segmentation may be performed on the image to be inspected to obtain a defect connected domain as a candidate defect region, and the region corresponding to the defect connected domain in the image to be inspected may be used as a candidate defect region.

[0034] In some embodiments, a reference image may be introduced, and with the aid of the reference image, the defect detection area may be determined. Specifically, the reference image is obtained when the target object is defect-free, so the reference image can be used as a defect-free standard, and the pixel-level feature similarity between the image to be detected and the reference image may be obtained. The pixel-level feature similarity includes the feature similarity of corresponding pixels between the image to be detected and the reference image; the area composed of pixels whose corresponding feature similarity in the image to be detected is less than the feature similarity threshold is used as a candidate defect area.

[0035] In some embodiments, the final candidate defect region (intersection or union) may be determined based on the candidate defect region determined by defect detection and the candidate defect region determined after introducing the reference image.

[0036] In some embodiments, the final candidate defect region (intersection or union) may be determined based on the candidate defect region determined by defect segmentation and the candidate defect region determined after introducing the reference image.

[0037] There may be one or more candidate defect regions determined. In some embodiments, candidate defect regions whose distance is less than a distance threshold may be merged, and the merged candidate defect regions are used for subsequent processing. In some embodiments, candidate defect regions that are noises may also be filtered, for example, candidate defect regions whose area is less than an area threshold are filtered out, and the remaining candidate defect regions after filtering are used for subsequent processing.

[0038] S12: Classify the candidate defect areas to obtain defect classification results.

[0039] The classification of the candidate defect regions can be based on any classification network with classification capabilities. The classification network can extract the features of the candidate defect regions; classify the candidate defect regions based on the features of the candidate defect regions to obtain defect classification results. The defect classification results can include the probability that the candidate defect regions belong to each candidate defect category.

[0040] S13: Determine whether the defect classification result can be used to determine the defect category of the target object.

[0041] The defect classification result can be used to determine the defect category of the target object based on the probability that the candidate defect area belongs to each candidate defect category meeting the preset conditions. The preset conditions can be that the probability of belonging to one of the candidate defect categories is the largest and greater than the probability threshold, or that the difference between the probability of belonging to one of the candidate defect categories and the probability of belonging to each of the other candidate defect categories is greater than the difference threshold, etc. Thus, it can be determined whether the probability of the candidate defect area belonging to each candidate defect category meets the preset conditions; in response to not meeting the preset conditions, it is determined that the defect classification result cannot be used to determine the defect category of the target object; in response to meeting the preset conditions, it is determined that the defect classification result can be used to determine the defect category of the target object.

[0042] In response to the defect classification result being unable to be used to determine the defect category of the target object, S14 is executed; otherwise, S15 is executed.

[0043] S14: Compare the features of the candidate defect area with the template features of each candidate defect category to determine the defect category of the target object.

[0044] Each candidate defect category may include a non-defect category and each defined defect category. The template feature may be obtained through calibration. For example, the defect categories of silicon wafers include non-defect, end face collapse, chamfer collapse, end face contamination, reverse texture, multiple pieces, and missing pieces. The template feature of the candidate defect category is the feature of the template area where the candidate defect category exists, and is the standard of the feature under the candidate defect category. The size of the template area is consistent with that of the candidate defect area.

[0045] In some embodiments, the similarity (regional similarity) between the features of the candidate defect regions and the template features of each candidate defect category may be obtained, and the candidate defect category with the largest corresponding region similarity may be used as the defect category of the target object.

[0046] Combined with reference Figure 2 In some embodiments, S14 may include the following sub-steps:

[0047] S141: Obtaining the regional similarity between the features of the candidate defect region and the template features of each candidate defect category.

[0048] S142: Determine whether the target candidate defect category exists in each candidate defect category.

[0049] The region similarity corresponding to the template features of the target candidate defect category meets the similarity requirement.

[0050] The similarity requirement may be that the regional similarity is maximum and greater than a regional similarity threshold, or that the similarity difference between the regional similarity of the target candidate defect category and the regional similarities of other candidate defect categories is greater than a similarity difference threshold, and so on.

[0051] In response to the existence of the target candidate defect category, S143 is executed; in response to the absence of the target candidate defect category, S144 is executed.

[0052] S143: Determine the defect category of the target object as the target candidate defect category.

[0053] S144: Determine the defect category of the target object as a pending category.

[0054] The absence of a target candidate defect category means that the defect category of the target object has not been defined, so it is determined as a pending category.

[0055] It can be understood that, compared with the method of directly taking the candidate defect category with the largest corresponding regional similarity as the defect category of the target object, S141 to S144 take into account the situation where the defect category of the target object is not defined, and thus have higher accuracy.

[0056] S15: Determine the defect category of the target object based on the defect classification result.

[0057] The greater the probability of belonging to a candidate defect category, the greater the possibility that the candidate defect area belongs to the candidate defect category. Therefore, the aforementioned "one of the candidate defect categories" can be used as the defect category of the target object, that is, the candidate defect category with the largest corresponding probability and greater than the threshold is used as the defect category of the target object, or the candidate defect category with the difference between the corresponding probability and the probability of other candidate defect categories greater than the difference threshold is used as the defect category of the target object.

[0058] Through the implementation of this embodiment, when the defect classification result cannot be used to determine the defect category of the target object, that is, when the defect classification result is not accurate, the template features of each candidate defect category are regarded as the standard of the features of each candidate defect area, and the features of the candidate defect area are compared with the template features of each candidate defect category to determine the defect category of the target object. Therefore, even if the classification algorithm used as the basis for classification is not accurate and cannot be used to determine the category of the target object, the defect category of the target object can be determined by comparing with the template features, thereby improving the accuracy of the defect category of the determined target object.

[0059] Figure 3 FIG. 1 is a flow chart of another embodiment of the defect detection method of the present application. It should be noted that if there are substantially the same results, this embodiment is not used. Figure 3 The process sequence shown is limited. This embodiment is a further extension of S11 to determine the candidate defect area by introducing a reference image. Figure 3 As shown, this embodiment may include:

[0060] S21: Obtain pixel-level feature similarity between the image to be detected and the reference image.

[0061] The reference image is acquired when the target object is defect-free.

[0062] The so-called obtaining pixel-level feature similarity refers to obtaining the feature similarity of corresponding pixel points between the image to be detected and the reference image, so as to obtain the feature similarity between each pixel pair between the image to be detected and the reference image.

[0063] The features of the reference image and the features of the image to be detected can be extracted, and the pixel-level feature similarity between the image to be detected and the reference image can be calculated based on the features of the reference image and the features of the image to be detected.

[0064] S22: Perform pixel-level defect segmentation on the image to be inspected to obtain a defect segmentation result.

[0065] The so-called pixel-level defect segmentation refers to classifying defects at each pixel in the image to be detected and obtaining defect segmentation results for each pixel. The defect segmentation results represent the location information of the candidate defect area determined by defect segmentation, which may include the probability that each pixel in the image to be detected belongs to the candidate defect area.

[0066] S23: Determine candidate defect areas based on feature similarity and defect segmentation results.

[0067] In some embodiments, the initial candidate defect region can be determined based on the defect segmentation result, and the initial candidate defect region can be corrected based on the feature similarity to obtain the final candidate defect region. Specifically, the region composed of pixels whose probability of belonging to the candidate defect region in the image to be detected is greater than the probability threshold can be used as the initial candidate defect region; the pixels whose corresponding feature similarity is less than the feature similarity threshold in the initial candidate defect region are removed to obtain the final candidate defect region.

[0068] In some embodiments, the initial candidate defect region can be determined based on feature similarity, and the initial candidate defect region can be modified based on the defect segmentation result. Specifically, the region composed of pixels whose corresponding feature similarity in the image to be detected is less than the feature similarity threshold can be used as the initial candidate defect region; the pixels whose probability of belonging to the candidate defect region is greater than the probability threshold in the initial candidate defect region are removed to obtain the final candidate defect region.

[0069] Combined with reference Figure 4 In some embodiments, S23 may include the following sub-steps:

[0070] S231: Determine a first candidate defect region of the image to be detected based on feature similarity.

[0071] The first candidate defect area is composed of pixel points whose corresponding feature similarity in the image to be detected is less than the feature similarity threshold.

[0072] S232: Determine a second candidate defect region of the image to be detected based on the defect segmentation result.

[0073] The region consisting of pixels in the image to be detected whose probability of belonging to the candidate defect region is greater than a probability threshold may be taken as the second candidate defect region.

[0074] S233: Obtain an intersection area of ​​the first candidate defect area and the second candidate defect area as a candidate defect area.

[0075] Different from other embodiments, in this embodiment, in the process of determining the candidate defect area from the image to be detected, the pixel-level feature similarity between the image to be detected and the background image is considered, that is, the correlation between the background pixels (i.e., the pixels outside the candidate defect area) and the difference of the foreground pixels (i.e., the pixels in the candidate defect area) between the image to be detected and the background image is considered. Therefore, even if there is interference in the texture of the area outside the candidate defect area, the candidate defect area in the image to be detected can be distinguished from the area outside the candidate defect area to a certain extent through the feature similarity. Therefore, through the implementation of this embodiment, the accuracy of the candidate defect area finally determined can be improved.

[0076] Further, based on any of the above embodiments, S11 is implemented through a region determination network. The structure of the region determination network is adapted to the method of determining the candidate defect region. If the method of determining the candidate defect region is defect detection, the region determination network is a neural network capable of defect detection; if the method of determining the candidate defect region is defect segmentation, the region determination network is a neural network capable of defect segmentation; if the method of determining the candidate defect region introduces a reference image, the region determination network is a neural network capable of obtaining pixel-level feature similarity. In some embodiments, the region determination network can be a Big-CNN, such as ResNet, VGG, and the like.

[0077] The following takes the method of determining the candidate defect area by defect segmentation and introducing the reference image as an example to illustrate the training of the area determination network:

[0078] The region determination network is obtained by training a training sample pair, wherein the training sample pair includes a first sample image and a second sample image of a sample object, at least one of the first sample image and the second sample image is annotated with a pixel-level reference feature similarity between the first sample image and the second sample image, and the first sample image is also annotated with a reference defect segmentation result.

[0079] The sample object can be an object of the same category as the target object, or an object of a similar category to the target object. The training sample pairs used to train the region determination network are batched. The first sample image in a single training sample pair can be obtained when the sample object is defect-free or when the sample object is defective. The second sample image can be obtained when the sample object is defect-free or when the sample object is defective. Each pixel value in the reference feature similarity can represent whether the corresponding pixel points between the first sample image and the second sample image are similar. For example, the pixel value representing similarity in the reference feature similarity is distance 0, and the pixel value representing dissimilarity is distance 1. Each pixel value in the reference defect segmentation result can represent whether the corresponding pixel point in the first sample image belongs to the candidate defect area. In the case of insufficient training samples, more training samples can be generated through data augmentation (crop, resize, recolor).

[0080] The region determination network includes two branches, one of which is used to obtain the pixel-level similarity of the first sample feature between the first sample image and the second sample image; and the other branch is used to obtain the sample defect segmentation result. The two branches can be trained separately or combined.

[0081] Figure 5 FIG. 1 is a flow chart of an embodiment of a training method for a region determination network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 5The process sequence shown is limited. In this embodiment, the two branches of the classification network are combined for training, such as Figure 5 As shown, this embodiment may include:

[0082] S31: Obtaining a first sample feature similarity at a pixel level between a first sample image and a second sample image.

[0083] Features of the first sample image and features of the second sample image may be extracted, and pixel-level feature similarity between the first sample image and the second sample image may be calculated based on the features of the first sample image and the features of the second sample image.

[0084] The first sample feature similarity is similar to the feature similarity mentioned above and will not be described in detail here.

[0085] S32: Perform pixel-level defect segmentation on the first sample image to obtain a sample defect segmentation result.

[0086] The sample defect segmentation results are similar to the defect segmentation results mentioned above and will not be repeated here.

[0087] S33: Adjusting parameters of the region determination network based on the difference between the first sample feature similarity and the reference feature similarity, and the difference between the sample defect segmentation result and the reference semantic segmentation result.

[0088] Among them, loss 1 (such as distance metric loss) can be obtained based on the difference between the first sample feature similarity and the reference feature similarity; and loss 2 (such as contrast loss) can be obtained based on the difference between the sample defect segmentation result and the reference semantic segmentation result; loss 1 and loss 2 are weighted and used to adjust the parameters of the region determination network until the training end condition is met. The training end condition can be that the loss is small enough, the number of training times is large enough, the training time is long enough, etc.

[0089] Combined with reference Figure 6 , the acquisition of the contrast loss between the sample defect segmentation result and the reference semantic segmentation result is explained. x represents the first sample image, y represents the second sample image, x is annotated with the reference feature similarity Y, x and y are mapped to the feature space, and the feature f(x) of x and the feature f(y) of y are obtained; the Euclidean distance Dw (first sample feature similarity) between f(x) and f(y) is obtained; the contrast loss is constructed based on the difference between Dw and Y:

[0090]

[0091] Wherein, m is a margin value greater than 0, a constant.

[0092] By training the region determination network with contrast loss, the feature similarity between different pixels belonging to the candidate defect area can be improved in the features of the image to be detected / reference image extracted by the region determination network during the application process, and the feature similarity between the pixels belonging to the candidate defect area and the pixels belonging to the area outside the candidate defect area can be reduced, which plays a positive role in the subsequent feature similarity calculation and defect segmentation.

[0093] Combine as follows Figure 7 , taking an example, the determination of candidate defect regions by the region determination network is described in detail:

[0094] The region determination network includes two branches. The first branch includes a feature extraction layer and a feature similarity acquisition layer. The second branch shares the feature extraction layer with the first branch, and also includes a defect segmentation layer.

[0095] The image A to be detected and the reference image B are sent to the region determination network; the feature extraction layer extracts feature a of A and feature b of B; the feature extraction layer obtains the pixel-level feature similarity between feature a and feature b; the defect segmentation layer performs pixel-level defect segmentation on feature a to obtain the defect segmentation result; the feature similarity and the defect segmentation result are restored to the size of A / B; the first candidate defect region in A is determined based on the restoration result of the feature similarity, and the second candidate defect region in A is determined based on the restoration result of the defect segmentation result; the intersection area of ​​the first candidate defect region and the second candidate defect region is used as candidate defect region 1.

[0096] Further, based on any of the above embodiments, S12 is implemented based on a classification network. The classification network may be, but is not limited to, resnext50. The classification network is obtained by training based on a sample set, the sample set includes a plurality of third sample images of sample objects, and the third sample images are annotated with reference defect classification results. The reference defect classification results characterize the defect category of the third sample image.

[0097] The training of the classification network can continue to use the aforementioned first sample image / second sample image, that is, the third sample image can be the first sample image / second sample image, or other sample images. The classification network can include a feature extraction layer and a classification layer (softmax layer), the feature extraction layer is used to extract features of the third sample image, and the classification layer is used to classify the third sample image based on the features of the third sample image to obtain a sample defect classification result of the sample object. The training of the classification network can be implemented based on at least one of the features of the third sample image and the sample defect classification result.

[0098] Figure 8 1 is a flow chart of an embodiment of the training method of the classification network of the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 8 In this embodiment, the classification network is trained by using the features of the third sample image and the sample defect classification results. Figure 8 As shown, this embodiment may include:

[0099] S41: Extracting features of each third sample image.

[0100] S42: Obtain the second sample feature similarities between the features of each third sample image.

[0101] S43: Determine a first loss based on the second sample feature similarity of the third sample image under the same reference defect classification result and the second sample feature similarity of the third sample image under different reference defect classification results.

[0102] That is, the first loss is determined based on the similarity of the second sample features of the third sample images of the same defect category and the similarity of the second sample features of the third sample images of different defect categories.

[0103] S44: Classify the features of each third sample image to obtain a sample defect classification result of each third sample image.

[0104] The sample defect classification results are similar to the above defect classification results and will not be repeated here.

[0105] S45: Determine a second loss based on the difference between the sample defect classification result and the reference defect classification result.

[0106] S46: Based on the first loss and the second loss, adjust the parameters of the classification network.

[0107] The first loss may be a distance metric loss. Under the constraint of the distance metric loss, the distance between the features of the images of the same defect category extracted by the classification network (intra-class distance) becomes smaller and smaller, while the distance between the features of the images of different defect categories (inter-class distance) becomes larger and larger.

[0108] Fig. 9 Schematic diagram of the structure of an embodiment of the electronic device of the present application. Fig. 9 As shown, the electronic device includes a processor 21 and a memory 22 coupled to the processor 21 .

[0109] The memory 22 stores program instructions for implementing the method of any of the above embodiments; the processor 21 is used to execute the program instructions stored in the memory 22 to implement the steps of the above method embodiments. The processor 21 may also be referred to as a CPU (Central Processing Unit). The processor 21 may be an integrated circuit chip with signal processing capabilities. The processor 21 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0110] Fig.10 Schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present application. Fig.10 As shown, the computer-readable storage medium 30 of the embodiment of the present application stores program instructions 31, and the program instructions 31 are executed to implement the method provided in the above embodiment of the present application. Among them, the program instructions 31 can form a program file and be stored in the above computer-readable storage medium 30 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 30 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0112] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A defect detection method, It is characterized in that include: Determine a candidate defect region in the image to be inspected of the target object; Classifying the candidate defect areas to obtain defect classification results; Determining whether the defect classification result can be used to determine the defect category of the target object; In response to the defect classification result not being able to be used to determine the defect category of the target object, comparing the features of the candidate defect region with template features of each candidate defect category to determine the defect category of the target object; The comparing the features of the candidate defect area with the template features of each candidate defect category to determine the defect category of the target object includes: Obtaining the regional similarity between the feature of the candidate defect region and the template feature of each candidate defect category; Determine whether there is a target candidate defect category among the candidate defect categories, and whether the region similarity corresponding to the template feature of the target candidate defect category meets the similarity requirement; In response to the existence of the target candidate defect category, determining the defect category of the target object to be the target candidate defect category; In response to the target candidate defect category not existing, the defect category of the target object is determined to be a pending category.

2. The method according to claim 1, It is characterized in that The defect classification result includes the probability that the candidate defect area belongs to each candidate defect category; The determining whether the defect classification result can be used to determine the defect category of the target object includes: Determine whether the probability that the candidate defect region belongs to each candidate defect category meets a preset condition, wherein the preset condition is that the difference between the probability of belonging to one of the candidate defect categories and the probability of belonging to other candidate defect categories is greater than a difference threshold; In response to not satisfying the preset condition, determining that the defect classification result cannot be used to determine the defect category of the target object; In response to satisfying the preset condition, it is determined that the defect classification result can be used to determine the defect category of the target object.

3. The method according to claim 2, It is characterized in that The method further comprises: In response to the defect classification result being able to be used to determine the defect category of the target object, one of the candidate defect categories is used as the defect category of the target object.

4. The method according to claim 1, It is characterized in that The step of determining a candidate defect region in the image to be detected of the target object comprises: Obtaining pixel-level feature similarity between the image to be inspected and a reference image, wherein the reference image is obtained when the target object has no defects; Performing pixel-level defect segmentation on the image to be detected to obtain a defect segmentation result, wherein the defect segmentation result represents position information of the candidate defect area determined by defect segmentation; The candidate defect area is determined based on the feature similarity and the defect segmentation result.

5. The method according to claim 4, It is characterized in that The determining the candidate defect area based on the feature similarity and the defect segmentation result includes: Determine a first candidate defect region of the image to be detected based on the feature similarity, wherein the first candidate defect region is composed of pixel points in the image to be detected whose corresponding feature similarity is less than a feature similarity threshold; Based on the defect segmentation result, determining a second candidate defect area of ​​the image to be detected; An intersection area of ​​the first candidate defect area and the second candidate defect area is obtained as the candidate defect area.

6. The method according to claim 4, It is characterized in that The step of determining the candidate defect area in the image to be detected of the target object is implemented based on a region determination network, and the region determination network is trained by training sample pairs, and the training sample pairs include a first sample image and a second sample image of the sample object, and at least one of the first sample image and the second sample image is annotated with a pixel-level reference feature similarity between the first sample image and the second sample image, and the first sample image is also annotated with a reference defect segmentation result.

7. The method according to claim 6, It is characterized in that The training steps of the region determination network include: Obtaining a first sample feature similarity at a pixel level between the first sample image and the second sample image; Performing pixel-level defect segmentation on the first sample image to obtain a sample defect segmentation result; Based on the difference between the first sample feature similarity and the reference feature similarity, and the difference between the sample defect segmentation result and the reference semantic segmentation result, the parameters of the region determination network are adjusted.

8. The method according to claim 1, It is characterized in that The step of classifying the candidate defect areas to obtain defect classification results is implemented based on a classification network, and the classification network is trained based on a sample set, and the sample set includes a third sample image of multiple sample objects, and the third sample image is annotated with a reference defect classification result.

9. The method according to claim 8, It is characterized in that The training steps of the classification network include: extracting features of each of the third sample images; Obtaining second sample feature similarities between features of each of the third sample images; determining a first loss based on a second sample feature similarity of the third sample image under the same reference defect classification result and a second sample feature similarity of the third sample image under different reference defect classification results; Classifying the features of each of the third sample images to obtain a sample defect classification result of each of the third sample images; Determining a second loss based on a difference between the sample defect classification result and the reference defect classification result; Based on the first loss and the second loss, a parameter of the classification network is adjusted.

10. A defect detection method, It is characterized in that include: Determine a candidate defect region in the image to be inspected of the target object; Classifying the candidate defect areas to obtain defect classification results; Determining whether the defect classification result can be used to determine the defect category of the target object; In response to the defect classification result not being able to be used to determine the defect category of the target object, comparing the features of the candidate defect region with template features of each candidate defect category to determine the defect category of the target object; The step of determining a candidate defect region in the image to be detected of the target object comprises: Obtaining pixel-level feature similarity between the image to be inspected and a reference image, wherein the reference image is obtained when the target object has no defects; Performing pixel-level defect segmentation on the image to be detected to obtain a defect segmentation result, wherein the defect segmentation result represents position information of the candidate defect area determined by defect segmentation; The candidate defect area is determined based on the feature similarity and the defect segmentation result.

11. The method according to claim 10, It is characterized in that The defect classification result includes the probability that the candidate defect area belongs to each candidate defect category; The determining whether the defect classification result can be used to determine the defect category of the target object includes: Determine whether the probability that the candidate defect region belongs to each candidate defect category meets a preset condition, wherein the preset condition is that the difference between the probability of belonging to one of the candidate defect categories and the probability of belonging to other candidate defect categories is greater than a difference threshold; In response to not satisfying the preset condition, determining that the defect classification result cannot be used to determine the defect category of the target object; In response to satisfying the preset condition, it is determined that the defect classification result can be used to determine the defect category of the target object.

12. The method according to claim 11, It is characterized in that The method further comprises: In response to the defect classification result being able to be used to determine the defect category of the target object, one of the candidate defect categories is used as the defect category of the target object.

13. The method according to claim 10, It is characterized in that The determining the candidate defect area based on the feature similarity and the defect segmentation result includes: Determine a first candidate defect region of the image to be detected based on the feature similarity, wherein the first candidate defect region is composed of pixel points in the image to be detected whose corresponding feature similarity is less than a feature similarity threshold; Based on the defect segmentation result, determining a second candidate defect area of ​​the image to be detected; An intersection area of ​​the first candidate defect area and the second candidate defect area is obtained as the candidate defect area.

14. The method according to claim 10, It is characterized in that The step of determining the candidate defect area in the image to be detected of the target object is implemented based on a region determination network, and the region determination network is trained by training sample pairs, and the training sample pairs include a first sample image and a second sample image of the sample object, and at least one of the first sample image and the second sample image is annotated with a pixel-level reference feature similarity between the first sample image and the second sample image, and the first sample image is also annotated with a reference defect segmentation result.

15. The method according to claim 14, It is characterized in that The training steps of the region determination network include: Obtaining a first sample feature similarity at a pixel level between the first sample image and the second sample image; Performing pixel-level defect segmentation on the first sample image to obtain a sample defect segmentation result; Based on the difference between the first sample feature similarity and the reference feature similarity, and the difference between the sample defect segmentation result and the reference semantic segmentation result, the parameters of the region determination network are adjusted.

16. The method according to claim 10, It is characterized in that The step of classifying the candidate defect areas to obtain defect classification results is implemented based on a classification network, and the classification network is trained based on a sample set, and the sample set includes a third sample image of multiple sample objects, and the third sample image is annotated with a reference defect classification result.

17. The method according to claim 16, It is characterized in that The training steps of the classification network include: extracting features of each of the third sample images; Obtaining second sample feature similarities between features of each of the third sample images; determining a first loss based on a second sample feature similarity of the third sample image under the same reference defect classification result and a second sample feature similarity of the third sample image under different reference defect classification results; Classifying the features of each of the third sample images to obtain a sample defect classification result of each of the third sample images; Determining a second loss based on a difference between the sample defect classification result and the reference defect classification result; Based on the first loss and the second loss, a parameter of the classification network is adjusted.

18. An electronic device, It is characterized in that The method comprises a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 17.

19. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program instructions, which can be executed by a processor and implement the method according to any one of claims 1 to 17 when executed.

Images