Defect type detection method and device, electronic equipment and storage medium

Abstract

The invention provides a defect type detection method and device, electronic equipment and a storage medium. The detection method comprises the following steps: determining a plurality of first defect areas and corresponding first defect types in a first image of a target object acquired by first image acquisition equipment, and screening out defect parts to be detected from the plurality of first defect areas; second image acquisition equipment is used for acquiring a to-be-detected image comprising the defect part of the target object, and the image quality of the first image is different from that of the to-be-detected image; aiming at the to-be-detected image, determining a defect area corresponding to the defect part; for the to-be-detected image, determining a defect region feature of the defect region, the dimension size of the defect region feature on each dimension being a preset value irrelevant to the area of the defect region; and determining the defect type of the target object in the defect part based on the defect area features. By adopting the method and the device, the calculation amount of secondary reinspection can be reduced, and the method and the device can be better focused on concerned defect types.

Description

Technical Field

[0001] The present application relates to the field of industrial inspection, and more specifically to a defect type inspection method and device, electronic equipment and storage medium. Background Art

[0002] In the industrial field, with the continuous development of production technology, people have put forward higher requirements for the production quality and production efficiency of industrial products. With the development of image processing, more and more product inspections are based on images. If the inspection accuracy is low, the practicality of the image-based inspection method will also be reduced.

[0003] For example, in the wafer inspection process, the related technology usually uses automated optical inspection (AOI) equipment to perform appearance inspection on the wafer to determine the type of defects on the wafer. However, the different defects on the wafer are highly random, complex and changeable. In some existing defect detection methods, a high-magnification color camera is used to collect color images of the wafer for manual judgment by staff, which is inefficient. In other defect detection methods, a low-magnification camera is used to collect a grayscale image of the wafer and match it with an ideal template collected in advance to perform defect detection on the wafer. The accuracy of detection by this method is difficult to guarantee. Summary of the invention

[0004] The present application is proposed in view of the above problems. The present application provides a defect type detection method, a defect type detection device, an electronic device and a storage medium.

[0005] According to one aspect of the present application, a defect type detection method is provided. The defect type detection method includes:

[0006] In the image to be tested of the target object, determining a defect area corresponding to a defective portion of the target object;

[0007] Determine the defect region feature of the defect region for the image to be tested, wherein the dimension size of the defect region feature in each dimension is a preset value that is independent of the area of ​​the defect region;

[0008] Based on the defect area characteristics, the defect type of the target object is determined.

[0009] Exemplarily, for the image to be tested, determining the defect area features of the defect area includes:

[0010] Performing feature extraction on the image to be tested to obtain initial features of the image to be tested;

[0011] The initial features corresponding to the defect area are fused to obtain the defect area features of the defect area.

[0012] Exemplarily, feature fusion is performed on the initial features corresponding to the defective region to obtain defective region features of the defective region, including:

[0013] Determine a first preset number of anchor points among feature points of the initial features corresponding to the defect area, and determine anchor point features of the anchor points based on the initial features corresponding to the defect area;

[0014] For each anchor point,

[0015] Based on a second preset number of neighboring feature points of the anchor point, determine the neighboring point features corresponding to the anchor point, wherein the neighboring feature points corresponding to the anchor point are the feature points that are the nearest neighbors of the anchor point in the initial features;

[0016] Determine a deviation feature between the anchor point feature of the anchor point and the neighboring point feature corresponding to the anchor point as the deviation feature corresponding to the anchor point;

[0017] Determine a splicing feature corresponding to the anchor point based on the anchor point feature of the anchor point and the deviation feature corresponding to the anchor point, wherein in at least one dimension, the dimension size of the splicing feature corresponding to the anchor point is greater than the dimension size of the defect area feature;

[0018] Performing a first fusion operation on the splicing feature corresponding to the anchor point to obtain a fused feature corresponding to the anchor point, wherein the fused feature and the defect area feature have the same dimension size in each dimension;

[0019] A second fusion operation is performed on the fusion features corresponding to each of the first preset number of anchor points to obtain defect area features.

[0020] Exemplarily, determining the splicing feature corresponding to the anchor point based on the anchor point feature of the anchor point and the deviation feature corresponding to the anchor point includes any one of the following:

[0021] splicing the anchor point feature of the anchor point with the deviation feature corresponding to the anchor point to obtain the splicing feature corresponding to the anchor point; or

[0022] The anchor point feature of the anchor point is mapped into a feature space through a feature mapping layer in a defect classification model to obtain a mapping feature corresponding to the anchor point, wherein the anchor point feature of the anchor point and the mapping feature corresponding to the anchor point are in different feature spaces; the mapping feature corresponding to the anchor point and the deviation feature corresponding to the anchor point are spliced ​​to obtain a spliced ​​feature corresponding to the anchor point, wherein the defect classification model is used to determine the defect type of the target object based on the defect area feature.

[0023] Exemplarily, determining a first preset number of anchor points among feature points of the initial features corresponding to the defect area, and determining anchor point features of the anchor points based on the initial features corresponding to the defect area includes:

[0024] Performing farthest point sampling on feature points of the initial features corresponding to the defect area to obtain a first preset number of anchor points;

[0025] Based on the initial features corresponding to the defect area, determining first features respectively corresponding to a first preset number of anchor points;

[0026] For each anchor point, the first feature corresponding to the anchor point and the position information corresponding to the anchor point in the initial feature are fused to obtain the anchor point feature corresponding to the anchor point.

[0027] Exemplarily, feature fusion is performed on the initial features corresponding to the defective region to obtain defective region features of the defective region, including:

[0028] According to the positions of the pixels included in the defect area, interpolation sampling is performed on the initial features corresponding to the defect area to obtain a first number of defect features, wherein the first number is the total number of feature points corresponding to the defect area in the initial features, and the defect features and the defect area features have the same dimension size in each dimension;

[0029] The first number of defect features are averaged to obtain a defect region feature of the defect region.

[0030] Exemplarily, determining a plurality of first defect regions in a first image of a target object acquired by a first image acquisition device includes: determining the plurality of first defect regions of the target object based on a difference between the first image and a template image of the target object.

[0031] Exemplarily, determining the defect type of the target object based on the defect area feature includes any of the following:

[0032] Input the defect area features into the classification head in the defect classification model for classification to obtain the defect type of the target object; or

[0033] The defect area feature is classified and compared with multiple preset features in the preset feature library to obtain the similarity between the defect area feature and each preset feature; the defect type of the target object is determined according to the similarity between the defect area feature and each preset feature and the defect type corresponding to each preset feature.

[0034] Exemplarily, the above detection method further includes:

[0035] Determine the defect level of the target object based on the defect type of the target object;

[0036] If the defect level meets the preset conditions, the target object is reclassified to obtain a new defect type corresponding to the target object.

[0037] According to another aspect of the present application, a defect type detection device is also provided. The defect type detection device may include: a defect area determination module, a feature determination module and a defect type determination module;

[0038] A defect area determination module is used to determine the defect area corresponding to the defective part of the target object in the image to be tested of the target object;

[0039] A feature determination module, used for determining defect region features of a defect region for the image to be tested, wherein the dimension size of the defect region features in each dimension is a preset value that is independent of the area of ​​the defect region;

[0040] The defect type determination module is used to determine the defect type of the target object based on the defect area characteristics.

[0041] According to another aspect of the present application, an electronic device is provided. The electronic device includes: a processor and a memory. The memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect type detection method when the processor is running.

[0042] According to another aspect of the present application, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned defect type detection method when running.

[0043] According to another aspect of the present application, a computer program product is also provided. The computer program product includes computer program instructions, and the computer program instructions are used to execute the above-mentioned defect type detection method when running.

[0044] According to the above scheme of the embodiment of the present application, the defective area of ​​the target object and the defect type corresponding to the defective area are first determined, and then these defective areas can be screened. As a result, only some defective areas in need are re-judged. This can significantly improve the efficiency of defect detection of the target object. In addition, in order to increase the computing speed and improve the accuracy, an image with low image quality can be used for initial judgment, and then a camera with high image quality can be used to take a second shot of the area of ​​high concern and perform secondary processing using the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0046] Figure 1A A schematic diagram showing a defect type detection method according to an embodiment of the present application is shown;

[0047] Figure 1B A schematic flow chart showing a defect type detection method according to an embodiment of the present application is shown;

[0048] Figure 2 A schematic diagram showing a generation process of defect region features according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram showing a defect area and its corresponding defect features in the initial features according to an embodiment of the present application is shown;

[0050] Figure 4 A schematic block diagram showing a defect type detection device according to an embodiment of the present application; and

[0051] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0053] In the field of industrial inspection, the appearance of wafer defects of different defect types may vary greatly, and the appearance of wafer defects of the same defect type may also vary greatly. Therefore, it is difficult to ensure the accuracy of defect detection by matching and comparing with ideal templates. Directly processing the large number of wafer images obtained in this process using machine learning models will require huge computational complexity.

[0054] In order to at least partially solve the above problems, an embodiment of the present application provides a method for detecting defect types. Figure 1A A schematic diagram of a defect type detection method according to an embodiment of the present application is shown.

[0055] like Figure 1AAs shown, the embodiment of the present application can first use a grayscale camera to capture a grayscale image of the target object. Then, by detecting and identifying these grayscale images, it is preliminarily determined that there are defect types in these grayscale images and the defect positions corresponding to each defect type. Then, the defect positions that need secondary detection are screened out. Based on these position information, a color camera is used to capture a color image of the defect to be detected of the target object. The position information and the defect image to be detected can be input into a defect category detection module to determine the defect category of the target object in the defect image to be detected. It can be understood that the detection module corresponds to Figure 1A The content in the dotted box. In the process of secondary detection of these defect images to be detected by using the detection module, the position information and the defect image to be detected are input into the large model for classification. The classification result is threshold-controlled according to the confidence of each defect category, and the defect image to be detected is divided into two parts, namely the first defect image whose classification confidence is greater than the first threshold and the second defect image whose classification confidence is less than or equal to the first threshold. Optionally, the defect category of the classification result corresponding to the first defect image is manually re-judged. The second defect image is clustered. Optionally, the defect category of the clustered second defect image is manually re-judged. Determine whether the defect category of the defect image to be detected is correct through manual re-judgment. If correct, determine the defect category of the defect image to be detected, output the result, and the defect image to be detected and the corresponding defect category can be used to train the large model with a small sample. If incorrect, the defect category of the defect image to be detected is corrected and marked, and the defect image to be detected and the corresponding defect category after correction and marking can output the result, and it can be used to train the clustering algorithm and the large model with a small sample.

[0056] The following will refer to Figure 1B , specifically describing the defect type detection method of the embodiment of the present application.

[0057] like Figure 1B As shown, the method may include the following steps S110 to S150.

[0058] In step S110, a plurality of first defect regions and corresponding first defect types are determined in a first image of a target object acquired by a first image acquisition device, and defective parts to be detected are screened out from the plurality of first defect regions.

[0059] In the present application, the target object may be any type of object such as a wafer, an electronic component, etc. The first image acquisition device may be a device for acquiring low-resolution images. In order to effectively screen a large number of wafer images and reduce the amount of subsequent calculations, the embodiment of the present application may first process low-resolution or low-quality images to screen out the detection area that needs secondary detection. To this end, the first image acquisition device may be a black camera or a grayscale camera. In the embodiment of the present application, after the first image is acquired, a plurality of first defective areas in the image to be tested may be determined using a conventional image processing method. Specifically, the defective area may be determined according to the pixel characteristics of the first image. For example, based on the difference between the pixel values ​​of the pixels in the defective area in the image to be tested and the pixel values ​​of the pixels in other areas, the image to be tested may be segmented using methods such as threshold segmentation, seed growth, and edge detection to determine the defective area. The first defective part may be a part of the target object that does not meet the quality inspection standard. For example, a single crystal with cracks in a wafer, a part of a single crystal with foreign matter, etc.

[0060] Thus, the embodiments of the present application can obtain a large number of defective areas. Subsequently, the embodiments of the present application can perform a secondary screening on these defective areas. For the target object being a wafer, the probability of secondary defects occurring therein is very high relative to the main defects, such as dust, color difference, and water stains. If all wafers are subjected to complex defect classification detection using high-precision images, the efficiency will be very low. Based on this, in the case where the defective areas correspond to a variety of defect types, the defective areas that have been confirmed as secondary defects using traditional algorithms can be screened out by screening.

[0061] To this end, those skilled in the art need to predetermine the major defect level and the minor defect level. The defect level is used to indicate the severity of the defect. In a specific embodiment, the defect level may include major defects and minor defects. After the defective areas of the minor defects are screened out, the embodiment of the present application will perform a secondary shot on the defective parts of these major defects, that is, in step S120, a second image acquisition device is used to acquire an image to be tested including the defective part of the target object, wherein the image quality of the first image is different from that of the image to be tested. The second image acquisition device mentioned here refers to a device whose image quality (for example, image resolution) of the captured image is higher than that of the first image acquisition device. For example, the second image acquisition device may be a high-definition color camera.

[0062] In the case where defective parts that need to be photographed again have been determined based on the first step, the defect type detection device of the present application can control the second image acquisition to move around (for example, above) these defective parts to photograph them, thereby obtaining the image to be tested containing these defective parts.

[0063] In step S130, a defective area corresponding to a defective portion of the target object is determined in the image to be tested of the target object.

[0064] It should be noted that although the shooting is performed on the defective parts that have been determined in the previous processing, the defect position in each image to be tested is unclear, so it is necessary to detect the defective area from the image to be tested. In some embodiments, the image to be tested can be input into a trained detection model (such as a trained convolutional neural network, a generative adversarial network, etc.) to output the defective area corresponding to the defective part of the target object in the image to be tested. The training process of the detection model may include: obtaining a training data set. The detection model is iteratively trained through the training data set, and the model parameters of the detection model are adjusted through the loss function until the training is completed. The above-mentioned training data set includes multiple images, each image is marked with a defective area. The conditions for the completion of the above-mentioned training may be that the loss function value calculated by the loss function tends to be stable, the number of iterations reaches a preset number, etc. Based on the above-mentioned training process, a detection model can be obtained to determine the defective area in the image to be tested.

[0065] In step S140, for the image to be tested, the defect region feature of the defect region is determined, wherein the dimension size of the defect region feature in each dimension is a preset value that is irrelevant to the area of ​​the defect region.

[0066] In some embodiments, feature extraction processing can be performed on the defective area in the image to be tested. For example, by using a fixed-size convolution kernel and a fixed moving step length, sliding sampling is performed in the defective area to obtain the feature to be processed. Then, the feature points in the feature to be processed can be deleted, or the feature points of the feature to be processed can be filled, so that the defective area features with a dimension size of a preset value in each dimension can be obtained. Combined with the actual scene, the morphology of the defective area may be different between different defective areas. For example, the defective area where the crack on the wafer is located is usually slender, while the defective area where the pin mark out of bounds on the wafer is located is usually close to a square, and the morphology of the above defective areas is obviously different. In addition, due to the individuality of defects, the respective area sizes and aspect ratios of defective areas belonging to the same defect type may also be different. Therefore, for different defective areas, using the same convolution kernel and the same moving step length, the feature size (feature size includes the dimension size in each dimension) of the feature to be processed is also different. Therefore, the feature to be processed can be deleted and filled to obtain defective area features that are independent of the area of ​​the defective area.

[0067] In some embodiments, the size of the moving step can also be dynamically adjusted for different defect areas to obtain defect area features with fixed feature sizes. For example, if the moving step is divided into a lateral moving step and a longitudinal moving step, the size of the lateral moving step can be positively correlated with the lateral size of the defect area, and the longitudinal moving step can be positively correlated with the longitudinal size of the defect area. This enables defect areas of different sizes to obtain defect area features with fixed feature sizes after being convolved with a dynamically adjusted convolution kernel.

[0068] The defect area feature can be represented by a one-dimensional vector, a two-dimensional matrix, or a three-dimensional tensor. The preset values ​​on different dimensions can be the same or different, and the size of the preset value has nothing to do with the area of ​​the defect area. For example, if the characteristic size of the defect area feature obtained is 1*768, then the "1" in 1*768 corresponds to the first dimension, and the preset value of the dimension size of the defect area feature in the first dimension is 1. The "768" in 1*768 corresponds to the second dimension, and the preset value of the dimension size of the defect area feature in the second dimension is 768. The specific total number of dimensions and the preset value of each dimension can be determined according to the needs of the technician, and the embodiments of the present application do not limit this.

[0069] In step S150, the defect type of the target object is determined based on the defect region characteristics.

[0070] The defect type is used to indicate the type of defect of the target object. For example, if the target object is a wafer, the defect type may include: cracks, foreign matter, pin marks out of bounds, scratches, etc., depending on the quality inspection standard. In some embodiments, the defect type may be determined based on an algorithm or a machine learning model, which is not limited in the present application.

[0071] According to the defect type detection method provided in the embodiment of the present application, improvements can be made in the control of the entire detection process, that is, the defective part with defects and the corresponding defect type are first located through the first image with low image quality, and the defective area that needs secondary re-inspection is screened out from these defective parts, and then subsequent operations are performed, which greatly reduces the subsequent processing volume and is more focused and more accurate. In the image to be tested of the target object, the defect area corresponding to the defective part of the target object is determined. Then the defect area characteristics of the defective area are determined. Finally, based on the defect area characteristics, the defect type of the target object is determined. In this scheme, by determining the defect area characteristics, defects of different areas in the image to be tested can be represented by features of a uniform scale, which is convenient for reducing the classification difficulty of subsequent classification processing and also ensures the classification accuracy of defects, which is conducive to improving the detection speed of subsequent defect types.

[0072] Exemplarily, step S130 of determining the defective area corresponding to the defective portion of the target object in the image to be tested of the target object may include: determining the defective area corresponding to the defective portion of the target object based on a difference between the image to be tested and a template image of the target object.

[0073] The template image may be a preset image of the target object that does not include defective parts. For example, if the target object is a wafer, the template image is an image of a wafer that meets the quality inspection standards. In some embodiments, the template image and the image to be tested may be preprocessed (for example, denoising, contrast enhancement, etc.) respectively. The template image and the image to be tested are then registered to align them to the same pixel coordinate system. The template image and the image to be tested are then difference analyzed to determine the defective area in the image to be tested. For example, if the difference between the pixel values ​​of a pixel in the image to be tested and the corresponding pixel in the template image is greater than the preset difference, it can be considered that the above-mentioned pixel in the image to be tested (the first pixel is used as a synonym in this example) belongs to the defective area.

[0074] In some embodiments, the first pixels in the image to be tested may also be screened. For example, if the first pixel is not adjacent to other first pixels, the first pixel is screened out. For another example, if the total number of first pixels in an area formed by connecting the first pixels adjacent to each other is less than a preset threshold, all first pixels in the area are screened out.

[0075] According to the technical solution provided in the embodiment of the present application, the defect area corresponding to the defective part of the target object can be determined based on the difference between the image to be tested and the template image, so as to achieve rapid positioning of the defect area, which is conducive to reducing the overall time consumption of the detection process.

[0076] Exemplarily, step S140 determines defect area features of a defect area for the image to be tested, and may include step S141 and step S142.

[0077] In step S141, feature extraction is performed on the image to be tested to obtain initial features of the image to be tested.

[0078] In some embodiments, the image to be tested may be input into a feature extraction network (e.g., a feature extraction layer in a convolutional neural network, a feature extraction layer in a recurrent neural network, etc.) to obtain initial features. In other embodiments, the image to be tested may be subjected to feature extraction by conventional computer vision methods (e.g., a texture feature extraction algorithm, a histogram algorithm of color features, etc.) to obtain initial features. In still other embodiments, the image to be tested may be subjected to convolution processing by the convolution kernel described above to obtain initial features.

[0079] In step S142, feature fusion is performed on the initial features corresponding to the defective area to obtain defective area features of the defective area.

[0080] It can be understood that the defect area is a part of the image to be tested. Based on the position of the defect area in the image to be tested, the initial features corresponding to the defect area can be determined in the initial features of the image to be tested.

[0081] In some embodiments, feature fusion can be performed by convolution fusion, fusion based on attention mechanism, etc. The embodiment of the present application lists an example for reference: if the image size of the image to be tested is 224*224*3 (i.e., including 224*224*3 pixels), the feature size of the corresponding initial feature is 16*16*768 (i.e., including 16*16*768 feature points). The feature size of the feature occupied by a defective area with a smaller area in the initial feature (i.e., the initial feature corresponding to the defective area in the above text) is 5*5*768. Then, each feature with a feature size of 5*5 in the third dimension (the dimension corresponding to 768 in this example) can be first fused into a feature with a feature size of 1*1, and then spliced ​​into a defective area feature of 1*1*768 in the third dimension. The feature size of the feature occupied by another defective area with a larger area in the initial feature (i.e., the initial feature corresponding to the defective area in the above text) is 8*7*768. Each feature with a feature size of 8*7 in the third dimension (the dimension corresponding to 768 in this example) can be first fused into a feature with a feature size of 1*1, and then spliced ​​into a defect area feature with a size of 1*1*768 in the third dimension. Therefore, although the initial features corresponding to the defect areas of two different areas are of different dimensions, the defect area features obtained after fusion have the same dimensional size in each dimension, which has nothing to do with the area of ​​the defect area.

[0082] According to the above technical solution provided in the embodiment of the present application, feature extraction can be performed on the image to be tested to obtain the initial features of the image to be tested. Then, feature fusion can be performed on the initial features corresponding to the defective area to obtain the defective area features. The above technical solution not only ensures that the dimensional size of the defective area features in each dimension is a value that is independent of the area of ​​the defective area through the feature fusion operation, but also enables the features of each feature point in the initial features corresponding to the defective area to be represented by the defective area features, thereby improving the representativeness of the defective area features, and further improving the accuracy of the defect type obtained based on the defective area features.

[0083] Exemplarily, step S142 performs feature fusion on the initial features corresponding to the defective area to obtain the defective area features of the defective area, which may include steps S1421a to S1426a. Figure 2 , Figure 2 A schematic diagram showing a process for generating defect region features according to an embodiment of the present application is shown.

[0084] In step S1421a, a first preset number of anchor points are determined among the feature points of the initial features corresponding to the defect area, and anchor point features of the anchor points are determined based on the initial features corresponding to the defect area.

[0085] The specific value of the first preset number can be determined according to the actual needs of the technicians, and the embodiments of the present application do not limit this. The anchor point can be a feature point with a higher representativeness among all feature points included in the initial features corresponding to the defect area. In some embodiments, the feature points in the initial features corresponding to the defect area can be sampled by uniform sampling, density-based sampling, feature-based sampling, farthest point sampling, etc. to obtain the above anchor point. Figure 2 , the pixels with darker grayscale in the initial features are the feature points corresponding to the defect area. In one example, Figure 2 The farthest point in is sampled to obtain multiple anchor points.

[0086] In some embodiments, the feature points of the initial features corresponding to the defect area can be sampled to obtain a first preset number of anchor points, and then based on the initial features corresponding to the defect area, the first features corresponding to the first preset number of anchor points in the initial features are determined. Finally, the first features corresponding to each anchor point are used as the anchor point features corresponding to each anchor point. Since the anchor point is a feature point in the initial features, the first feature corresponding to the anchor point in the initial features can be directly used as the anchor point feature of the anchor point. In some embodiments, the first feature of the anchor point can also be subjected to feature enhancement, feature pooling and other feature processing operations, and then the processed first feature is used as the anchor point feature of the anchor point.

[0087] In step S1422a, for each anchor point, based on a second preset number of neighboring feature points of the anchor point, the neighboring feature points corresponding to the anchor point are determined, wherein the neighboring feature points corresponding to the anchor point are the feature points that are the closest neighbors of the anchor point in the initial feature.

[0088] For example, for each anchor point a j , based on anchor point a j The second preset number of neighboring feature points determines the anchor point a j The corresponding neighbor point features. Further, j is a positive integer less than or equal to the first preset number.

[0089] The specific value of the second preset number can be determined according to the actual needs of the technicians, and is not limited in this embodiment of the present application.

[0090] Anchor point a j The corresponding neighbor feature point is the anchor point a in the initial feature j The nearest neighbor feature point. The nearest neighbor feature is used to represent the anchor point a jThe second preset number of neighboring feature points (refer to Figure 2 In some embodiments, the features of the second preset number of neighbor feature points can be spliced ​​by a preset feature splicing method to obtain neighbor point features. The above feature splicing algorithm is used to splice different features to obtain an overall feature, which is convenient for subsequent unified processing. For example, the above feature splicing method can be spliced ​​by functions with splicing functions such as tf.concat, tf.add, tf.keras.layers.concatenate in the TensorFlow framework. For another example, the above feature splicing method can be spliced ​​by functions with splicing functions such as torch.cat in the PyTorch framework. In other embodiments, the features of the second preset number of neighbor feature points can also be fused by feature fusion methods such as convolution fusion and fusion based on attention mechanism in related technologies to obtain neighbor point features. It should be understood that the feature sizes of neighbor point features obtained by different methods may not be equal. In this case, the technician can fill or delete the neighbor point features according to the feature size required in the subsequent steps. The features obtained by feature splicing and feature fusion in the following text can be processed similarly, so they will not be described in detail.

[0091] In step S1423a, for each anchor point, a deviation feature between the anchor point feature and the neighboring point feature corresponding to the anchor point is determined as the deviation feature corresponding to the anchor point.

[0092] For example, for each anchor point a j , determine the anchor point a j The anchor feature and anchor a j The deviation features between the corresponding neighboring point features are used as anchor point a j The corresponding deviation characteristics.

[0093] The above-mentioned deviation feature is used to represent the feature difference between the anchor point feature and the neighbor point feature. In some embodiments, the feature size of the anchor point feature may be different from the feature size of the neighbor point feature, so the anchor point feature or the neighbor point feature may be adjusted so that the two can be differentiated. For example, if the feature size of the anchor point feature is 1*768 and the feature size of the neighbor point feature is 32*768, the neighbor point feature can be split into 32 sub-features with a feature size of 1*768. Then, the anchor point feature is subtracted from the 32 sub-features to obtain 32 calculation results. Finally, the 32 calculation results are combined to obtain a deviation feature with a feature size of 32*768.

[0094] In step S1424a, for each anchor point, based on the anchor point feature of the anchor point and the deviation feature corresponding to the anchor point, the splicing feature corresponding to the anchor point is determined, wherein in at least one dimension, the dimension size of the splicing feature corresponding to the anchor point is greater than the dimension size of the defect area feature.

[0095] For example, for each anchor point a j , based on anchor point a j The anchor feature and anchor a j Corresponding deviation features, determine the anchor point a j The corresponding splicing features. Among them, in at least one dimension, the anchor point a j The dimension size of the corresponding splicing feature is larger than the dimension size of the defect area feature.

[0096] In some embodiments, the anchor point a can be spliced ​​by the feature splicing method described above. j The corresponding anchor point features, anchor point a j The corresponding deviation features are spliced ​​to obtain the anchor point a j For example, if the feature size of the anchor feature is 1*768 and the feature size of the deviation feature is 32*768, the anchor feature can be copied into 32 pieces (see Figure 2 After the 32 anchor point features are stitched with the deviation features, a stitching feature with a feature size of 32*1536 can be obtained. In these embodiments, in two dimensions of the stitching feature corresponding to the anchor point, the dimension size of the stitching feature corresponding to the anchor point is greater than the corresponding dimension size of the defect area feature.

[0097] In step S1425a, for each anchor point, a first fusion operation is performed on the splicing features corresponding to the anchor point to obtain the fusion features corresponding to the anchor point.

[0098] For each anchor point a j , for anchor point a j The corresponding splicing features are subjected to the first fusion operation to obtain the anchor point a j The corresponding fusion features. The fusion features and the defect area features have the same dimension size in each dimension.

[0099] The first fusion operation (see Figure 2 In some embodiments, the anchor point a can be fused by a feature fusion method such as convolution fusion or fusion based on an attention mechanism. j The corresponding splicing features are fused to obtain the anchor point a jCorresponding fusion features. It should be understood that the relevant parameters involved in the above-mentioned feature fusion method can be adjusted so that the fusion feature and the defect area feature have the same dimension size in each dimension. For example, the size, depth, stride, etc. of the convolution kernel used in convolution fusion are adjusted. In one example, if the feature size of the splicing feature is 32*1536, then after the first fusion operation, a fusion feature with a feature size of 1*768 can be obtained.

[0100] In step S1426a, a second fusion operation is performed on the fusion features corresponding to each of the first preset number of anchor points to obtain defect area features.

[0101] Furthermore, the second fusion operation (see Figure 2 The global feature fusion in Figure 2 The first preset number of fused features can be fused by the feature fusion method described above (the relevant parameters of the second fusion operation can be different from the relevant parameters of the first fusion operation) to obtain the defect area feature. For example, if the first preset number is 8 and the feature size of the fused feature is 1*768, then the 8 fused features can be subjected to the second fusion operation to obtain the defect area feature with a feature size of 1*768.

[0102] This application provides a specific example for reference. In this specific example, the first preset number is 8, the second preset number is 32, and the feature size of the initial feature is 16*16*768. 8 anchor feature mapping layers, 8 anchor feature fusion layers (the total number of anchor feature mapping layers and the total number of anchor feature fusion layers are related to the total number of anchors) and 1 global feature fusion layer can be constructed. 8 anchor points a can be selected from the feature points of the initial feature corresponding to the defect area. j , anchor point a j The corresponding anchor feature size is 1*768. Then determine the 32 feature points of the nearest neighbors of each anchor point in the initial feature as anchor point a j The corresponding neighbor feature points. The features of the 32 neighbor feature points are then spliced ​​into a neighbor feature with a feature size of 32*768. Then the neighbor feature is subtracted from the anchor feature to obtain a deviation feature with a feature size of 32*768. The deviation feature is then spliced ​​with the anchor feature to obtain a spliced ​​feature with a feature size of 32*1536. The spliced ​​features are then subjected to the first fusion operation to obtain a fused feature with a feature size of 1*768. Finally, the 8 fused features corresponding to the 8 anchor points are subjected to the second fusion operation to obtain the defect area feature with a feature size of 1*768.

[0103] According to the above technical solution provided by the embodiment of the present application, an anchor point can be determined in the feature points of the initial features corresponding to the defect area, and the anchor point features of the anchor point can be determined based on the initial features corresponding to the defect area. Then, based on the neighboring feature points of the anchor point, the neighboring point features corresponding to the anchor point are determined. Then, the deviation features between the anchor point features and the neighboring point features are determined. Then, based on the anchor point features and the deviation features, the fused features corresponding to each anchor point are determined. Finally, a second fusion operation is performed on the fused features corresponding to each anchor point to obtain the defect area features. The deviation features obtained in the above technical solution can effectively characterize the relative change relationship between the anchor point features, and can also reduce the influence of background information or image noise commonly present in the image to be tested on the anchor point features. Therefore, the fused features obtained based on the deviation features can effectively characterize the overall features of the local area where the anchor point is located. Furthermore, the defect area features obtained based on the fused features in the above technical solution can effectively characterize the feature information of the entire defect area, which is highly representative and is conducive to improving the accuracy of the defect type obtained based on the defect area features. In addition, the number of anchor points is preset, so even if the total number of feature points occupied by the defect area in the initial feature is different, it can be converted into a defect area feature of a preset size, which is conducive to reducing the difficulty of subsequent classification processing based on the defect area feature.

[0104] Exemplarily, step S1421a determines a first preset number of anchor points among feature points of the initial features corresponding to the defect area, and determines anchor point features of the anchor points based on the initial features corresponding to the defect area, which may include steps S1421a1 to S1421a3.

[0105] In step S1421a1, the farthest point sampling is performed on the feature points of the initial features corresponding to the defect area to obtain a first preset number of anchor points.

[0106] In some embodiments, the sampling method of farthest point sampling may include the first step to the third step. Step 1: first randomly select a starting point in the initial feature corresponding to the defect area, and add the starting point to the set. Step 2: For the feature point A in the initial feature corresponding to the defect area that has not yet been added to the set, determine the distance between each feature point A and all the feature points in the set, and add the feature point A with the farthest distance to the set. Step 3: Repeat the second step until the total number of feature points in the set reaches a first preset number. The feature points in the set obtained by the above-mentioned farthest point sampling can be used as anchor points.

[0107] Furthermore, the position information corresponding to the anchor point in the initial feature can be determined. The position information corresponding to the anchor point in the initial feature can be expressed as the coordinates corresponding to the anchor point in the initial feature. For example, the position information of the anchor point in the 5th row and the 24th column in the initial feature can be (5, 24).

[0108] In step S1421a2, based on the initial features corresponding to the defect area, first features corresponding to a first preset number of anchor points are determined.

[0109] In some embodiments, since the anchor point is a feature point in the initial feature, the feature corresponding to the anchor point in the initial feature can be directly used as the first feature corresponding to the anchor point. In other embodiments, the first feature corresponding to the anchor point can also be determined based on the features of all feature points in a region where the anchor point is located in the initial feature.

[0110] In step S1421a3, for each anchor point, the first feature corresponding to the anchor point and the position information corresponding to the anchor point in the initial feature are fused to obtain the anchor point feature corresponding to the anchor point.

[0111] For example, for each anchor point a j , the anchor point a j The corresponding first feature and anchor point a j The corresponding position information is fused to obtain the anchor point a j Corresponding anchor point features (see Figure 2 ).

[0112] The above fusion can be expressed as the feature splicing method or the feature fusion method mentioned above.

[0113] Combined with the specific example above, anchor point a j The corresponding first feature size is 1*768, anchor point a j The size of the corresponding position information in the initial feature can be 1*2 (ie, the horizontal coordinate and the vertical coordinate in the initial feature occupy 1 dimension size respectively in the dimension corresponding to '2'). The first feature is fused with the position information to obtain an anchor feature of size 1*770.

[0114] In an embodiment where the feature corresponding to the anchor point includes the position information corresponding to the anchor point, based on the anchor point a j The neighbor point features obtained by the corresponding neighbor feature points may also include the location information of each neighbor feature point. In the above specific example, the feature size of the neighbor point feature may be 32*770. The feature size of the deviation feature obtained based on the neighbor point feature is 32*770. The feature size of the splicing feature obtained based on the deviation feature is 32*1540.

[0115] According to the above technical solution provided in the embodiment of the present application, the feature points of the initial features corresponding to the defect area can be sampled at the farthest point to obtain the anchor point. Then, based on the initial features corresponding to the defect area, the first features corresponding to the anchor points are determined. Finally, the position information corresponding to the first feature and the anchor point in the initial features are fused to obtain the anchor point feature corresponding to the anchor point. In the above technical solution, the farthest point sampling is used so that the anchor point finally obtained can better represent the defect area. And the anchor point feature of the anchor point includes the position information of the anchor point, so it can characterize the position of the anchor point in the initial feature, which improves the representativeness of the anchor point feature. In addition, the representativeness of the defect area feature obtained based on the anchor point feature can be improved, which is conducive to improving the accuracy of the defect type obtained based on the defect area feature.

[0116] Exemplarily, step S1424a determines the splicing feature corresponding to the anchor point based on the anchor feature of the anchor point and the deviation feature corresponding to the anchor point, and may include: splicing the anchor feature of the anchor point with the deviation feature corresponding to the anchor point to obtain the splicing feature corresponding to the anchor point.

[0117] For example, anchor point a j The anchor feature and anchor a j The corresponding deviation features are spliced ​​to obtain the anchor point a j The corresponding stitching features.

[0118] In some embodiments, the anchor point a can be connected to j The anchor feature and anchor a j The corresponding deviation features are spliced ​​to obtain the anchor point a j The corresponding stitching features.

[0119] According to the above technical solution, the anchor point a can be j The anchor feature and anchor a j The corresponding deviation features are directly spliced ​​to obtain the anchor point a j The corresponding splicing features are obtained in a simpler way, which is conducive to improving the overall detection efficiency of defect types.

[0120] Exemplarily, step S1424a may also include: mapping the anchor point feature of the anchor point through the feature mapping layer in the defect classification model to obtain the mapping feature corresponding to the anchor point. The anchor point feature of the anchor point and the mapping feature corresponding to the anchor point are located in different feature spaces. Then, the mapping feature corresponding to the anchor point is spliced ​​with the deviation feature corresponding to the anchor point to obtain the spliced ​​feature corresponding to the anchor point.

[0121] For example, first, move the anchor point a jThe anchor point features are mapped into feature space through the feature mapping layer in the defect classification model to obtain the anchor point a j The corresponding mapping feature a j '. Among them, the anchor point a j The anchor feature and anchor a j The corresponding mapping feature a j 'The two are in different feature spaces. Then, the anchor point a j The corresponding mapping feature a j 'With anchor point a j The corresponding deviation features are spliced ​​to obtain the anchor point a j Corresponding splicing features (see Figure 2 feature map).

[0122] The defect classification model can be used to classify the defect area features to obtain the defect type of the target object. In some embodiments, the feature mapping layer can be a module that can further process the features. For example, the feature mapping layer can be a module that performs operations such as feature conversion and feature enhancement. It should be understood that during the training process of the defect classification model, the classification head in the defect classification model is obtained by training based on the features output by the feature mapping layer, so the feature space where the mapping features are located is the same as the feature space that the classification head focuses on when performing the classification operation.

[0123] Through the spatial mapping operation, the feature space where the mapping feature corresponding to the anchor point and the anchor point feature of the anchor point obtained by the feature mapping layer in the defect classification model are located is different. In subsequent processing, the mapping feature corresponding to the anchor point and the deviation feature corresponding to the anchor point are spliced ​​to obtain the spliced ​​feature of the anchor point.

[0124] According to the above technical solution, the anchor point features of the anchor point can be mapped into feature space through the feature mapping layer in the defect classification model to obtain the mapping features corresponding to the anchor point. Then the mapping features are spliced ​​with the deviation features to obtain the splicing features corresponding to the anchor point. The mapping features obtained in the above solution are more representative, thereby improving the accuracy of the defect type obtained based on the mapping features.

[0125] Exemplarily, step S142 performs feature fusion on initial features corresponding to the defective area to obtain defective area features of the defective area, which may include step S1421b and step S1422b.

[0126] In step S1421b, the initial features corresponding to the defective area are interpolated and sampled according to the positions of the pixels included in the defective area to obtain a first number of defective features, wherein the first number is the total number of feature points corresponding to the defective area in the initial features.

[0127] In some embodiments, the correspondence between the pixels in the defect area and the feature points in the initial features may be determined first, and then the defect features in the initial features may be sampled. For example, a corresponding mask in the image to be tested may be generated, and then the defect features in the initial features may be determined based on the mask.

[0128] The embodiment of the present application provides an example for reference. The image size of the image to be tested may be 224*224*3, and the size of the mask may be 224*224 (i.e., including 224*224 mask elements). The mask elements in the mask correspond one-to-one to the pixels in the image to be tested. The value of the mask element is used to indicate whether the corresponding pixel is a pixel in the defective area. For example, the value of the mask element corresponding to the pixel in the defective area in the mask may be 1, and the value of the mask element corresponding to the pixel in the non-defective area in the mask may be 0.

[0129] In one example, the mask can be obtained by comparing the image to be tested with the template image described above. For example, by comparing whether the difference between the pixel values ​​of the corresponding pixels in the image to be tested and the template image is greater than a difference threshold, the value of each mask element in the mask is determined.

[0130] The coordinate values ​​of the mask elements with a value of 1 in the mask can then be normalized to obtain normalized coordinate values. For example, if the intervals of the horizontal and vertical coordinate values ​​of the mask elements are both 1 to 224, then the mask element A with a coordinate value of (112, 224) and an identifier of 1 in the mask has a normalized coordinate value of (112 / 224, 224 / 224), that is, (0.5, 1). The normalized coordinate values ​​are then converted into corresponding coordinate values ​​in the initial features, and based on the corresponding coordinate values ​​in the initial features, the defect features corresponding to the mask elements with a value of 1 in the initial features are determined. For example, in the above example, the feature size of the initial features of the image to be tested is 16*16*768 (i.e., includes 16*16*768 feature points). It should be understood that the initial feature with a feature size of 16*16*768 can be understood as a 16*16 feature point sequence, and each feature point sequence also includes 768 feature points. The first dimension of 16 in 16*16 can be used as the horizontal axis, and the second dimension of 16 can be used as the vertical axis to represent the position of each feature point sequence. The coordinate value corresponding to the mask element A in the initial feature is (0.5*16,1*16), that is, (8,16).

[0131] In combination with actual scenarios, since the coordinate value range of the mask (1 to 224 in the above example) usually includes the coordinate value range of the initial feature (1 to 16 in the above example), the coordinate value corresponding to the normalized coordinate value in the initial feature may not be an integer.

[0132] In some embodiments, the closest integer coordinate value corresponding to the normalized coordinate value in the initial feature can be used as the coordinate value of the defect feature. For example, the mask element B with a coordinate value of (120, 210) in the mask and an identifier of 1 has a normalized coordinate value of (120 / 224, 210 / 224), that is, (0.5357, 0.9375). The normalized coordinate value is then converted to the corresponding coordinate value in the initial feature to obtain (0.5357*16, 0.9375*16), that is, (8.5712, 16). Since 8.5712 is closer to 9, the feature points (1*768 feature points in the above example) included in the feature point sequence with a coordinate value of (9, 16) in the initial feature can be used as the defect feature corresponding to mask element B.

[0133] In other embodiments, the defect area features in the initial features can be determined by interpolation sampling. For example, the coordinate value corresponding to the mask element B in the above example in the initial features is (8.5712, 16), then the features of the feature points in the two feature point sequences with coordinate values ​​of (8, 16) and (9, 16) in the initial features can be interpolated to obtain the interpolation result. The interpolation result is then used as the defect feature.

[0134] See also Figure 3 , Figure 3 A schematic diagram showing a defect region and its corresponding defect feature in the initial feature according to an embodiment of the present application is shown. Figure 3 , the defect area of ​​image A (in this example, the image to be tested or a part of the image to be tested) with an image size of 512*512*3 is a linear defect with the darkest grayscale (such as a crack or a linear foreign body), and the multiple defect features A corresponding to the defect area are also distributed linearly in the initial features with a feature size of 16*16*768. The defect area of ​​image B (in this example, the image to be tested or a part of the image to be tested) with an image size of 512*512*3 is a rectangular defect with the darkest grayscale (such as a rectangular foreign body), and the multiple defect features B corresponding to the defect area are also distributed rectangularly in the initial features with a feature size of 16*16*768. Figure 3 The defect characteristics in the example can be obtained by the method in any of the above embodiments.

[0135] In step S1422b, the first number of defect features are averaged to obtain defect area features of the defect area.

[0136] The above average calculation may be a simple average calculation or a weighted average calculation. In some examples, if the weighted average calculation is used, the weight of the feature point may be determined based on the relative position between the feature points corresponding to the defect area in the initial feature. For example, a higher weight may be set for the feature point corresponding to the defect area in the initial feature that is closer to the center of the defect area.

[0137] According to the above technical solution provided in the embodiment of the present application, the initial features corresponding to the defective area can be interpolated and sampled according to the positions of the pixels included in the defective area to obtain a first number of defect features. The first number of defect features are then averaged to obtain defective area features. The defective area features obtained by the above technical solution are obtained based on defect features that have a corresponding relationship with the defective area, so the representativeness of the defective area features can be improved, which is beneficial to improving the accuracy of the defect type obtained based on the defective area features. In addition, the calculation method of the above technical solution has a low demand for computing power, which is beneficial to improving the overall detection efficiency of defect types.

[0138] Exemplarily, step S150 determines the defect type of the target object based on the defect region feature, which may include: inputting the defect region feature into a classification head in a defect classification model for classification to obtain the defect type of the target object.

[0139] In some embodiments, the training process of the defect classification model may include: obtaining a training data set, the training data set including multiple images and the defect type corresponding to each image. The defect classification model is iteratively trained by the training data set, and the model parameters of the defect classification model are adjusted by the loss function, so that the defect classification model can learn a general method for determining the defect type for an image, that is, a trained defect classification model is obtained.

[0140] In some embodiments, the model parameters of the feature extraction layer of the defect classification model may be frozen during the training process, and only the model parameters of the classification head of the defect classification model are updated to reduce computational overhead and thereby improve the training efficiency of the model.

[0141] The above technical solution uses the classification head of the defect classification model to classify the defect area features. The defect type of the target object obtained is more accurate and has strong universality for the defect type.

[0142] Exemplarily, step S150 determines the defect type of the target object based on the defect area characteristics, and may include steps S151 and S152.

[0143] In step S151, the defect region feature is classified and compared with a plurality of preset features in a preset feature library to obtain the similarity between the defect region feature and each preset feature.

[0144] The preset features in the preset feature library can be obtained by extracting features from typical defect areas of various types. Each preset feature can represent the characteristics of the defect area to which it corresponds. Multiple preset features can be compared with the features of the defect area features of the image to be tested of the target object. The more similar the defect area feature is to a preset feature, the more likely the image to be tested belongs to the type of defect area corresponding to the preset feature.

[0145] The dimension size of the preset feature and the defect area feature in each dimension can be the same, so as to facilitate classification and comparison of the defect area feature and the preset feature.

[0146] In some embodiments, the above classification comparison can be achieved by calculating Euclidean distance, cosine similarity, etc.

[0147] In step S152, the defect type of the target object is determined according to the similarity between the defect region feature and each preset feature and the defect type corresponding to each preset feature.

[0148] The defect type corresponding to a target preset feature having the highest similarity to the defect region feature may be used as the defect type of the target object.

[0149] In some embodiments, the target defect types corresponding to the first few target preset features with the highest similarity to the defect area feature can also be integrated, and the cumulative similarity value corresponding to each target defect type is calculated, and the target defect type with the highest cumulative similarity value is used as the defect type of the target object. For example, the first few target preset features with the highest similarity are target preset features 1 to 5, target preset features 1 to 5 correspond to defect types 1 to 5, and target preset features 1 to 5 correspond to similarities 1 to 5. If defect types 1 and 2 belong to a larger type target defect type A, and defect types 3 to 5 belong to a larger type target defect type B, then defect types 1 to 5 are integrated into target defect types A (i.e., defect types 1 and 2), target defect types B (i.e., defect types 3 to 5), and target defect type C (i.e., defect type 4). Then calculate the cumulative similarity value corresponding to target defect type A (i.e., the sum of similarities 1 and 2), and the cumulative similarity value corresponding to target defect type B (i.e., the sum of similarities 3 to 5). Compare the cumulative similarity values ​​corresponding to target defect type A, target defect type B, and target defect type C. In the above example, the cumulative similarity value of target defect type A is: 1+2=3, the cumulative similarity value of target defect type B is: 3+4+5=12, and the cumulative similarity value of target defect type C is: 4. The cumulative similarity value corresponding to target defect type B is the highest, so target defect type B is used as the defect type of the target object.

[0150] The above technical solution can reduce the classification complexity of defect area features by presetting the feature library, and can improve the overall detection speed of defect types.

[0151] Exemplarily, the detection method may further include: determining the defect level of the target object based on the defect type of the target object. If the defect level meets the preset conditions, the target object is reclassified to obtain a new defect type corresponding to the target object. The preset conditions are used to measure whether the defect level is allowed by the quality inspection standard, which may depend on the actual situation. It can be understood that in some scenarios, target objects with major defects need to be strictly detected, while target objects with minor defects can have a certain degree of tolerance. Therefore, the defect type of the target object is determined by using the above steps S110 to S150, and then its defect level is determined. For example, for the target object being a wafer, according to an embodiment of the present application, the defect type of the wafer can be roughly determined by using the above steps S110 to S150, and then the defect level can be determined, and then only the wafers with a defect level of major defects can be re-judged. This will significantly improve the efficiency of wafer defect detection.

[0152] According to the defect type detection method provided in the embodiment of the present application, improvements can be made in the control of the entire detection process, that is, the defective part with defects and the corresponding defect type are first located through the first image with low image quality, and the defective area that needs secondary re-inspection is screened out from these defective parts, and then subsequent operations are performed, which greatly reduces the subsequent processing volume and is more focused and more accurate. In the image to be tested of the target object, the defect area corresponding to the defective part of the target object is determined. Then the defect area characteristics of the defective area are determined. Finally, based on the defect area characteristics, the defect type of the target object is determined. In this scheme, by determining the defect area characteristics, defects of different areas in the image to be tested can be represented by features of a uniform scale, which is convenient for reducing the classification difficulty of subsequent classification processing and also ensures the classification accuracy of defects, which is conducive to improving the detection speed of subsequent defect types.

[0153] An embodiment of the present application also provides a defect type detection device. Figure 4 FIG. 1 is a schematic block diagram of a defect type detection device according to an embodiment of the present application. Figure 4 As shown, the detection device 400 may include a defective part screening module 410, a to-be-detected image acquisition module 420, a defective region determination module 430, a feature determination module 440, and a defect type determination module 450. The defective part screening module 410 is used to determine a plurality of first defective regions and corresponding first defect types in a first image of a target object acquired by a first image acquisition device and to screen out defective parts to be detected from the plurality of first defective regions.

[0154] The image acquisition module 420 is used to acquire the image to be tested including the defective part of the target object by using a second image acquisition device, wherein the image quality of the first image and the image to be tested are different;

[0155] The defect area determination module 430 is used to determine the defect area corresponding to the defective part of the target object in the image to be tested. The feature determination module 440 is used to determine the defect area features of the defect area for the image to be tested, wherein the dimension size of the defect area features in each dimension is a preset value that is independent of the area of ​​the defect area. The defect type determination module 450 is used to determine the defect type of the target object based on the defect area features.

[0156] Exemplarily, the feature determination module 440 may include an initial feature extraction module and a defect region feature generation module.

[0157] The initial feature extraction module is used to extract features from the image to be tested to obtain the initial features of the image to be tested. The defect area feature generation module is used to perform feature fusion on the initial features corresponding to the defect area to obtain the defect area features of the defect area.

[0158] Exemplarily, the defect area feature generation module may include: an anchor feature determination module, a neighbor feature extraction module, a deviation feature determination module, a splicing feature determination module, a first fusion module and a second fusion module.

[0159] The anchor feature determination module is used to determine a first preset number of anchor points in the feature points of the initial features corresponding to the defect area, and determine the anchor features of the anchor points based on the initial features corresponding to the defect area. The neighbor feature extraction module is used to determine the neighbor feature corresponding to each anchor point based on the second preset number of neighbor feature points of the anchor point. The neighbor feature point corresponding to the anchor point is the feature point of the nearest neighbor of the anchor point in the initial features. The deviation feature determination module is used to determine the deviation feature between the anchor feature of the anchor point and the neighbor feature corresponding to the anchor point for each anchor point, as the deviation feature corresponding to the anchor point. The splicing feature determination module is used to determine the splicing feature corresponding to each anchor point based on the anchor feature of the anchor point and the deviation feature corresponding to the anchor point, wherein in at least one dimension, the dimension size of the splicing feature corresponding to the anchor point is greater than the dimension size of the defect area feature. The first fusion module is used to perform a first fusion operation on the splicing feature corresponding to the anchor point to obtain the fused feature corresponding to the anchor point. The fused feature has the same dimension size as the defect area feature in each dimension. The second fusion module is used to perform a second fusion operation on the fusion features corresponding to each of the first preset number of anchor points to obtain defect area features.

[0160] Exemplarily, the splicing feature determination module may include: a first splicing module or a second splicing module.

[0161] The first splicing module is used to splice the anchor feature of the anchor point with the deviation feature corresponding to the anchor point to obtain the splicing feature corresponding to the anchor point. The second splicing module is used to map the anchor feature of the anchor point through the feature mapping layer in the defect classification model to obtain the mapping feature corresponding to the anchor point, wherein the anchor feature of the anchor point and the mapping feature corresponding to the anchor point are located in different feature spaces. The second splicing module is also used to splice the mapping feature corresponding to the anchor point with the deviation feature corresponding to the anchor point to obtain the splicing feature corresponding to the anchor point. The defect classification model is used to determine the defect type of the target object based on the defect area feature.

[0162] Exemplarily, the anchor feature determination module may include: a first sampling module, a first feature determination module and an anchor feature generation module.

[0163] The first sampling module is used to perform farthest point sampling on the feature points of the initial features corresponding to the defect area to obtain a first preset number of anchor points. The first feature determination module is used to determine the first features corresponding to the first preset number of anchor points based on the initial features corresponding to the defect area. The anchor feature generation module is used to fuse the first feature corresponding to each anchor point with the position information corresponding to the anchor point in the initial features to obtain the anchor feature corresponding to the anchor point.

[0164] Exemplarily, the defect region feature generation module may include: a second sampling module and an average calculation module.

[0165] The second sampling module is used to perform interpolation sampling on the initial features corresponding to the defective area according to the positions of the pixels included in the defective area to obtain a first number of defective features, wherein the first number is the total number of feature points corresponding to the defective area in the initial features, and the defective features and the defective area features have the same dimensional size in each dimension. The average calculation module is used to perform average calculation on the first number of defective features to obtain the defective area features of the defective area.

[0166] Exemplarily, the defective region determination module 430 may include a difference determination module.

[0167] The difference determination module is used to determine a defect area corresponding to a defective part of the target object based on a difference between the image to be tested and a template image of the target object.

[0168] Exemplarily, the defect type determination module 450 may include: a first classification module or a second classification module.

[0169] The first classification module is used to input the defect area feature into the classification head in the defect classification model for classification to obtain the defect type of the target object. The second classification module is used to classify and compare the defect area feature with multiple preset features in the preset feature library to obtain the similarity between the defect area feature and each preset feature. It is also used to determine the defect type of the target object based on the similarity between the defect area feature and each preset feature and the defect type corresponding to each preset feature.

[0170] Exemplarily, the above detection device further includes: a defect level determination module and a defect type regeneration module.

[0171] The defect level determination module is used to determine the defect level of the target object based on the defect type of the target object. The defect type regeneration module is used to reclassify the target object if the defect level meets the preset conditions to obtain a new defect type corresponding to the target object.

[0172] According to the defect type detection device provided in the embodiment of the present application, improvements can be made in the control of the entire detection process, that is, the defective part with defects and the corresponding defect type are first located through the first image with low image quality, and the defective area that needs secondary re-inspection is screened out from these defective parts, and then subsequent operations are performed, which greatly reduces the subsequent processing volume and is more focused and more accurate. In the image to be tested of the target object, the defect area corresponding to the defective part of the target object is determined. Then the defect area characteristics of the defective area are determined. Finally, based on the defect area characteristics, the defect type of the target object is determined. In this scheme, by determining the defect area characteristics, defects of different areas in the image to be tested can be represented by features of uniform scale, which is convenient for reducing the classification difficulty of subsequent classification processing and also ensures the classification accuracy of defects, which is conducive to improving the detection speed of subsequent defect types.

[0173] According to another aspect of the present application, an electronic device is also provided. Figure 5 Schematic block diagram of an electronic device according to an embodiment of the present application is shown. Figure 5 As shown, the electronic device 300 includes a processor 310 and a memory 320. The memory 320 stores a computer program. When the computer program instructions are executed by the processor 310, they are used to execute the above-mentioned defect classification method.

[0174] In addition, according to another aspect of the present application, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned defect type detection method of the embodiment of the present application, and is used to implement the corresponding modules in the above-mentioned defect type detection device according to the embodiment of the present application or the corresponding modules in the above-mentioned electronic device. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0175] According to another aspect of the present application, a computer program product is provided, including computer program instructions. When the computer program instructions are executed by a computer or a processor, the computer or the processor executes corresponding steps of the above-mentioned defect type detection method.

[0176] A person skilled in the art may understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the above-mentioned description of the detection method of the defect type, and for the sake of brevity, it will not be described here in detail.

[0177] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0178] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0180] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0181] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0182] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0183] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0184] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the defect type detection device according to the embodiment of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0185] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0186] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A defect type detection method, characterized in that: The method comprises: Determine a plurality of first defect regions and corresponding first defect types in a first image of a target object acquired by a first image acquisition device, and select defective parts to be detected from the plurality of first defect regions; Using a second image acquisition device to acquire an image to be tested that includes the defective portion of the target object, wherein the image quality of the first image and the image to be tested are different; For the image to be tested, determining a defect area corresponding to the defective part; Determine the defect region feature of the defect region for the image to be tested, wherein the dimension size of the defect region feature in each dimension is a preset value that is independent of the area of ​​the defect region; Based on the defect area characteristics, the defect type of the target object in the defect portion is determined.

2. The method according to claim 1, characterized in that The step of determining the defect region feature of the defect region for the image to be tested includes: Performing feature extraction on the image to be tested to obtain initial features of the image to be tested; The initial features corresponding to the defect area are subjected to feature fusion to obtain defect area features of the defect area.

3. The method according to claim 2, characterized in that The step of fusing the initial features corresponding to the defective region to obtain defective region features of the defective region includes: Determine a first preset number of anchor points among the feature points of the initial features corresponding to the defect area, and determine anchor point features of the anchor points based on the initial features corresponding to the defect area; For each anchor point, Based on a second preset number of neighboring feature points of the anchor point, determine the neighboring point features corresponding to the anchor point, wherein the neighboring feature points corresponding to the anchor point are the feature points of the nearest neighbors of the anchor point in the initial features; Determine a deviation feature between the anchor point feature of the anchor point and the neighboring point feature corresponding to the anchor point as the deviation feature corresponding to the anchor point; Determine a splicing feature corresponding to the anchor point based on the anchor point feature of the anchor point and the deviation feature corresponding to the anchor point, wherein in at least one dimension, the dimension size of the splicing feature corresponding to the anchor point is greater than the dimension size of the defect area feature; Performing a first fusion operation on the splicing feature corresponding to the anchor point to obtain a fusion feature corresponding to the anchor point, wherein the fusion feature and the defect area feature have the same dimension size in each dimension; A second fusion operation is performed on the fusion features corresponding to each of the first preset number of anchor points to obtain the defect area features.

4. The method according to claim 3, characterized in that The determining of the splicing feature corresponding to the anchor point based on the anchor point feature of the anchor point and the deviation feature corresponding to the anchor point includes any one of the following: splicing the anchor point feature of the anchor point with the deviation feature corresponding to the anchor point to obtain the splicing feature corresponding to the anchor point; or The anchor point feature of the anchor point is mapped into a feature space through a feature mapping layer in a defect classification model to obtain a mapping feature corresponding to the anchor point, wherein the anchor point feature of the anchor point and the mapping feature corresponding to the anchor point are located in different feature spaces; the mapping feature corresponding to the anchor point and the deviation feature corresponding to the anchor point are spliced ​​to obtain a spliced ​​feature corresponding to the anchor point, wherein the defect classification model is used to determine the defect type of the target object based on the defect area feature.

5. The method according to claim 3, characterized in that The step of determining a first preset number of anchor points among the feature points of the initial features corresponding to the defect area, and determining the anchor point features of the anchor points based on the initial features corresponding to the defect area, comprises: Performing farthest point sampling on feature points of the initial features corresponding to the defect area to obtain a first preset number of anchor points; Based on the initial features corresponding to the defect area, determining first features respectively corresponding to the first preset number of anchor points; For each anchor point, the first feature corresponding to the anchor point and the position information corresponding to the anchor point in the initial feature are fused to obtain the anchor point feature corresponding to the anchor point.

6. The method according to claim 2, characterized in that The step of fusing the initial features corresponding to the defective region to obtain defective region features of the defective region includes: According to the positions of the pixels included in the defect area, interpolation sampling is performed on the initial features corresponding to the defect area to obtain a first number of defect features, wherein the first number is the total number of feature points corresponding to the defect area in the initial features, and the defect features have the same dimension size as the defect area features in each dimension; An average calculation is performed on the first number of defect features to obtain a defect region feature of the defect region.

7. The method according to any one of claims 1 to 6, characterized in that Determining a plurality of first defect regions in a first image of a target object acquired by a first image acquisition device includes: determining the plurality of first defect regions of the target object based on a difference between the first image and a template image of the target object.

8. The method according to any one of claims 1 to 6, characterized in that The determining the defect type of the target object based on the defect area feature includes any one of the following: Inputting the defect area features into a classification head in a defect classification model for classification to obtain the defect type of the target object; or The defect area feature is classified and compared with multiple preset features in a preset feature library to obtain the similarity between the defect area feature and each preset feature; the defect type of the target object is determined according to the similarity between the defect area feature and each preset feature and the defect type corresponding to each preset feature.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determining a defect level of the target object based on a defect type of the target object; If the defect level meets the preset conditions, the target object is reclassified to obtain a new defect type corresponding to the target object.

10. A defect type detection device, characterized in that: The device comprises: A defective part screening module, used to determine a plurality of first defective areas and corresponding first defect types in a first image of a target object acquired by a first image acquisition device and to screen out defective parts to be detected from the plurality of first defective areas; a test image acquisition module, used for acquiring a test image including the defective portion of the target object by using a second image acquisition device, wherein the image quality of the first image and the test image are different; A defect area determination module, used to determine a defect area corresponding to a defective part of the target object in the image to be tested of the target object; A feature determination module, used for determining the defect region feature of the defect region for the image to be tested, wherein the dimension size of the defect region feature in each dimension is a preset value that is independent of the area of ​​the defect region; The defect type determination module is used to determine the defect type of the target object based on the defect area characteristics.

11. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the defect type detection method according to any one of claims 1 to 9 when the processor runs the computer program instructions.

12. A storage medium storing a computer program / instruction, characterized in that: The computer program / instructions are used to execute the defect type detection method according to any one of claims 1 to 9 when running.

Images