Defect type detection method and device, electronic device and storage medium

Abstract

The present invention provides a defect type detection method and device, electronic device and storage medium, the detection method comprising: determining a defect area in a test image of a test object; extracting image features corresponding to the defect area in the test image; based on the image features corresponding to the defect area in the test image, using a trained aggregation projection module, generating a first aggregation feature; based on the first aggregation feature, determining the actual defect type of the test object. The trained aggregation projection module in the present invention is trained based on the first global feature and the first defect feature of the training image, so the first aggregation feature output by the trained aggregation projection module also pays attention to the global features of the test image to a certain extent, which is conducive to improving the generation accuracy of the actual defect type.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to a defect type detection method, a defect type detection device, an electronic device, a storage medium and a computer program product. Background Art

[0002] In the industrial field, with the continuous development of production technology, people have put forward higher requirements for the production quality and efficiency of industrial products. With the development of image processing technology, more and more product inspections are based on images. If the inspection accuracy is low, the practicality of the image-based inspection method will also be reduced.

[0003] For example, in the process of wafer inspection, the related technology usually uses Automated Optical Inspection (AOI) equipment to perform appearance inspection on the wafer to determine the type of defects on the wafer. However, the different defects on the wafer are highly random, complex and changeable. Therefore, how to accurately detect the defect type is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0004] The present invention is proposed in view of the above problems. The present invention provides a defect type detection method, a defect type detection device, an electronic device, a storage medium and a computer program product.

[0005] According to one aspect of the present invention, a method for detecting defect types is provided, the method comprising:

[0006] Determining a defective area in the image to be tested of the object to be tested;

[0007] Extracting image features corresponding to defective areas in the image to be tested;

[0008] Based on the image features corresponding to the defective area in the image to be tested, a first aggregate feature is generated by using a trained aggregate projection module, wherein the trained aggregate projection module is trained based on a first global feature and a first defect feature of the training image of the object to be tested, the first global feature is the image feature of the training image as a whole, and the first defect feature is the image feature corresponding to the defective area in the training image;

[0009] Based on the first aggregated feature, an actual defect type of the object to be tested is determined.

[0010] Exemplarily, extracting image features corresponding to defective areas in the image to be tested includes:

[0011] Dividing the image to be tested into a plurality of first image blocks;

[0012] Extracting a first image block feature of a first image block located in a defect area in the image to be tested;

[0013] All first image block features are fused to obtain fused features as image features corresponding to the defect area.

[0014] Exemplarily, determining the actual defect type of the object to be tested based on the first aggregated feature includes:

[0015] Matching the first aggregate feature with a preset feature in a feature library, and determining a defect type corresponding to the first aggregate feature based on defect types corresponding to each of the matched multiple preset features, wherein the feature library stores multiple preset features and defect types corresponding to each preset feature;

[0016] Based on the defect type corresponding to the first aggregated feature, an actual defect type of the object to be tested is determined.

[0017] Exemplarily, the above detection method further includes:

[0018] For each initial image,

[0019] Determining a defect area in the initial image, wherein the initial image corresponds to a defect type;

[0020] Dividing the initial image into a plurality of second image blocks, and generating second image block features of the second image blocks located in the defective area of ​​the initial image;

[0021] Performing feature fusion on all second image block features of the initial image to obtain second fused features of the initial image;

[0022] Inputting the second fused features of the initial image into an untrained aggregation projection module to obtain a second aggregation feature of the initial image;

[0023] For each defect type, the second aggregation features of all initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature.

[0024] Exemplarily, the above detection method further includes:

[0025] Determining defect areas in training images;

[0026] Dividing the training image into a plurality of third image blocks, and generating third image block features of the third image blocks located in the defect area of ​​the training image as first defect features;

[0027] Performing feature fusion on all third image blocks to obtain a third fusion feature;

[0028] Inputting the third fusion feature into the untrained aggregation projection module to obtain a third aggregation feature;

[0029] Matching the third aggregate feature with the preset features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the matched multiple preset features;

[0030] For each preset feature that matches the third aggregate feature, the preset feature in the feature library is updated based on the third aggregate feature.

[0031] Exemplarily, matching the third aggregate feature with a preset feature in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the matched multiple preset features, includes at least one of the following:

[0032] Matching the third aggregate feature with all preset features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched preset feature and the third aggregate feature;

[0033] Matching the third aggregate feature with all first features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched first feature and the third aggregate feature, wherein the first feature is a preset feature whose defect type is the same as the defect type corresponding to the training image;

[0034] The third aggregate feature is matched with all the second features in the feature library, and based on the difference between each matched second feature and the third aggregate feature, the module parameters of the untrained aggregate projection module are adjusted, wherein the second feature is a preset feature whose defect type is different from the defect type corresponding to the training image.

[0035] Exemplarily, the above detection method further includes:

[0036] Generate a second global feature of the image to be tested, wherein the second global feature is an image feature of the entire image to be tested;

[0037] Inputting the second global feature into the trained aggregation projection module to obtain a fourth aggregation feature;

[0038] Based on the first aggregated feature, determining the actual defect type of the object to be tested includes:

[0039] Based on the first aggregate feature and the fourth aggregate feature, an actual defect type of the object to be tested is determined.

[0040] Exemplarily, determining the actual defect type of the object to be tested based on the first aggregate feature and the fourth aggregate feature includes:

[0041] Matching the fourth aggregate feature with preset features in the feature library, and determining the defect type corresponding to the fourth aggregate feature based on the defect types corresponding to each of the matched multiple preset features;

[0042] Based on the defect type corresponding to the first aggregate feature and the defect type corresponding to the fourth aggregate feature, an actual defect type of the object to be tested is determined.

[0043] Exemplarily, the above detection method further includes:

[0044] For each initial image,

[0045] Generate a third global feature of the initial image, wherein the initial image corresponds to a defect type and the third global feature is an image feature of the entire initial image;

[0046] Inputting the third global feature of the initial image into an untrained aggregation projection module to obtain a fifth aggregation feature of the initial image;

[0047] For each defect type, the fifth aggregation features of all initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature.

[0048] Exemplarily, the above detection method further includes:

[0049] Generate a fourth global feature of the training image, wherein the fourth global feature is an image feature of the entire training image;

[0050] Inputting the fourth global feature into an untrained aggregation projection module to obtain a sixth aggregation feature;

[0051] Matching the sixth aggregate feature with a preset feature in a feature library, and adjusting module parameters of an untrained aggregate projection module based on the matched multiple preset features;

[0052] For each preset feature that matches the sixth aggregate feature, the preset feature in the feature library is updated based on the sixth aggregate feature.

[0053] Exemplarily, matching the sixth aggregate feature with a preset feature in a feature library, and adjusting a module parameter of an untrained aggregate projection module based on a plurality of matched preset features, includes at least one of the following:

[0054] Matching the sixth aggregate feature with all preset features in the feature library, and adjusting module parameters of the untrained aggregate projection module based on the difference between each matched preset feature and the sixth aggregate feature;

[0055] Matching the sixth aggregate feature with all third features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched third feature and the sixth aggregate feature, wherein the third feature is a preset feature whose defect type is the same as the defect type corresponding to the training image;

[0056] The sixth aggregate feature is matched with all fourth features in the feature library, and based on the difference between each matched fourth feature and the sixth aggregate feature, the module parameters of the untrained aggregate projection module are adjusted, wherein the fourth feature is a preset feature whose defect type is different from the defect type corresponding to the training image.

[0057] According to yet another aspect of the present invention, a defect type detection device is provided, which includes: a defect area determination module, an image feature extraction module, a first aggregate feature generation module and an actual defect type determination module.

[0058] The defect area determination module is used to determine the defect area in the image to be tested of the object to be tested;

[0059] The image feature extraction module is used to extract the image features corresponding to the defect area in the image to be tested;

[0060] The first aggregate feature generation module is used to generate a first aggregate feature based on the image feature corresponding to the defect area in the image to be tested, using the trained aggregate projection module, wherein the trained aggregate projection module is trained based on the first global feature and the first defect feature of the training image of the object to be tested, the first global feature is the image feature of the training image as a whole, and the first defect feature is the image feature corresponding to the defect area in the training image;

[0061] The actual defect type determination module is used to determine the actual defect type of the object to be tested based on the first aggregated feature.

[0062] According to another aspect of the present invention, an electronic device is provided, which includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect type detection method when the processor is running.

[0063] According to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned defect type detection method when running.

[0064] According to another aspect of the present invention, a computer program product is also provided. The computer program product includes computer program instructions, and the computer program instructions are used to execute the above-mentioned defect type detection method when running.

[0065] According to the above scheme of the embodiment of the present invention, the defect area can be determined in the image to be tested of the object to be tested. Then, the image features corresponding to the defect area in the image to be tested are extracted. Based on the image features corresponding to the defect area in the image to be tested, the trained aggregation projection module is used to generate the first aggregation feature. Finally, based on the first aggregation feature, the actual defect type of the object to be tested is determined. The trained aggregation projection module in the above scheme is trained based on the first global feature and the first defect feature of the training image. Therefore, the first aggregation feature output by the trained aggregation projection module also pays attention to the global features of the image to be tested to a certain extent, which is conducive to improving the generation accuracy of the actual defect type. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0067] Figure 1 A schematic flow chart of a defect type detection method according to an embodiment of the present invention is shown;

[0068] Figure 2 A schematic diagram showing a defect type of scratch according to an embodiment of the present invention is shown;

[0069] Figure 3 A schematic diagram showing a defect type detection method according to an embodiment of the present invention;

[0070] Figure 4 A schematic block diagram showing a defect type detection device according to an embodiment of the present invention; and

[0071] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0073] In the field of industrial component manufacturing, if defective industrial components (such as wafers) are packaged and tested, it will waste manpower and material resources, so it is necessary to inspect industrial components so that unqualified industrial components can be detected and eliminated. However, there may be many defects on industrial components. For example, taking wafers as an example, there may be defects such as foreign matter and cracks on the wafer. For another example, taking the wafer grains on the wafer as an example, there may be defective needle marks such as needle marks out of bounds and too many needle marks on the wafer grains. The above-mentioned needle marks are the traces left when the needle test probe contacts the wafer grains during the needle test of the wafer grains. Under normal circumstances, needle marks do not affect the test results and performance of the wafer grains, but when the needle marks are offset or the needle marks are too deep, it may cause inaccurate needle test results and affect the performance of the wafer grains (for example: the needle marks are too deep to pierce the solder joints), thereby reducing the overall yield of the wafer.

[0074] With the continuous advancement of semiconductor technology and the development of industrial component manufacturing technology, the detection of defects on industrial components has become an indispensable part of the industrial component manufacturing process. With the continuous advancement of semiconductor technology, the device density and integration of industrial components are increasing, and the requirements for the detection of defects on industrial components are also getting higher and higher.

[0075] In the related art, the detection of defects on industrial components mainly relies on traditional image processing technology. However, the above detection method is not only inefficient and has a high error rate, but also cannot identify the type of defects on the industrial components.

[0076] In order to at least partially solve the above problems, an embodiment of the present invention provides a defect type detection method. Figure 1 FIG. 2 is a schematic flow chart of a defect type detection method according to an embodiment of the present invention. Figure 1 As shown, the detection method may include steps S110 to S140.

[0077] In step S110 , a defect area is determined in the image to be tested of the object to be tested.

[0078] The object to be tested may be an industrial component that needs to be quality inspected, such as a circuit board, a wafer, a die in a wafer, etc.

[0079] The image to be tested can be a static image or any video frame in a dynamic video. The image to be tested can be an original image acquired by an image acquisition device (for example, an original image acquired by an image sensor in a camera), or an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.).

[0080] In some embodiments, the defective area in the image to be tested can be determined by matching the image to be tested with the image template of the object to be tested. The image template of the object to be tested can be an image related to the target of interest during the quality inspection of the object to be tested, or it can be an image including the complete object to be tested. For example, when the object to be tested is a wafer, and the target of interest during the quality inspection is the pad on the wafer, the image template of the object to be tested can be an image of a wafer without any defects, or it can be an image including a single pad without any defects.

[0081] Taking the image template of the object to be tested as an example, which is an image related to the object to be tested and the target of interest during quality inspection, the image to be tested and the image template of the object to be tested can be preprocessed first, such as grayscale conversion, filtering and denoising, etc., to improve the accuracy and efficiency of matching. Then, a suitable matching method is selected according to the application requirements, such as matching based on square difference (SQDIFF), normalized square difference (SQDIFF_NORMED), and correlation coefficient (CCORR). The above matching process may include: placing the image template of the object to be tested at the upper left corner of the image to be tested, and then sliding on the image to be tested to compare each possible position. Then, for each position in the image to be tested, the similarity or difference between the image template of the object to be tested and the corresponding area of ​​the image to be tested is calculated, which can be a pixel-by-pixel comparison, and finally a result matrix is ​​obtained. Among them, each element in the result matrix represents the matching degree of the image template of the object to be tested at the corresponding position in the image to be tested. When the result matrix is ​​obtained, the cv2.minMaxLoc() function can be used to analyze the result matrix to find the position with the highest matching degree. The position with the highest matching degree can be used as the target area. For example, the target area may be the area corresponding to the pad in the image to be tested. Then, the pixels in the target area whose difference with the image template pixels is higher than a preset value may be regarded as part of the defect area, that is, the defect area in the image to be tested is obtained.

[0082] In some other embodiments, the image to be tested may be compared with a reference image, and based on the comparison result, at least a portion of the defective area outside the target area may be determined. The reference image may be a reference image in which the object to be tested does not have any defects and the target area is masked. For example, if the object to be tested is a wafer, an image of a wafer without defects such as pin marks, foreign matter, or scratches may be selected, and the target area in the image may be masked, so that targeted detection can be performed on the area outside the target area.

[0083] When comparing the image to be tested with the reference image, the pixel difference between each pixel in the image to be tested and its corresponding pixel in the reference image can be calculated, and when the absolute value of the calculated result is greater than the threshold, the pixels in the image to be tested whose absolute value of the pixel difference is greater than the threshold are clustered, and then based on the clustering result, the above-mentioned defect area is obtained. For example, if pixels 1 to 10 belong to a clustering result, the central pixel of pixels 1 to 10 can be calculated, and using the preset length and preset width, the defect area with the central pixel as the center of the area can be obtained. For another example, the pixels at the outermost periphery (smallest horizontal coordinate, smallest vertical coordinate, largest horizontal coordinate, largest vertical coordinate) of pixels 1 to 10 can be connected to obtain the defect area.

[0084] It should be understood that the above-mentioned image to be tested can also be detected by a defect area detection model to obtain a defect area. For example, the training process of the above-mentioned defect area detection model (such as a convolutional neural network, a generative adversarial network, etc.) may include: obtaining an image set, which includes an image of a reference object and a defect area corresponding to the image. The object type of the above-mentioned reference object is the same as the object type of the object to be tested. For example, if the object to be tested is a wafer, the reference object is also a wafer; if the object to be tested is a wafer grain, the reference object is also a wafer grain. The defect area detection model is iteratively trained through the image set, and the model parameters of the defect area detection model are adjusted through the loss function until the training is completed. The conditions for the completion of the above-mentioned training may be that the loss value calculated by the loss function tends to be stable, the number of iterations reaches a preset number, etc. Based on the above-mentioned training process, a defect area detection model can be obtained to determine the defect area in the image to be tested.

[0085] In step S120, image features corresponding to the defective area in the image to be tested are extracted.

[0086] The defective area in the image to be tested can be subjected to feature extraction by any feature extraction method in the related technology to obtain the image features corresponding to the defective area in the image to be tested. For example, feature extraction can be achieved by using a feature extraction algorithm (such as Harris corner detection operator, scale-invariant feature transformation, etc.) or a feature extractor (such as convolutional neural network, recurrent neural network, etc.).

[0087] In step S130, based on the image features corresponding to the defective area in the image to be tested, a first aggregate feature is generated using the trained aggregate projection module.

[0088] The trained aggregate projection module is trained based on the first global feature and the first defect feature of the training image of the object to be tested. The first global feature is the image feature of the training image as a whole, and the first defect feature is the image feature corresponding to the defect area in the training image. Exemplarily, the above-mentioned first global feature can be obtained by the feature extraction method described above. For example, the training image can be completely input into the corresponding operator or feature extractor of the feature extraction algorithm to obtain the first global feature. For another example, the image corresponding to the defect area in the training image can be input into the corresponding operator or feature extractor of the feature extraction algorithm to obtain the first defect feature. In some examples, the aggregate projection module may include a network layer in a neural network in the related technology, such as a fully connected layer, a pooling layer, etc. In some other examples, multiple aggregate projection modules can be connected to enhance the learning depth of deep learning and better fit complex data distributions.

[0089] Optionally, the image features corresponding to the defective area in the image to be tested can be directly input into the trained aggregation projection module to obtain the first aggregated features. Alternatively, the image features corresponding to the defective area in the image to be tested can be subjected to feature-related processing and then input into the trained aggregation projection module to obtain the first aggregated features. For example, the feature-related processing can include: feature screening, feature enhancement, feature dimensionality reduction, etc., which can be determined according to the actual needs of the developer.

[0090] In step S140, the actual defect type of the object to be tested is determined based on the first aggregated feature.

[0091] Optionally, the actual defect type of the object to be tested can be determined directly based on the first aggregate feature, for example, the first aggregate feature is input into the trained classification head to obtain the actual defect type of the object to be tested. The training process of the classification head may include: obtaining a feature set, which includes the image features of the reference object and the defect type corresponding to the image features. The object type of the above-mentioned reference object is the same as the object type of the object to be tested. For example, if the object to be tested is a wafer, the reference object is also a wafer; if the object to be tested is a wafer grain, the reference object is also a wafer grain. The classification head is iteratively trained through the feature set, and the model parameters of the classification head are adjusted through the loss function until the training is completed. The conditions for the completion of the above-mentioned training may be that the loss value calculated by the loss function tends to be stable, the number of iterations reaches a preset number, etc. Based on the above-mentioned training process, a trained classification head can be obtained to determine the defect area in the image to be tested.

[0092] Alternatively, the actual defect type of the object to be tested can also be determined by combining other image features of the object to be tested and the first aggregate feature. For example, the other image features may include global image features of the image to be tested, or image features whose feature extraction scale is different from the image features corresponding to the defect area. In one example, the other image features may be input into the above-mentioned trained aggregate projection module, and then the obtained image features are feature fused with the first aggregate features, and then input into the trained classification head to obtain the actual defect type of the object to be tested. It should be understood that the feature set of the trained classification head can be adjusted to adapt to the scene of this example. For example, the image features of the reference object in the feature set may include features obtained by feature fusion of the image features corresponding to the defect area of ​​the reference object and the above-mentioned other image features of the reference object. For another example, the image features of the reference object in the feature set may include features obtained by feature fusion of the image features corresponding to the defect area of ​​the reference object and image features with different feature extraction scales of the reference object.

[0093] According to the above scheme of the embodiment of the present invention, the defect area can be determined in the image to be tested of the object to be tested. Then, the image features corresponding to the defect area in the image to be tested are extracted. Based on the image features corresponding to the defect area in the image to be tested, the trained aggregation projection module is used to generate the first aggregation feature. Finally, based on the first aggregation feature, the actual defect type of the object to be tested is determined. The trained aggregation projection module in the above scheme is trained based on the first global feature and the first defect feature of the training image. Therefore, the first aggregation feature output by the trained aggregation projection module also pays attention to the global features of the image to be tested to a certain extent, which is conducive to improving the generation accuracy of the actual defect type.

[0094] Exemplarily, step S120 may include steps S121 to S123.

[0095] In step S121, the image to be tested is divided into a plurality of first image blocks.

[0096] The image block sizes of different first image blocks may be the same. In one example, the first image block may be obtained by processing an image segmentation algorithm. The image block size may be a value preset by a developer. In another example, the image to be tested may be input into a DINOV2 model in the related art to obtain the first image block and the image block features corresponding to the first image block.

[0097] In step S122, a first image block feature of a first image block located in a defective area in the image to be tested is extracted.

[0098] Optionally, the defective area may be represented as a set of pixel coordinates (for example, it may include the coordinates of all pixel points in the defective area, or it may include the coordinates of the four corner points of the defective area). Through the coordinate set, combined with the position of the first image block in the image to be tested, the first image block located in the defective area can be determined. In one example, feature extraction may be performed only on the first image block of the defective area in the image to be tested to obtain the first image block features of each first image block. In another example, based on the position information of the defective area, the image block features of the first image block located in the defective area may be screened from the image block features corresponding to the first image block output by the DINOV2 model above to serve as the first image block features.

[0099] In step S123, all first image block features are fused to obtain fused features as image features corresponding to the defect area.

[0100] In one example, the first image block features may be fused by a concat operator (concatenation operator) or an add operator (addition operator) in the related art to obtain a fused feature. In another example, the average value of all first image block features may be calculated as the fused feature.

[0101] According to the above scheme of the embodiment of the present invention, the image to be tested can be divided into multiple first image blocks. Then, the first image block features of the first image block located in the defect area in the image to be tested are extracted. Finally, all the first image block features are fused to obtain the fused features as the image features corresponding to the defect area. The above scheme can effectively retain local detail information by dividing the image to be tested into multiple first image blocks and then extracting features from the first image blocks, especially when the defect area on the image to be tested is small or the shape of the defect area is irregular. The above scheme can improve the representativeness of the image features corresponding to the defect area, which is conducive to improving the accuracy of the actual defect type of the object to be tested.

[0102] Exemplarily, step S140 of determining the actual defect type of the object to be tested based on the first aggregated feature may include steps S141 and S142.

[0103] In step S141, the first aggregated feature is matched with a preset feature in a feature library, and based on the defect types corresponding to each of the matched preset features, the defect type corresponding to the first aggregated feature is determined.

[0104] The feature library stores multiple preset features and the defect type corresponding to each preset feature. The generation method of the preset features can be based on the pre-acquired image with the defect type annotated, and refer to the generation method of the first aggregated feature, which is not described in detail in the embodiment of the present invention.

[0105] The above-mentioned matching of the first aggregate feature with the preset feature in the feature library can determine the similarity between the first aggregate feature and the preset feature by calculating Euclidean distance, cosine similarity, Manhattan distance, etc. In some embodiments, the defect type of the preset feature with the highest similarity can also be directly used as the defect type corresponding to the first aggregate feature. Optionally, for the similarity between each preset feature and the first aggregate feature, if the similarity of the preset feature is higher than the preset similarity, the preset feature can be used as one of the multiple preset features for matching. Alternatively, for the similarity between each preset feature and the first aggregate feature, if the similarity of the preset feature is higher than the preset similarity, it can be used as one of the preset features for preliminary matching. Then, among the preset features for preliminary matching, the preset features with the highest similarity and a preset number are selected as the multiple preset features for matching the first aggregate feature. It should be understood that the developer can determine the rules used for matching in combination with actual needs, and the embodiment of the present invention is not limited here.

[0106] In some embodiments, the above-mentioned determination of the defect type corresponding to the first aggregate feature based on the defect type corresponding to each of the matched multiple preset features may include: taking the defect type with the largest total number among the defect types corresponding to each of the matched multiple preset features as the defect type corresponding to the first aggregate feature. For example, if the multiple matched preset features include preset features 1 to 5, the defect types of preset features 1 to 3 are all the existence of cracks, and the defect types of preset features 4 and 5 are both pin prints out of bounds. Then the existence of cracks is the defect type with the largest total number, and the defect type corresponding to the first aggregate feature may be determined to be the existence of cracks. In some other embodiments, the above-mentioned determination of the defect type corresponding to the first aggregate feature based on the defect type corresponding to each of the matched multiple preset features may include: according to the similarity, determining the defect type with the largest sum of similarities among the defect types corresponding to each of the matched multiple preset features as the defect type corresponding to the first aggregate feature. Taking the preset features 1 to 5 in the above text as an example, the similarity of preset feature 1 is 0.5, the similarity of preset feature 2 is 0.3, the similarity of preset feature 3 is 0.4, the similarity of preset feature 4 is 0.6, and the similarity of preset feature 5 is 0.7. The sum of the similarities of the defect type of cracks is 1.2 (0.5+0.3+0.4), and the sum of the similarities of the defect type of pin print out of bounds is 1.3 (0.6+0.7). In this case, pin print out of bounds can be used as the defect type corresponding to the first aggregate feature. In some other embodiments, the mean similarity of matching defect types of the same type can also be calculated. Then, the defect type with the largest mean similarity is used as the defect type corresponding to the first aggregate feature. In combination with the above example, the mean similarity of the defect type of cracks is 0.4 (1.2 / 3), and the mean similarity of the defect type of pin print out of bounds is 0.65 (1.3 / 2). In this case, pin print out of bounds can be used as the defect type corresponding to the first aggregate feature.

[0107] In one example, if a new defect type is to be added, the image of the new defect type can be input into the trained aggregation projection module to obtain a new defect feature, and the new defect feature and the new defect type can be directly saved in the feature library.

[0108] In step S142, based on the defect type corresponding to the first aggregated feature, the actual defect type of the object to be tested is determined.

[0109] In one example, the defect type corresponding to the first aggregate feature can be directly used as the actual defect type of the object to be tested. In another example, the actual defect type of the object to be tested can also be determined in combination with the defect types corresponding to other image features. For example, the actual defect type of the object to be tested can be determined in combination with the defect types corresponding to other image features of the image to be tested above. For example, the defect type corresponding to the global image feature and the defect type corresponding to the first aggregate feature with a higher similarity (for example, the defect type corresponding to the first aggregate feature is the pin print out of bounds mentioned above, then the similarity corresponding to this defect type can be 1.3 or 0.7, and the similarity calculation process of the global image feature can refer to the similarity calculation process of the first aggregate feature) can be used as the actual defect type of the object to be tested. The similarity generation process can refer to the above, and the embodiments of the present invention will not be described in detail here.

[0110] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram showing a defect type of scratch according to an embodiment of the present invention. Figure 2 , Scratches 1 and 2 are the same defect type that occur at different processing stages. Therefore, due to the different processing stages, the visual effects of the same defect type may be quite different. If a machine learning model is used to determine the actual defect type, the adaptability of the machine learning model will be low, and the accuracy of the actual defect type that may be output may be low. However, the preset features of the feature library in the above solution can be flexibly updated, so it has good adaptability for any defect type at different processing stages, and is also conducive to adding preset features of new defect types according to different quality inspection requirements.

[0111] According to the above scheme of the embodiment of the present invention, the first aggregate feature can be matched with the preset feature in the feature library, and the defect type corresponding to the first aggregate feature can be determined based on the defect types corresponding to each of the matched multiple preset features. Then, based on the defect type corresponding to the first aggregate feature, the actual defect type of the object to be tested is determined. The above scheme can determine the actual defect type of the object to be tested by setting the feature library. Compared with the method of using a machine learning model to determine the actual defect type, when facing a new defect type, the above scheme can directly add the characteristics of the defect type as a preset feature to the feature library, avoiding the need to retrain the machine learning model, which is conducive to adapting to actual scenarios. Especially in the case of too many defect types for wafers and wafer grains, if the machine learning model is retrained, the training cost and model update cost will be too high.

[0112] Exemplarily, the above detection method may further include steps S210 to S250.

[0113] In step S210 , for each initial image, a defect area is determined in the initial image.

[0114] The above-mentioned initial image may be the image of the reference object mentioned above. The method for determining the defect area in the initial image may refer to the method for determining the defect area in the image to be tested, and the embodiment of the present invention will not be described in detail here.

[0115] In step S220, for each initial image, the initial image is divided into a plurality of second image blocks, and second image block features of the second image blocks located in the defective area of ​​the initial image are generated.

[0116] The method of dividing the above-mentioned initial image into multiple second image blocks can refer to the method of dividing the image to be tested into multiple first image blocks mentioned above, and the method of generating the second image block features can refer to the method of generating the first image block features, which will not be elaborated in the embodiment of the present invention.

[0117] In step S230, for each initial image, all second image block features of the initial image are fused to obtain second fused features of the initial image.

[0118] The feature fusion method of the second image block features may refer to the feature fusion method of the first image block mentioned above, and the embodiment of the present invention will not be described in detail here.

[0119] In step S240, for each initial image, the second fusion feature of the initial image is input into an untrained aggregation projection module to obtain the second aggregation feature of the initial image.

[0120] In one example, the untrained aggregation projection module may include a network layer in a neural network in the related art, such as a fully connected layer, a pooling layer, etc. Optionally, for each initial image, the second fusion feature of the initial image may be directly input into the untrained aggregation projection module to obtain the second aggregation feature of the initial image. Alternatively, multiple untrained aggregation projection modules may be connected to enhance the learning depth of subsequent deep learning and better fit complex data distribution. For example, for each initial image, the second fusion feature of the initial image may be input into the sequentially connected, untrained aggregation projection modules.

[0121] In step S250, for each defect type, the second aggregation features of all initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature.

[0122] The above clustering operation may include K clustering (K-mean clustering), hierarchical clustering, etc. For each defect type, the second aggregate features of all initial images corresponding to the defect type are clustered to obtain the cluster center features of the second aggregate features of all initial images. It should be understood that the cluster center feature may be different from any second aggregate feature value. Since the second aggregate feature may have a large distribution span, there may be multiple cluster center features for this defect type. For example, if the clustering operation is obtained through K clustering, the value of K in the K clustering may be set to generate K cluster center features.

[0123] Combined with the actual scenario, here we take the object to be tested as a wafer grain and use K clustering for clustering operation as an example. If the defect type is pin print out of bounds and K is 4, the second aggregation feature can be subjected to K clustering operation to obtain 4 cluster center features. The defect type corresponding to each cluster center feature is pin print out of bounds. Based on this, a preset feature corresponding to the pin print out of bounds in the feature library can be established. By matching the preset feature in the feature library with the first aggregation feature of the object to be tested, it can be determined whether the actual defect type of the object to be tested is pin print out of bounds. Similarly, by adding the cluster center features of multiple defect types to the feature library, the specific actual defect type of the object to be tested can be determined.

[0124] In one example, if you want to add a new defect type, you can input the image of the new defect type into the trained aggregation projection module to obtain a new defect feature. Then, through the clustering operation described above, based on the defect features of the new image, you can obtain the cluster center feature of the new defect type. Finally, save the cluster center feature and the new defect type into the feature library.

[0125] According to the above scheme of the embodiment of the present invention, for each initial image, a defect area can be determined in the initial image. Then, the initial image is divided into a plurality of second image blocks, and a second image block feature of the second image block located in the defect area in the initial image is generated. All the second image block features of the initial image are then subjected to feature fusion to obtain the second fused feature of the initial image. Then, the second fused feature of the initial image is input into an untrained aggregation projection module to obtain the second aggregate feature of the initial image. Finally, for each defect type, the second aggregate features of all the initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature. The above scheme can effectively retain local detail information by generating the second image block feature, especially when the defect area on the initial image is small or the shape of the defect area is irregular. On the one hand, the above scheme can improve the representativeness of the second fusion feature, and then improve the representativeness of the preset features in the feature library, which is conducive to improving the accuracy of the actual defect type; on the other hand, the above scheme obtains a more representative cluster center feature by clustering the second aggregation feature, which can improve the matching efficiency of the preset features and also improve the accuracy of the actual defect type.

[0126] Exemplarily, the above detection method may further include steps S310 to S360.

[0127] In step S310 , a defect area is determined in a training image.

[0128] The process of determining the defect area in the training image may refer to the process of determining the defect area in the image to be tested described above, and the embodiments of the present invention will not be described in detail here.

[0129] In step S320, the training image is divided into a plurality of third image blocks, and third image block features of the third image blocks located in the defective area of ​​the training image are generated as first defect features.

[0130] The method of dividing the above-mentioned training image into multiple third image blocks can refer to the method of dividing the test image into multiple first image blocks mentioned above, and the method of generating the third image block features can refer to the method of generating the first image block features, which will not be elaborated in the embodiments of the present invention.

[0131] In step S330, all third image blocks are subjected to feature fusion to obtain a third fusion feature.

[0132] The feature fusion method of the third image block features may refer to the feature fusion method of the first image block mentioned above, and the embodiment of the present invention will not be described in detail here.

[0133] In step S340, the third fusion feature is input into the untrained aggregation projection module to obtain the third aggregation feature.

[0134] In one example, the untrained aggregation projection module may include a network layer in a neural network in the related art, such as a fully connected layer, a pooling layer, etc. Optionally, the third fusion feature may be directly input into the untrained aggregation projection module to obtain the third aggregation feature. Alternatively, multiple untrained aggregation projection modules may be connected to enhance the learning depth of subsequent deep learning and better fit complex data distribution. For example, the third fusion feature may be input into sequentially connected, untrained aggregation projection modules to obtain the third aggregation feature.

[0135] In step S350, the third aggregate feature is matched with the preset features in the feature library, and based on the matched multiple preset features, the module parameters of the untrained aggregate projection module are adjusted.

[0136] The module parameters of the untrained aggregate projection module can be adjusted based on the difference between the defect type of each of the matched multiple preset features and the defect type of the training image and the difference between the preset feature and the third aggregate feature of the training image. For example, if the defect type of the preset feature is the same as the defect type of the training image, reducing the difference between the preset feature and the third aggregate feature of the training image can be used as a training goal to adjust the module parameters of the untrained aggregate projection module. For another example, if the defect type of the preset feature is different from the defect type of the training image, increasing the difference between the preset feature and the third aggregate feature of the training image can be used as a training goal to adjust the module parameters of the untrained aggregate projection module. It should be understood that the loss function used to adjust the module parameters may also include L1 loss, L2 loss, etc. in the relevant technology, and the corresponding adjustment method is not described in detail in the embodiment of the present invention.

[0137] In step S360, for each preset feature matching the third aggregate feature, the preset feature in the feature library is updated based on the third aggregate feature.

[0138] In one example, for each preset feature that matches the third aggregate feature, an average value of the matched preset feature and the third aggregate feature may be directly calculated to update the preset feature in the feature library to the average value.

[0139] In yet another example, the preset features matched by the third aggregated feature may be updated by the following formula.

[0140] ;

[0141] Among them, β is the average interaction coefficient, C iis the third aggregate feature of the i-th training image, C iter is the preset feature for matching, C iter+1 is the updated preset feature, and the updated preset feature is used for matching the training images and the preset features in the iter+1th batch.

[0142] According to the above scheme of the embodiment of the present invention, the defect area can be determined in the training image. Then, the training image is divided into a plurality of third image blocks, and the third image block features of the third image blocks located in the defect area in the training image are generated as the first defect features. Then, all the third image blocks are subjected to feature fusion to obtain the third fusion feature. Then, the third fusion feature is input into the untrained aggregation projection module to obtain the third aggregation feature. Then, the third aggregation feature is matched with the preset feature in the feature library, and the module parameters of the untrained aggregation projection module are adjusted based on the matched multiple preset features. Finally, for each preset feature matched with the third aggregation feature, the preset feature in the feature library is updated based on the third aggregation feature. The above scheme can effectively retain the local detail information of the training image by generating the third image block feature, especially when the defect area on the training image is small or the shape of the defect area is irregular, which is conducive to improving the learning efficiency and feature expression ability of the aggregation projection module. On the one hand, the above scheme can improve the representativeness of the third fusion feature, and then improve the representativeness of the preset features in the feature library, which is beneficial to improving the accuracy of the actual defect type; on the other hand, the above scheme obtains a cluster center feature with higher representativeness by clustering the third aggregation feature, which can improve the matching efficiency of the preset feature; on the other hand, the preset features in the feature library can be dynamically updated through the cluster center features of the third aggregation features of the training image to gradually improve the representativeness of the preset features in the feature library, thereby improving the learning efficiency of the aggregation projection module, which is beneficial to improving the accuracy of the actual defect type of the object to be tested.

[0143] Exemplarily, step S350 matches the third aggregate feature with the preset features in the feature library, and adjusts the module parameters of the untrained aggregate projection module based on the matched multiple preset features, which may include: matching the third aggregate feature with all preset features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the differences between each matched preset feature and the third aggregate feature.

[0144] In one example, the difference between the preset features and the third aggregate feature can be determined by a loss function in the related art. In another example, loss1 can be determined by the following formula, where loss1 is used to represent the difference between the preset features and the third aggregate feature.

[0145] ;

[0146] Where N is the total number of matched preset features. If the defect type of the matched j-th preset feature is the same as the defect type corresponding to the training image, If yes, it is 1, otherwise it is 0. is the similarity between the jth preset feature and the third aggregated feature.

[0147] According to the above scheme of the embodiment of the present invention, the third aggregate feature can be matched with all preset features in the feature library, and based on the difference between each matched preset feature and the third aggregate feature, the module parameters of the untrained aggregate projection module are adjusted. In the above scheme, the defect type of the preset feature matched may be the same as or different from the defect type of the training image, so the generalization ability of the aggregate projection module can be improved, which is conducive to improving the accuracy of the actual defect type of the object to be tested.

[0148] Exemplarily, step S350 matches the third aggregate feature with the preset features in the feature library, and adjusts the module parameters of the untrained aggregate projection module based on the matched multiple preset features, which may include: matching the third aggregate feature with all the first features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the differences between each matched first feature and the third aggregate feature.

[0149] The first feature is a preset feature whose defect type is the same as the defect type corresponding to the training image.

[0150] In one example, the difference between the first feature and the third aggregate feature can be determined by a loss function in the related art. In another example, loss2 can be determined by the following formula, and loss2 is used to represent the difference between the first feature and the third aggregate feature.

[0151] ;

[0152] Where N is the total number of matched first features, is 1, is the similarity between the mth first feature and the third aggregated feature.

[0153] According to the above scheme of the embodiment of the present invention, the third aggregate feature can be matched with all the first features in the feature library, and based on the difference between each matched first feature and the third aggregate feature, the module parameters of the untrained aggregate projection module are adjusted. In the above scheme, the defect type of the matched first feature should be the same as the defect type of the training image, so the recognition ability of the aggregate projection module for features of the same defect type can be improved, which is conducive to improving the accuracy of the actual defect type of the object to be tested.

[0154] Exemplarily, step S350 matches the third aggregate feature with the preset features in the feature library, and adjusts the module parameters of the untrained aggregate projection module based on the matched multiple preset features, which may include: matching the third aggregate feature with all the second features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the differences between each matched second feature and the third aggregate feature.

[0155] The second feature is a preset feature whose defect type is different from the defect type corresponding to the training image.

[0156] In one example, the difference between the second feature and the third aggregate feature can be determined by a loss function in the related art. In another example, loss3 can be determined by the following formula, and loss3 is used to represent the difference between the first feature and the third aggregate feature.

[0157] ;

[0158] Where N is the total number of matched second features, is 0, is the similarity between the nth second feature and the third aggregated feature.

[0159] According to the above scheme of the embodiment of the present invention, the third aggregate feature can be matched with all the second features in the feature library, and based on the difference between each matched second feature and the third aggregate feature, the module parameters of the untrained aggregate projection module are adjusted. In the above scheme, the defect type of the second feature matched should be different from the defect type of the training image, so the recognition ability of the aggregate projection module for features of different defect types can be improved, which is conducive to improving the accuracy of the actual defect type of the object to be tested.

[0160] In some embodiments, the overall loss of the aggregated projection module may include loss1, loss2, and loss3 mentioned above.

[0161] Exemplarily, the above detection method further includes steps S410 and S420.

[0162] In step S410, a second global feature of the image to be tested is generated.

[0163] The second global feature is the image feature of the entire image to be tested. The image to be tested can be subjected to feature extraction by any of the feature extraction methods described above to obtain the second global feature corresponding to the image to be tested.

[0164] In step S420, the second global feature is input into the trained aggregation projection module to obtain a fourth aggregation feature.

[0165] Optionally, the second global feature can be directly input into the trained aggregation projection module to obtain the fourth aggregation feature. Alternatively, the trained aggregation projection modules can be connected to enhance the learning depth of subsequent deep learning and better fit the complex data distribution. For example, the second global feature is input into the trained aggregation projection modules connected in sequence.

[0166] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing a defect type detection method according to an embodiment of the present invention. Figure 3 , the image to be tested can be divided into multiple first image blocks. Then, the image block features of each first image block are generated, and the image block features of the first image block located in the defect area are used as the first image block features. The first image block features are then subjected to the feature fusion described above to obtain the fusion features, and are input into the sequentially connected aggregation projection modules to obtain the first aggregation features. In this example, a second global feature of the entire image to be tested can also be generated. Then, the second global feature is input into the sequentially connected aggregation projection modules to obtain the fourth aggregation feature.

[0167] Step S140 determines the actual defect type of the object to be tested based on the first aggregate feature, which may include: Step S141, determining the actual defect type of the object to be tested based on the first aggregate feature and the fourth aggregate feature.

[0168] Optionally, the first aggregate feature and the fourth aggregate feature may be feature fused, and then matched with the preset features in the feature library. Alternatively, the defect types corresponding to the first aggregate feature and the fourth aggregate feature may be determined respectively, and then the actual defect type of the object to be tested may be determined comprehensively. For example, multiple preset features matched by the first aggregate feature and multiple preset features matched by the second aggregate feature may be determined respectively, and then the defect types of the multiple preset features matched by the first aggregate feature and the defect types of the multiple preset features matched by the second aggregate feature may be combined to determine the actual defect type of the object to be tested.

[0169] According to the above scheme of the embodiment of the present invention, a second global feature of the image to be tested can be generated. Then, the second global feature is input into the trained aggregation projection module to obtain a fourth aggregation feature. Finally, based on the first aggregation feature and the fourth aggregation feature, the actual defect type of the object to be tested is determined. The above scheme comprehensively considers the first aggregation feature and the fourth aggregation feature, and the actual defect type determined from the local and global levels is more accurate.

[0170] Exemplarily, step S141 may include steps S1411 and S1412.

[0171] In step S1411, the fourth aggregate feature is matched with a preset feature in a feature library, and based on the defect types corresponding to each of the matched preset features, the defect type corresponding to the fourth aggregate feature is determined.

[0172] The above-mentioned matching of the fourth aggregate feature with the preset features in the feature library can determine the similarity between the fourth aggregate feature and the preset features by calculating Euclidean distance, cosine similarity, Manhattan distance, etc. In some embodiments, the defect type of the preset feature with the highest similarity can also be directly used as the defect type corresponding to the fourth aggregate feature. Optionally, for the similarity between each preset feature and the fourth aggregate feature, if the similarity of the preset feature is higher than the preset similarity, the preset feature can be used as one of the multiple preset features for matching. Alternatively, for the similarity between each preset feature and the fourth aggregate feature, if the similarity of the preset feature is higher than the preset similarity, it can be used as one of the preset features for preliminary matching. Then, among the preset features for preliminary matching, the preset features with the highest similarity and a preset number are selected as the multiple preset features for matching the fourth aggregate feature. It should be understood that the developer can determine the rules used for matching in combination with actual needs, and the embodiment of the present invention is not limited here.

[0173] In some embodiments, the above-mentioned determination of the defect type corresponding to the fourth aggregate feature based on the defect type corresponding to each of the matched multiple preset features may include: taking the defect type with the largest total number among the defect types corresponding to each of the matched multiple preset features as the defect type corresponding to the fourth aggregate feature. For example, if the multiple matched preset features include preset features 6 to 10, the defect types of preset features 6 to 8 are all the presence of foreign matter, and the defect types of preset features 9 and 10 are all the excessive number of pin marks in the pad area. If the presence of foreign matter is the defect type with the largest total number, the defect type corresponding to the fourth aggregate feature may be determined to be the presence of foreign matter. In some other embodiments, the above-mentioned determination of the defect type corresponding to the fourth aggregate feature based on the defect type corresponding to each of the matched multiple preset features may include: according to the similarity, determining the defect type with the largest sum of similarities among the defect types corresponding to each of the matched multiple preset features as the defect type corresponding to the fourth aggregate feature. Taking the preset features 6 to 10 in the above text as an example, the similarity of preset feature 6 is 0.5, the similarity of preset feature 7 is 0.3, the similarity of preset feature 8 is 0.4, the similarity of preset feature 9 is 0.6, and the similarity of preset feature 10 is 0.7. The sum of the similarities of the defect type of the presence of foreign matter is 1.2 (0.5+0.3+0.4), and the sum of the similarities of the defect type of the excessive number of pin marks in the pad area is 1.3 (0.6+0.7). The excessive number of pin marks in the pad area can be used as the defect type corresponding to the fourth aggregate feature. In some other embodiments, the mean similarity of the matching defect type of the same type can also be calculated. Then, the defect type with the largest mean similarity is used as the defect type corresponding to the fourth aggregate feature. Combining the above example, the mean similarity of the defect type of the presence of foreign matter is 0.4 (1.2 / 3), and the mean similarity of the defect type of the excessive number of pin marks in the pad area is 0.65 (1.3 / 2). Therefore, the excessive number of pin marks in the pad area can be regarded as the defect type corresponding to the fourth aggregation feature.

[0174] In step S1412, based on the defect type corresponding to the first aggregate feature and the defect type corresponding to the fourth aggregate feature, the actual defect type of the object to be tested is determined.

[0175] In one example, the defect type corresponding to the fourth aggregate feature and the defect type corresponding to the first aggregate feature with a higher similarity (refer to the calculation process of the similarity of the first aggregate feature in the above text) can be used as the actual defect type of the object to be tested. In another example, the defect type corresponding to the fourth aggregate feature and the defect type corresponding to the first aggregate feature can also be used as part of the actual defect type, that is, the actual defect type can be a composite defect type.

[0176] According to the above scheme of the embodiment of the present invention, the fourth aggregate feature can be matched with the preset features in the feature library, and the defect type corresponding to the fourth aggregate feature can be determined based on the defect types corresponding to the matched multiple preset features. Then, based on the defect type corresponding to the first aggregate feature and the defect type corresponding to the fourth aggregate feature, the actual defect type of the object to be tested is determined. The above scheme can combine the defect type corresponding to the first aggregate feature and the defect type corresponding to the fourth aggregate feature to comprehensively determine the actual defect type of the object to be tested, and the actual defect type determined from the local and global levels is more accurate.

[0177] Illustratively, the above detection method may include: steps S510 to S530.

[0178] In step S510 , for each initial image, a third global feature of the initial image is generated.

[0179] The initial image corresponds to a defect type, and the third global feature is an image feature of the entire initial image. The initial image can be feature extracted by any feature extraction method described above to obtain the third global feature corresponding to the initial image.

[0180] In step S520, for each initial image, the third global feature of the initial image is input into an untrained aggregation projection module to obtain a fifth aggregation feature of the initial image.

[0181] Optionally, for each initial image, the third global feature of the initial image can be directly input into an untrained aggregation projection module to obtain a fifth aggregation feature of the initial image. Alternatively, the untrained aggregation projection modules can be connected to enhance the learning depth of subsequent deep learning and better fit complex data distribution. For example, for each initial image, the third global feature of the initial image is input into sequentially connected, untrained aggregation projection modules to obtain the fifth aggregation feature of the initial image.

[0182] In step S530, for each defect type, the fifth aggregation features of all initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature.

[0183] The fifth aggregation feature may be processed by any of the above clustering operations to obtain a cluster center feature, and added to the feature library as a preset feature, which will not be described in detail in the embodiment of the present invention.

[0184] In one example, if you want to add a new defect type, you can input the image of the new defect type completely into the trained aggregation projection module to obtain a new global feature. Then, through the clustering operation mentioned above, you can get the cluster center feature of the new defect type. Finally, save the cluster center feature and the new defect type into the feature library.

[0185] In another example, before the aggregate projection module is trained, the feature library may include the cluster center feature obtained by clustering the second aggregate feature of the defect area and the cluster center feature obtained by clustering the fifth aggregate feature of the third global feature. Then, the preset features in the feature library may be dynamically updated as the aggregate projection module is trained, so that the representativeness of the preset features in the feature library continues to increase.

[0186] According to the above scheme of the embodiment of the present invention, for each initial image, a third global feature of the initial image can be generated. Then, the third global feature of the initial image is input into the untrained aggregation projection module to obtain the fifth aggregation feature of the initial image. Finally, for each defect type, the fifth aggregation features of all the initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature. The above scheme obtains a more representative cluster center feature by clustering the fifth aggregation feature, which can improve the matching efficiency of the preset features and the accuracy of the actual defect type. In addition, the fifth aggregation feature is an image feature at the global level, so it can improve the generalization ability of the aggregation projection head.

[0187] Exemplarily, the above detection method further includes: steps S610 to S640.

[0188] In step S610, a fourth global feature of the training image is generated.

[0189] The fourth global feature is the image feature of the entire training image. The feature extraction of the training image can be performed by any of the feature extraction methods described above to obtain the fourth global feature corresponding to the training image.

[0190] In step S620, the fourth global feature is input into an untrained aggregation projection module to obtain a sixth aggregation feature.

[0191] Optionally, the fourth global feature may be directly input into an untrained aggregation projection module to obtain a sixth aggregation feature. Alternatively, the untrained aggregation projection modules may be connected to enhance the learning depth of subsequent deep learning and better fit complex data distribution. For example, the fourth global feature may be input into sequentially connected, untrained aggregation projection modules to obtain a sixth aggregation feature.

[0192] In step S630, the sixth aggregate feature is matched with the preset features in the feature library, and based on the matched multiple preset features, the module parameters of the untrained aggregate projection module are adjusted.

[0193] The module parameters of the untrained aggregate projection module can be adjusted based on the difference between the defect type of each of the matched multiple preset features and the defect type of the training image and the difference between the preset feature and the sixth aggregate feature of the training image. For example, if the defect type of the preset feature is the same as the defect type of the training image, reducing the difference between the preset feature and the sixth aggregate feature of the training image can be used as a training goal to adjust the module parameters of the untrained aggregate projection module. For another example, if the defect type of the preset feature is different from the defect type of the training image, increasing the difference between the preset feature and the sixth aggregate feature of the training image can be used as a training goal to adjust the module parameters of the untrained aggregate projection module. It should be understood that the loss function used to adjust the module parameters may also include L1 loss, L2 loss, etc. in the relevant technology, and the corresponding adjustment method is not described in detail in the embodiment of the present invention.

[0194] In step S640, for each preset feature matching the sixth aggregate feature, the preset feature in the feature library is updated based on the sixth aggregate feature.

[0195] In one example, for each preset feature that matches the sixth aggregate feature, an average value of the matched preset feature and the sixth aggregate feature may be directly calculated to update the preset feature in the feature library to the average value.

[0196] In another example, the preset features matched by the sixth aggregated feature can be updated by the formula in step S360 above. In this example, C i It can be adjusted to the sixth aggregated feature of the i-th training image.

[0197] According to the above scheme of the embodiment of the present invention, the fourth global feature of the training image can be generated. Then, the fourth global feature is input into the untrained aggregation projection module to obtain the sixth aggregation feature. Then, the sixth aggregation feature is matched with the preset feature in the feature library, and the module parameters of the untrained aggregation projection module are adjusted based on the matched multiple preset features. Finally, for each preset feature matched with the sixth aggregation feature, the preset feature in the feature library is updated based on the sixth aggregation feature. On the one hand, in the above scheme, by clustering the sixth aggregation feature, a cluster center feature with higher representativeness is obtained, which can improve the matching efficiency of the preset feature; on the other hand, the preset feature in the feature library can be dynamically updated by the cluster center feature of the sixth aggregation feature of the training image, so as to gradually improve the representativeness of the preset feature in the feature library, thereby improving the learning efficiency of the aggregation projection module, which is conducive to improving the accuracy of the actual defect type of the object to be tested. In addition, the sixth aggregation feature is an image feature at the global level, so it can improve the generalization ability of the aggregation projection head.

[0198] Exemplarily, step S630 matches the sixth aggregate feature with a preset feature in the feature library, and adjusts the module parameters of the untrained aggregate projection module based on the matched multiple preset features, which may include: matching the sixth aggregate feature with all preset features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the differences between each matched preset feature and the sixth aggregate feature.

[0199] In one example, the difference between the preset features and the sixth aggregate feature can be determined by the loss function in the related art. In another example, the difference between the preset features and the sixth aggregate feature can be determined by the formula of loss1 above. In this example, is the similarity between the jth preset feature and the sixth aggregated feature.

[0200] According to the above scheme of the embodiment of the present invention, the sixth aggregate feature can be matched with all preset features in the feature library, and based on the difference between each matched preset feature and the sixth aggregate feature, the module parameters of the untrained aggregate projection module are adjusted. The defect type of the preset feature matched in the above scheme may be the same as or different from the defect type of the training image, so the generalization ability of the aggregate projection module can be improved, which is beneficial to improving the accuracy of the actual defect type of the object to be tested. In addition, using the sixth aggregate feature representing the global feature as one of the learning contents of the untrained aggregate projection module is beneficial to improving the feature learning ability of the trained aggregate projection module at the global level, thereby improving the accuracy of the actual defect type obtained.

[0201] Exemplarily, step S630 matches the sixth aggregate feature with a preset feature in the feature library, and adjusts the module parameters of the untrained aggregate projection module based on the matched multiple preset features, which may include: matching the sixth aggregate feature with all third features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched third feature and the sixth aggregate feature.

[0202] The third feature is a preset feature that the defect type is the same as the defect type corresponding to the training image.

[0203] In one example, the difference between the third feature and the sixth aggregate feature can be determined by the loss function in the related art. In another example, the difference between the third feature and the sixth aggregate feature can be determined by the formula of loss2 above. In this example, is the similarity between the mth third feature and the sixth aggregated feature.

[0204] According to the above scheme of the embodiment of the present invention, the sixth aggregate feature can be matched with all the third features in the feature library, and the module parameters of the untrained aggregate projection module can be adjusted based on the difference between each matched third feature and the sixth aggregate feature. The defect type of the third feature matched in the above scheme should be the same as the defect type of the training image, so the expression ability of the aggregate projection module for the features of the same defect type can be improved, which is beneficial to improving the accuracy of the actual defect type of the object to be tested. In addition, using the sixth aggregate feature representing the global feature as one of the learning contents of the untrained aggregate projection module is beneficial to improving the feature learning ability of the trained aggregate projection module at the global level, thereby improving the accuracy of the actual defect type obtained.

[0205] Exemplarily, step S630 matches the sixth aggregate feature with a preset feature in the feature library, and adjusts the module parameters of the untrained aggregate projection module based on the matched multiple preset features, which may include: matching the sixth aggregate feature with all fourth features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched fourth feature and the sixth aggregate feature.

[0206] The fourth feature is a preset feature that the defect type is different from the defect type corresponding to the training image.

[0207] In one example, the difference between the fourth feature and the sixth aggregate feature can be determined by the loss function in the related art. In another example, the difference between the fourth feature and the sixth aggregate feature can be determined by the formula of loss3 above. In this example, is the similarity between the nth fourth feature and the sixth aggregated feature.

[0208] According to the above scheme of the embodiment of the present invention, the sixth aggregate feature can be matched with all the fourth features in the feature library, and the module parameters of the untrained aggregate projection module can be adjusted based on the difference between each matched fourth feature and the sixth aggregate feature. The defect type of the fourth feature matched in the above scheme should be different from the defect type of the training image, so the expression ability of the aggregate projection module for features of different defect types can be improved, which is beneficial to improving the accuracy of the actual defect type of the object to be tested. In addition, using the sixth aggregate feature representing the global feature as one of the learning contents of the untrained aggregate projection module is beneficial to improving the feature learning ability of the trained aggregate projection module at the global level, thereby improving the accuracy of the actual defect type obtained.

[0209] An embodiment of the present invention also provides a defect type detection device. Figure 4 FIG. 7 is a schematic block diagram of a defect type detection device 700 according to an embodiment of the present invention. Figure 4 As shown, the detection device 700 may include: a defect area determination module 710, an image feature extraction module 720, a first aggregate feature generation module 730 and an actual defect type determination module 740.

[0210] The defect area determination module 710 is used to determine the defect area in the image to be tested of the object to be tested. The image feature extraction module 720 is used to extract the image features corresponding to the defect area in the image to be tested. The first aggregate feature generation module 730 is used to generate the first aggregate feature based on the image features corresponding to the defect area in the image to be tested, using the trained aggregate projection module, wherein the trained aggregate projection module is trained based on the first global feature and the first defect feature of the training image of the object to be tested, the first global feature is the image feature of the training image as a whole, and the first defect feature is the image feature corresponding to the defect area in the training image. The actual defect type determination module 740 is used to determine the actual defect type of the object to be tested based on the first aggregate feature.

[0211] Exemplarily, the image feature extraction module 720 may include: a first image block generation module, a first image block feature extraction module and a first image block fusion module.

[0212] The first image block generation module is used to divide the image to be tested into a plurality of first image blocks. The first image block feature extraction module is used to extract the first image block features of the first image block located in the defective area of ​​the image to be tested. The first image block fusion module is used to fuse all the first image block features to obtain the fused features as the image features corresponding to the defective area.

[0213] Exemplarily, the actual defect type determination module 740 may include: a first aggregated feature matching module and an actual defect type determination module.

[0214] The first aggregate feature matching module is used to match the first aggregate feature with a preset feature in a feature library, and determine the defect type corresponding to the first aggregate feature based on the defect types corresponding to the matched multiple preset features, wherein the feature library stores multiple preset features and the defect type corresponding to each preset feature. The actual defect type determination module is used to determine the actual defect type of the object to be tested based on the defect type corresponding to the first aggregate feature.

[0215] Exemplarily, the detection device 700 may further include: an initial image defect determination module, a second image block generation module, a second image block feature fusion module, a second aggregate feature generation module and a second aggregate feature clustering module.

[0216] The initial image defect determination module is used to determine a defect area in each initial image, wherein the initial image corresponds to a defect type.

[0217] The second image block generation module is used for dividing each initial image into a plurality of second image blocks, and generating second image block features of the second image blocks located in the defective area of ​​the initial image.

[0218] The second image block feature fusion module is used to perform feature fusion on all second image block features of each initial image to obtain a second fused feature of the initial image.

[0219] The second aggregate feature generation module is used for inputting the second fusion feature of each initial image into the untrained aggregate projection module to obtain the second aggregate feature of the initial image.

[0220] The second aggregate feature clustering module is used to cluster the second aggregate features of all initial images corresponding to each defect type, and use the cluster center feature obtained by clustering as one of the preset features in the feature library, and use the defect type as the defect type corresponding to the cluster center feature.

[0221] Exemplarily, the detection device 700 may further include: a training image defect determination module, a third image block generation module, a third image block feature fusion module, a third aggregate feature generation module, a third aggregate feature matching module and a feature library first update module.

[0222] The training image defect determination module is used to determine defect areas in the training image.

[0223] The third image block generation module is used to divide the training image into a plurality of third image blocks, and generate third image block features of the third image blocks located in the defect area of ​​the training image as the first defect features.

[0224] The third image block feature fusion module is used to perform feature fusion on all third image blocks to obtain a third fusion feature.

[0225] The third aggregate feature generation module is used to input the third fusion feature into the untrained aggregate projection module to obtain the third aggregate feature.

[0226] The third aggregate feature matching module is used to match the third aggregate feature with the preset features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the matched multiple preset features.

[0227] The feature library first updating module is used to update the preset feature in the feature library based on the third aggregate feature for each preset feature matching the third aggregate feature.

[0228] Exemplarily, the third aggregate feature matching module may include at least one of the following: a third aggregate feature first matching module, a third aggregate feature second matching module, and a third aggregate feature third matching module.

[0229] The third aggregate feature first matching module is used to match the third aggregate feature with all preset features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the difference between each matched preset feature and the third aggregate feature.

[0230] The third aggregate feature second matching module is used to match the third aggregate feature with all the first features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the differences between each matched first feature and the third aggregate feature, wherein the first feature is a preset feature whose defect type is the same as the defect type corresponding to the training image.

[0231] The third matching module of the third aggregate feature is used to match the third aggregate feature with all the second features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the difference between each matched second feature and the third aggregate feature, wherein the second feature is a preset feature whose defect type is different from the defect type corresponding to the training image.

[0232] Exemplarily, the detection device 700 may further include: a second global feature generation module and a fourth aggregate feature generation module.

[0233] The second global feature generation module is used to generate a second global feature of the image to be tested, wherein the second global feature is an image feature of the entire image to be tested.

[0234] The fourth aggregate feature generation module is used to input the second global feature into the trained aggregate projection module to obtain the fourth aggregate feature.

[0235] The actual defect type determination module 740 may further include an actual defect type first determination module.

[0236] The first actual defect type determination module is used to determine the actual defect type of the object to be tested based on the first aggregate feature and the fourth aggregate feature.

[0237] Exemplarily, the actual defect type first determination module may include: a fourth aggregate feature matching module and an actual defect type first determination submodule.

[0238] The fourth aggregate feature matching module is used to match the fourth aggregate feature with the preset features in the feature library, and determine the defect type corresponding to the fourth aggregate feature based on the defect types corresponding to the matched multiple preset features. The actual defect type first determination submodule is used to determine the actual defect type of the object to be tested based on the defect type corresponding to the first aggregate feature and the defect type corresponding to the fourth aggregate feature.

[0239] Exemplarily, the detection device 700 may further include: a third global feature generation module, a fifth aggregate feature generation module and a fifth aggregate feature clustering module.

[0240] The third global feature generation module is used to generate the third global feature of each initial image, wherein the initial image corresponds to a defect type and the third global feature is the image feature of the initial image as a whole. The fifth aggregate feature generation module is used to input the third global feature of each initial image into the untrained aggregate projection module to obtain the fifth aggregate feature of the initial image. The fifth aggregate feature clustering module is used to cluster the fifth aggregate features of all initial images corresponding to each defect type, and use the cluster center feature obtained by clustering as one of the preset features in the feature library, and use the defect type as the defect type corresponding to the cluster center feature.

[0241] Exemplarily, the detection device 700 may further include: a fourth global feature generation module, a sixth aggregate feature generation module, a sixth aggregate feature matching module and a feature library second update module.

[0242] The fourth global feature generation module is used to generate the fourth global feature of the training image, wherein the fourth global feature is the image feature of the training image as a whole. The sixth aggregate feature generation module is used to input the fourth global feature into the untrained aggregate projection module to obtain the sixth aggregate feature. The sixth aggregate feature matching module is used to match the sixth aggregate feature with the preset features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the matched multiple preset features. The second feature library update module is used to update the preset feature in the feature library based on the sixth aggregate feature for each preset feature that matches the sixth aggregate feature.

[0243] Exemplarily, the sixth aggregate feature matching module may include at least one of the following: a sixth aggregate feature first matching module, a sixth aggregate feature second matching module, and a sixth aggregate feature third matching module.

[0244] The sixth aggregate feature first matching module is used to match the sixth aggregate feature with all preset features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the difference between each matched preset feature and the sixth aggregate feature.

[0245] The second matching module of the sixth aggregate feature is used to match the sixth aggregate feature with all the third features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the differences between each matched third feature and the sixth aggregate feature, wherein the third feature is a preset feature whose defect type is the same as the defect type corresponding to the training image.

[0246] The third matching module of the sixth aggregate feature is used to match the sixth aggregate feature with all the fourth features in the feature library, and adjust the module parameters of the untrained aggregate projection module based on the differences between each matched fourth feature and the sixth aggregate feature, wherein the fourth feature is a preset feature whose defect type is different from the defect type corresponding to the training image.

[0247] According to another aspect of the present invention, an electronic device is also provided. Figure 5 FIG. 8 is a schematic block diagram of an electronic device 800 according to an embodiment of the present invention. Figure 5 As shown, the electronic device 800 includes a processor 810 and a memory 820. The memory 820 stores a computer program. When the computer program instructions are executed by the processor 810, they are used to execute the above-mentioned defect type detection method.

[0248] In addition, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned defect type detection method according to the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned defect type detection device according to the embodiment of the present invention or the corresponding modules in the above-mentioned electronic device. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above-mentioned storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media. According to another aspect of the present invention, a computer program product is provided, including computer program instructions, and when the above-mentioned computer program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned defect type detection method.

[0249] A person skilled in the art may understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the above-mentioned description of the detection method of the defect type, and for the sake of brevity, it will not be described here in detail.

[0250] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0251] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0252] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0253] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0254] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0255] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0256] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0257] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the defect type detection device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0258] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0259] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A defect type detection method, characterized in that: The method comprises: Determining a defective area in the image to be tested of the object to be tested; Extracting image features corresponding to defective areas in the image to be tested; Based on the image features corresponding to the defective area in the image to be tested, a first aggregate feature is generated by using a trained aggregate projection module, wherein the trained aggregate projection module is trained based on a first global feature and a first defect feature of the training image of the object to be tested, the first global feature is the image feature of the training image as a whole, and the first defect feature is the image feature corresponding to the defective area in the training image; Determining an actual defect type of the object to be tested based on the first aggregated feature; Wherein, the method further comprises: Generate a second global feature of the image to be tested, wherein the second global feature is an image feature of the entire image to be tested; Inputting the second global feature into the trained aggregation projection module to obtain a fourth aggregation feature; The determining, based on the first aggregated feature, an actual defect type of the object to be tested includes: An actual defect type of the object to be tested is determined based on the first aggregate feature and the fourth aggregate feature.

2. The method according to claim 1, characterized in that The extracting of image features corresponding to the defective area in the image to be tested includes: Dividing the image to be tested into a plurality of first image blocks; Extracting a first image block feature of a first image block located in a defect area in the image to be tested; All first image block features are fused to obtain fused features as image features corresponding to the defect area.

3. The method according to claim 1, characterized in that The determining, based on the first aggregated feature, an actual defect type of the object to be tested includes: Matching the first aggregate feature with a preset feature in a feature library, and determining a defect type corresponding to the first aggregate feature based on defect types corresponding to each of the matched multiple preset features, wherein the feature library stores multiple preset features and a defect type corresponding to each preset feature; Based on the defect type corresponding to the first aggregated feature, an actual defect type of the object to be tested is determined.

4. The method according to claim 3, characterized in that The method further comprises: For each initial image, Determining a defect area in the initial image, wherein the initial image is an image of a reference object and the initial image corresponds to a defect type; Dividing the initial image into a plurality of second image blocks, and generating second image block features of the second image blocks located in the defective area of ​​the initial image; Performing feature fusion on all second image block features of the initial image to obtain second fused features of the initial image; Inputting the second fused features of the initial image into an untrained aggregation projection module to obtain a second aggregation feature of the initial image; For each defect type, the second aggregation features of all initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature.

5. The method according to claim 3, characterized in that The method further comprises: Determining defect areas in training images; Dividing the training image into a plurality of third image blocks, and generating third image block features of the third image blocks located in the defect area of ​​the training image as the first defect features; Performing feature fusion on all third image blocks to obtain a third fusion feature; Inputting the third fusion feature into an untrained aggregation projection module to obtain a third aggregation feature; Matching the third aggregate feature with a preset feature in a feature library, and adjusting module parameters of the untrained aggregate projection module based on the matched multiple preset features; For each preset feature that matches the third aggregate feature, the preset feature in the feature library is updated based on the third aggregate feature.

6. The method according to claim 5, characterized in that The step of matching the third aggregate feature with a preset feature in a feature library, and adjusting module parameters of the untrained aggregate projection module based on the matched multiple preset features, includes at least one of the following: Matching the third aggregate feature with all preset features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched preset feature and the third aggregate feature; Matching the third aggregate feature with all first features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched first feature and the third aggregate feature, wherein the first feature is a preset feature with the same defect type as the defect type corresponding to the training image; The third aggregate feature is matched with all second features in the feature library, and based on the difference between each matched second feature and the third aggregate feature, the module parameters of the untrained aggregate projection module are adjusted, wherein the second feature is a preset feature with a defect type different from the defect type corresponding to the training image.

7. The method according to claim 1, characterized in that The determining, based on the first aggregate feature and the fourth aggregate feature, an actual defect type of the object to be tested includes: Matching the fourth aggregate feature with a preset feature in a feature library, and determining the defect type corresponding to the fourth aggregate feature based on defect types corresponding to each of the matched preset features; Based on the defect type corresponding to the first aggregate feature and the defect type corresponding to the fourth aggregate feature, an actual defect type of the object to be tested is determined.

8. The method according to claim 7, characterized in that The method further comprises: For each initial image, Generate a third global feature of the initial image, wherein the initial image is an image of a reference object, and the initial image corresponds to a defect type, and the third global feature is an image feature of the entire initial image; Inputting the third global feature of the initial image into an untrained aggregation projection module to obtain a fifth aggregation feature of the initial image; For each defect type, the fifth aggregation features of all initial images corresponding to the defect type are clustered, and the cluster center feature obtained by clustering is used as one of the preset features in the feature library, and the defect type is used as the defect type corresponding to the cluster center feature.

9. The method according to claim 7, characterized in that The method further comprises: generating a fourth global feature of the training image, wherein the fourth global feature is an image feature of the entire training image; Inputting the fourth global feature into an untrained aggregation projection module to obtain a sixth aggregation feature; Matching the sixth aggregate feature with preset features in a feature library, and adjusting module parameters of the untrained aggregate projection module based on the matched multiple preset features; For each preset feature matching the sixth aggregate feature, the preset feature in the feature library is updated based on the sixth aggregate feature.

10. The method according to claim 9, characterized in that The matching of the sixth aggregate feature with a preset feature in a feature library, and adjusting the module parameters of the untrained aggregate projection module based on the matched multiple preset features, includes at least one of the following: Matching the sixth aggregate feature with all preset features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched preset feature and the sixth aggregate feature; Matching the sixth aggregate feature with all third features in the feature library, and adjusting the module parameters of the untrained aggregate projection module based on the difference between each matched third feature and the sixth aggregate feature, wherein the third feature is a preset feature whose defect type is the same as the defect type corresponding to the training image; The sixth aggregate feature is matched with all fourth features in the feature library, and based on the difference between each matched fourth feature and the sixth aggregate feature, the module parameters of the untrained aggregate projection module are adjusted, wherein the fourth feature is a preset feature whose defect type is different from the defect type corresponding to the training image.

11. A defect type detection device, characterized in that: The device comprises: A defect area determination module, used to determine the defect area in the image to be tested of the object to be tested; An image feature extraction module, used to extract image features corresponding to defective areas in the image to be tested; A first aggregate feature generation module, configured to generate a first aggregate feature based on the image feature corresponding to the defective area in the image to be tested and using the trained aggregate projection module, wherein the trained aggregate projection module is trained based on the first global feature and the first defect feature of the training image of the object to be tested, the first global feature is the image feature of the training image as a whole, and the first defect feature is the image feature corresponding to the defective area in the training image; An actual defect type determination module, used to determine the actual defect type of the object to be tested based on the first aggregated feature; Wherein, the device further comprises: a second global feature generation module and a fourth aggregate feature generation module; The second global feature generation module is used to generate a second global feature of the image to be tested, wherein the second global feature is an image feature of the entire image to be tested; The fourth aggregate feature generation module is used to input the second global feature into the trained aggregate projection module to obtain a fourth aggregate feature; The actual defect type determination module includes an actual defect type first determination module; The actual defect type first determination module is used to determine the actual defect type of the object to be tested based on the first aggregate feature and the fourth aggregate feature.

12. An electronic device, characterized in that: It comprises a memory and a processor, wherein: the memory is used to store a computer program; the processor is used to execute the computer program to implement the defect type detection method according to any one of claims 1 to 10.

13. A storage medium storing a computer program / instruction, characterized in that: The computer program / instructions are used to execute the defect type detection method according to any one of claims 1 to 10 when running.

14. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the defect type detection method according to any one of claims 1 to 10 when running.

Images