Defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a defect detection method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of a to-be-detected image of a to-be-detected wafer particle, so as to obtain the global image feature of the to-be-detected image; for each preset text, performing feature extraction on the preset text to obtain a preset text feature of the preset text, and performing similarity calculation on the preset text feature of the preset text and the global image feature to obtain a similarity between the preset text feature and the global image feature; and based on the similarity between each preset text feature and the global image feature, determining a preset text matched with the to-be-detected image, and taking the preset text as a text description related to defects existing in the to-be-detected wafer grain. According to the scheme, the text description of the defects is more flexible, and the method can be adapted to actual scenes with complex and changeable defect types in wafer grain detection.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to a defect detection method, a defect detection device, an electronic device, a storage medium and a computer program product. Background Art

[0002] In the industrial field, with the continuous development of production technology, people have put forward higher requirements for the production quality and efficiency of industrial products. With the development of image processing technology, more and more product inspections are based on images. If the inspection accuracy is low, the practicality of the image-based inspection method will also be reduced.

[0003] For example, in the process of wafer inspection, the related technology usually uses automated optical inspection (AOI) equipment to perform appearance inspection on the wafer to determine the type of defects on the wafer. However, the different defects on the wafer are highly random and there are many types of defects. Therefore, how to accurately detect the defect type is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0004] The present invention is proposed in view of the above problems. The present invention provides a defect detection method, a defect detection device, an electronic device, a storage medium and a computer program product.

[0005] According to one aspect of the present invention, a defect detection method is provided, the detection method comprising:

[0006] Performing feature extraction on the image to be tested of the wafer to be tested to obtain global image features of the image to be tested;

[0007] For each preset text, feature extraction is performed on the preset text to obtain a preset text feature of the preset text, and similarity calculation is performed between the preset text feature of the preset text and the global image feature to obtain the similarity between the preset text feature and the global image feature, wherein the preset text is a text description related to a defect existing in a reference wafer grain;

[0008] Based on the similarity between each preset text feature and the global image feature, a preset text matching the image to be tested is determined to serve as a text description related to the defect existing in the wafer to be tested.

[0009] Exemplarily, feature extraction is performed on the image to be tested of the wafer to be tested to obtain global image features of the image to be tested, including:

[0010] The image to be tested is input into the trained image feature extraction module to obtain the global image features of the image to be tested.

[0011] Exemplarily, the trained image feature extraction module includes a plurality of image feature extraction submodules connected in sequence, and the image to be tested is input into the trained image feature extraction module to obtain the global image features of the image to be tested, including:

[0012] For each image feature extraction submodule, the image feature extraction submodule performs feature extraction processing on the first intermediate feature output by the image to be tested or other image feature extraction submodules connected to the image feature extraction submodule to obtain the first intermediate feature output by the image feature extraction submodule, and the first intermediate feature output by the last image feature extraction submodule is used as the global image feature.

[0013] Exemplarily, the detection method further comprises:

[0014] Preset text matching the image to be tested,

[0015] Calculate the similarity between the preset text feature of the preset text and each feature point in each first intermediate feature to obtain a first similarity matrix between the preset text and each first intermediate feature;

[0016] Fusing all first similarity matrices of the preset text to obtain a fused similarity matrix of the preset text;

[0017] Based on the target position information of each target similarity in the fusion similarity matrix of the preset text, the pixels corresponding to each target position are marked in the image to be tested as part of the defect area of ​​the image to be tested, wherein the target similarity is greater than the preset similarity.

[0018] Exemplarily, the detection method further comprises:

[0019] For each preset text,

[0020] Calculate the similarity between the preset text feature of the preset text and each feature point in each first intermediate feature to obtain a second similarity matrix between the preset text and each first intermediate feature;

[0021] Fusing all the second similarity matrices of the preset text to obtain a fused similarity matrix of the preset text;

[0022] Based on the similarity between each preset text feature and the global image feature, a preset text matching the image to be tested is determined as a text description related to the defect in the wafer to be tested, including:

[0023] For each preset text, based on the similarity between the preset text feature and the global image feature of the preset text and the fusion similarity matrix of the preset text, determine the matching degree between the preset text and the image to be tested;

[0024] Based on the matching degree between each preset text and the image to be tested, the preset text matching the image to be tested is determined as a text description related to the defect existing in the wafer to be tested.

[0025] Exemplarily, the trained image feature extraction module is obtained by at least the following steps:

[0026] Inputting a training image of a training wafer into an untrained image feature extraction module to obtain a global image feature of the training image, wherein the training image corresponds to a training text, which is a text description related to defects in the training wafer;

[0027] Inputting the training text corresponding to the training image into the text feature extraction module to obtain the training text features corresponding to the training image;

[0028] Based on the global image features and training text features of the training images, the predicted similarity is determined, and based on the difference between the predicted similarity and the preset highest similarity, the module parameters of the untrained image feature extraction module are adjusted.

[0029] Exemplarily, the untrained image feature extraction module includes a plurality of image feature extraction submodules connected in sequence, and the training image of the training wafer is input into the untrained image feature extraction module to obtain the global image features of the training image, including:

[0030] For each untrained image feature extraction submodule, the untrained image feature extraction submodule performs feature extraction processing on the second intermediate features output by the training image or other untrained image feature extraction submodules connected to the untrained image feature extraction submodule to obtain the second intermediate features output by the untrained image feature extraction submodule, and the second intermediate features output by the last untrained image feature extraction submodule are used as the global image features of the training image.

[0031] Exemplarily, based on the difference between the predicted similarity and the preset highest similarity, the module parameters of the untrained image feature extraction module are adjusted by the following steps:

[0032] Based on the training image, a reference similarity matrix is ​​generated, wherein the reference similarity corresponding to the defective area in the reference similarity matrix is ​​a preset highest similarity, and the reference similarities corresponding to other areas are a preset lowest similarity;

[0033] Calculate the similarity between the training text feature and each feature point in each second intermediate feature to obtain a third similarity matrix between the training text and each second intermediate feature;

[0034] Fusing all third similarity matrices of the training text to obtain a fused similarity matrix of the training image;

[0035] The parameters are adjusted based on the difference between the predicted similarity and the preset highest similarity and the difference between the fused similarity matrix of the training image and the benchmark similarity matrix.

[0036] Exemplarily, the preset text includes: a text description related to whether there is a defect in the reference wafer die, a text description related to a region in which a defect exists in the reference wafer die, and a text description related to a type of defect existing in the reference wafer die.

[0037] According to yet another aspect of the present invention, a defect detection device is provided. The defect detection device includes: a first feature extraction module, a matching module and a defect description module.

[0038] A first feature extraction module is used to extract features from the image to be tested of the wafer to be tested, so as to obtain global image features of the image to be tested;

[0039] A matching module is used for performing feature extraction on each preset text to obtain a preset text feature of the preset text, and performing similarity calculation between the preset text feature of the preset text and a global image feature to obtain a similarity between the preset text feature and the global image feature, wherein the preset text is a text description related to a defect existing in a reference wafer grain;

[0040] The defect description module is used to determine the preset text matching the image to be tested based on the similarity between each preset text feature and the global image feature, so as to serve as a text description related to the defects existing in the wafer grain to be tested.

[0041] According to another aspect of the present invention, an electronic device is provided, which includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect detection method when the processor is running.

[0042] According to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned defect detection method when running.

[0043] According to another aspect of the present invention, a computer program product is also provided, wherein the computer program product comprises computer program instructions, and the computer program instructions are used to execute the above-mentioned defect detection method when running.

[0044] According to the above scheme of the embodiment of the present invention, feature extraction can be performed on the image to be tested of the wafer grain to be tested to obtain the global image feature of the image to be tested. Then, for each preset text, feature extraction is performed on the preset text to obtain the preset text feature of the preset text, and the preset text feature of the preset text is calculated with the global image feature to obtain the similarity between the preset text feature and the global image feature. Finally, based on the similarity between each preset text feature and the global image feature, the preset text matching the image to be tested is determined as a text description related to the defects existing in the wafer grain to be tested. Compared with the method of obtaining defect labels using a defect classification model, the above scheme can use preset texts to provide a text description of the defects existing in the wafer grain to be tested. Therefore, the text description of defects in the above scheme is more flexible and can be adapted to actual scenarios with complex and changeable defect categories in wafer grain detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0046] Figure 1 A schematic flow chart of a defect detection method according to an embodiment of the present invention is shown;

[0047] Figure 2 A schematic diagram showing defects of a wafer die according to an embodiment of the present invention is shown;

[0048] Figure 3 A schematic diagram showing a training process of an image feature extraction module according to an embodiment of the present invention;

[0049] Figure 4 A schematic block diagram showing a defect detection device according to an embodiment of the present invention; and

[0050] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0052] In the field of component manufacturing, if defective industrial components (such as wafers) are packaged and tested, it will waste manpower and material resources, so it is necessary to inspect the industrial components so that unqualified industrial components can be detected and eliminated. However, there may be a variety of defects on industrial components. For example, taking wafers as an example, there may be defects such as foreign matter and cracks on the wafer. For another example, taking the wafer grains on the wafer as an example, there may be defects such as pin prints out of bounds, too many pin prints, and foreign matter on the wafer grains. However, the defect types that can be output by the defect detection model in the prior art are limited, which is not conducive to adapting to the actual scenario where there are complex and numerous defects in wafers or wafer grains.

[0053] In order to at least partially solve the above problem, an embodiment of the present invention provides a defect detection method. Figure 1 FIG. 1 is a schematic flow chart of a defect detection method according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps S110 to S130.

[0054] In step S110 , feature extraction is performed on the image to be tested of the wafer to be tested to obtain global image features of the image to be tested.

[0055] The image to be tested of the wafer to be tested may be a static image of the wafer to be tested, or any video frame in a dynamic video of the wafer to be tested. The image to be tested of the wafer to be tested may be an original image of the wafer to be tested acquired by an image acquisition device (for example, an original image acquired by an image sensor in a camera), or an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.).

[0056] The above-mentioned feature extraction operation can be obtained by processing the image to be tested of the wafer grain to be tested through a feature extraction algorithm or a feature extraction module. For example, the image to be tested can be processed through feature extraction algorithms such as principal component analysis and linear discriminant analysis to obtain the global image features of the image to be tested. For another example, the image to be tested of the wafer grain to be tested can be processed through feature extraction modules such as a feature extraction layer in a convolutional neural network and a feature extraction layer in a recurrent neural network to obtain the global image features of the image to be tested.

[0057] In step S120, for each preset text, feature extraction is performed on the preset text to obtain a preset text feature of the preset text, and similarity calculation is performed between the preset text feature of the preset text and the global image feature to obtain the similarity between the preset text feature and the global image feature.

[0058] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram showing a defect of a wafer die according to an embodiment of the present invention. Figure 2 , defect 1 may be the presence of foreign matter in the wafer grain, and defect 2 may be the presence of pin print out of bounds in the wafer grain. The preset text is a text description related to the defects present in the reference wafer grain. The preset text may include any text used to describe defects that may exist on the reference wafer grain. For example, the preset text may include text indicating the presence of foreign matter in the pad area, text indicating that the pin print in the pad area is out of bounds, text indicating that there are too many pin prints in the pad area, text indicating that there are scratches in the pad area, text indicating that there are foreign matter in the surface area, text indicating that there are scratches in the surface area, text indicating that there are foreign matter in the trimming area, text indicating that there are scratches in the trimming area, etc. Among them, there may be multiple reference wafer grains with different defects, and then a preset text for describing the defect is generated for each reference wafer grain. It should be understood that the preset text may also include text for describing the absence of defects on the reference wafer grain.

[0059] The preset text feature of the preset text is obtained by performing a feature extraction operation on the preset text. For example, the preset text can be input into a text encoder in a language model to obtain a text embedding of the preset text and use the text embedding as the preset text feature.

[0060] For each preset text, the global image feature and the preset text feature of the preset text can be calculated to obtain the similarity between the global image feature and the preset text feature. Accordingly, the similarity between the global image feature and the preset text feature of each preset text can be calculated, thereby obtaining the similarity between the global image feature and the preset text feature of each preset text. Taking the calculation of the similarity between the global image feature and the preset text feature of the preset text T1 as an example, for each feature point in the global image feature, the similarity between the feature point and the preset text feature of the preset text T1 can be calculated to obtain the similarity between the feature point and the preset text feature of the preset text T1. Accordingly, the similarity corresponding to each feature point in the global image feature can be obtained to form a similarity matrix. Similarly, the similarity matrix between the global image feature and the preset text features of the remaining preset texts is calculated to obtain the similarity matrix between the global image feature and the preset text features of each preset text. It should be understood that the similarity matrix can adapt to the dimension of the global image feature. For example, if the global image feature is converted into a one-dimensional vector, the similarity matrix can be converted into a one-dimensional arrangement of similarity. If the global image feature is a two-dimensional feature map, the similarity matrix can also be a two-dimensional arrangement of similarity. For each preset text, the highest similarity in the similarity matrix between the preset text feature of the preset text and the global image feature can be used as the similarity between the global image feature and the preset text feature. The above similarity can be obtained by calculating the Euclidean distance, calculating the cosine similarity, etc., and the embodiment of the present invention is not limited here. It should be understood that the similarity calculation operation can also be performed on the global image feature and the preset text feature of the preset text by methods such as attention mechanism and contrast learning, and the embodiment of the present invention is not limited here. In some embodiments, the similarity between the global image feature and the preset text feature can also be determined by deep learning, multimodal contrast learning, etc.

[0061] In step S130 , based on the similarity between each preset text feature and the global image feature, a preset text matching the image to be tested is determined as a text description related to the defect existing in the wafer to be tested.

[0062] The preset text corresponding to the preset text feature with the highest similarity to the global image feature can be used as the preset text matched with the image to be tested. The preset text matched with the image to be tested is used as a text description related to the defects existing in the wafer grain to be tested. For example, taking the similarity matrix corresponding to one between the global image feature and the preset text feature of each preset text in step S120 as an example, the global image feature corresponds to the preset text feature of the preset text T1, the global image feature corresponds to the preset text feature of the preset text T2, and the global image feature corresponds to the preset text feature of the preset text T3. The maximum value in the similarity matrix S1 is 0.8, the maximum value in the similarity matrix S2 is 0.7, and the maximum value in the similarity matrix S3 is 0.6. Then the preset text T1 corresponding to the similarity matrix S1 is the preset text matched with the image to be tested, and the preset text T1 can be used as the preset text matched with the image to be tested. In the case where the preset text T1 indicates that the pin print of the pad area is out of bounds, the text description related to the defects existing in the wafer grain to be tested indicates that the pin print of the pad area is out of bounds. In another example, for each preset text feature, the average value of multiple representative similarities in the similarity matrix corresponding to the preset text feature can also be calculated, and the average value is used as the similarity between the preset text and the global image feature. Then the preset text matching the image to be tested is determined. It should be understood that the preset text with the highest similarity and a preset number can also be used as the relevant text description of the defects that may exist in the wafer grain to be tested, and then the actual defects existing in the wafer grain to be tested are determined by manual or other defect detection methods.

[0063] According to the above scheme of the embodiment of the present invention, feature extraction can be performed on the image to be tested of the wafer grain to be tested to obtain the global image feature of the image to be tested. Then, for each preset text, feature extraction is performed on the preset text to obtain the preset text feature of the preset text, and the preset text feature of the preset text is calculated with the global image feature to obtain the similarity between the preset text feature and the global image feature. Finally, based on the similarity between each preset text feature and the global image feature, the preset text matching the image to be tested is determined as a text description related to the defects existing in the wafer grain to be tested. Compared with the method of obtaining defect labels using a defect classification model, the above scheme can use preset texts to provide a text description of the defects existing in the wafer grain to be tested. Therefore, the text description of defects in the above scheme is more flexible and can be adapted to actual scenarios with complex and changeable defect categories in wafer grain detection.

[0064] Exemplarily, the preset text includes: a text description related to whether there is a defect in the reference wafer die, a text description related to a region in which a defect exists in the reference wafer die, and a text description related to a type of defect existing in the reference wafer die.

[0065] Optionally, the preset text may be "in [x] area, [w] defect, has [y]". Among them, "in [x] area" corresponds to the above text description related to the area where defects exist in the reference wafer grain, "[w] defect" corresponds to the above text description related to whether there are defects in the reference wafer grain, and "has [y]" corresponds to the above text description related to the type of defects existing in the reference wafer grain. For example, the above text description related to the area where defects exist in the reference wafer grain (that is, "in [x] area") may include: "in [pad] area", "in [non-pad] area", "in [trim] area", "in [surface] area", etc. The above text description related to whether there are defects in the reference wafer grain (that is, "[w] defect") may include: "[has] defect", "[no] defect", etc. The above text description related to the type of defects existing in the reference wafer grain (that is, "has [y]") may include: "has [scratches]", "has [transparent foreign matter]", "has [opaque foreign matter]", "has [pin mark]", etc. The present disclosure does not limit the specific contents of the above text descriptions related to the area where defects exist in the reference wafer grain, the text descriptions related to whether there are defects in the reference wafer grain, and the text descriptions related to the types of defects in the reference wafer grain. It should be understood that the preset text may also include at least one of the above text descriptions, or include other text descriptions related to defects, and the embodiments of the present invention are not limited here.

[0066] In one example, relevant content in the preset text may also be added or deleted based on actual needs. For example, the preset text may also include the visual characteristics of the image of the reference wafer grain. Specifically, for example, the preset text may include "In the [non-pad] area, [no] defects" and "This image consists of red and white lines." In the preset text, "In the [non-pad] area" is a text description related to the area where defects exist in the reference wafer grain. "[No] defects" is a text description related to whether there are defects in the reference wafer grain. "This image consists of red and white lines" is a visual feature of the reference wafer grain image.

[0067] According to the above scheme of the embodiment of the present invention, by limiting the preset text, the detection results can include a variety of text descriptions related to the defects in the image to be tested of the wafer grain to be tested, and then the defects of the wafer grain to be tested can be described from different angles, which is conducive to improving the information breadth of the detection results.

[0068] Exemplarily, the above step S110 extracts features from the image to be tested of the wafer to be tested to obtain global image features of the image to be tested, and may include step S111.

[0069] In step S111 , the image to be tested is input into the trained image feature extraction module to obtain the global image features of the image to be tested.

[0070] The image feature extraction module may be an image encoder, for example, a convolutional neural network, a transformer-based encoder, a hybrid model architecture combining a convolutional neural network and a transformer-based encoder, etc. The trained image feature extraction module may be used to extract features from the image to be tested, and the output result of the trained image feature extraction module may be used as the global image feature of the image to be tested.

[0071] According to the above scheme of the embodiment of the present invention, the image to be tested can be input into the trained image feature extraction module to obtain the global image features of the image to be tested. Since the above scheme can obtain the global image features of the image to be tested through the trained image feature extraction module, compared with the method of obtaining the global image features of the image to be tested through an algorithm, the above scheme can enhance the robustness of the global image features.

[0072] Exemplarily, the trained image feature extraction module includes a plurality of image feature extraction sub-modules connected in sequence.

[0073] The above-mentioned image feature extraction submodule can be a feature extraction module in any machine learning model. For example, the image feature extraction submodule can be a convolutional feature extraction module in a convolutional neural network, a cyclic feature extraction module in a cyclic neural network, a graph convolutional feature extraction module in a graph neural network, etc. Here, the image feature extraction submodule is taken as an example of a feature extraction module in a convolutional feature extraction network. The image feature extraction submodule may include a convolutional layer, an activation layer, and a pooling layer connected in sequence. The above-mentioned convolutional layer can convolve the input data of the image feature extraction submodule, and then input it to the activation layer. The activation layer can process the input data after convolution through a preset activation function to obtain activation data, and input it to the pooling layer. The pooling layer can process the activation data through maximum pooling, average pooling, etc., as the output of the image feature extraction submodule.

[0074] In this example, the above step S111 also includes step S1111.

[0075] In step S1111, for each image feature extraction submodule, feature extraction processing is performed on the first intermediate feature output by the image to be tested or other image feature extraction submodules connected to the image feature extraction submodule to obtain the first intermediate feature output by the image feature extraction submodule, and the first intermediate feature output by the last image feature extraction submodule is used as the global image feature.

[0076] Taking the trained image feature extraction module including the image feature extraction submodule block1, the image feature extraction submodule block2 and the image feature extraction submodule block3 as an example, the image to be tested is input into the image feature extraction submodule block1. The image feature extraction submodule block1 performs a feature extraction operation on the image to be tested, and then generates and outputs the first intermediate feature f1 to the image feature extraction submodule block2. Reference can be made to the above description of performing feature extraction processing on the image to be tested by the image feature extraction submodule. The image feature extraction submodule block2 performs a feature extraction operation on the first intermediate feature f1, and then generates and outputs the first intermediate feature f2 to the image feature submodule block3. Reference can be made to the above description of performing feature extraction processing on the first intermediate feature output by other image feature extraction submodules connected to the image feature extraction submodule by the image feature extraction submodule. The image feature extraction submodule block3 performs a feature extraction operation on the first intermediate feature f2, and then generates and outputs the first intermediate feature f3. The first intermediate feature f3 output by the image feature extraction submodule block3 can be used as a global image feature. The first intermediate feature output by the last image feature extraction submodule may be used as the global image feature in the above text. It should be understood that the feature sizes of the first intermediate features output by different image feature extraction submodules may be different. Therefore, different first intermediate features may be converted into first intermediate features with the same feature size to facilitate subsequent calculations.

[0077] According to the above scheme of the embodiment of the present invention, the image to be tested can be input into the image feature extraction submodules connected in sequence, so that the last image feature extraction submodule outputs the global image feature. In the above scheme, multiple image feature extraction submodules are used to extract features from the image to be tested, so that the feature level of the obtained global image feature can be richer, thereby improving the representativeness of the global image feature, which is conducive to improving the accuracy of the detection result.

[0078] Exemplarily, the above defect detection method may include steps S210 to S230.

[0079] In step S210, for the preset text matched with the image to be tested, similarity is calculated between the preset text features of the preset text and each feature point in each first intermediate feature, so as to obtain a first similarity matrix between the preset text and each first intermediate feature.

[0080] The preset text features of the preset text can be respectively calculated for similarity with each feature point in the first intermediate feature output by each image feature extraction submodule. For example, taking the image feature extraction submodule block1, the image feature extraction submodule block2 and the image feature extraction submodule block3 as examples, the preset text features of the preset text are calculated for similarity with each feature point in the first intermediate feature f1 output by the image feature extraction submodule block1 to obtain a first similarity matrix S10. The preset text features of the preset text are calculated for similarity with each feature point in the first intermediate feature f2 output by the image feature extraction submodule block2 to obtain a first similarity matrix S11. The preset text features of the preset text are calculated for similarity with each feature point in the first intermediate feature f3 output by the image feature extraction submodule block3 to obtain a first similarity matrix S12.

[0081] The above similarity calculation can be obtained by calculating Euclidean distance, cosine similarity, etc. The above scheme can calculate the similarity between each feature point in the first intermediate feature and the preset text feature, obtain the similarity corresponding to each feature point in the first intermediate feature, and then obtain the first similarity matrix of the first intermediate feature. In this way, it is determined whether it matches the preset text feature. If there is a similarity with a higher value in the first similarity matrix, it can be proved that the feature point corresponding to the similarity is more consistent with the semantics of the preset text. Combined with actual scenarios, feature points corresponding to similarities with higher values ​​can be regarded as feature points that may have defects.

[0082] In one example, each first intermediate feature can be directly subjected to a similarity calculation operation with a preset text feature. If the feature size of each first intermediate feature is the same, each first intermediate feature can be directly subjected to a similarity calculation operation with a preset text feature. The similarity calculation result between the first intermediate feature and the preset text feature is used as the first similarity matrix of the first intermediate feature. The similarity calculation results thus obtained are of the same size, which facilitates subsequent operations.

[0083] In another example, at least one first intermediate feature may also be subjected to operations related to other features to meet actual needs. If there is a first intermediate feature whose feature size is different from that of other first intermediate features, each first intermediate feature may be converted into an image feature of a preset size, and then a similarity calculation operation may be performed with the preset text feature. The similarity calculation result between the first intermediate feature and the preset text feature is used as the first similarity matrix of the first intermediate feature. If the feature size of the first intermediate feature changes, the feature points in the first intermediate feature may be added or deleted. Specifically, if feature points need to be added, the first intermediate feature may be processed by an interpolation algorithm in the relevant technology.

[0084] In another example, a plurality of first intermediate features may be respectively subjected to similarity calculation operations with preset text features, and then the similarity calculation results may be converted to a preset size, which is not limited in the embodiment of the present invention. The similarity calculation results converted to the preset size are used as the first similarity matrix of the first intermediate feature. If the size of the first similarity matrix changes, the data in the first similarity matrix may be added or deleted. Specifically, if it is necessary to increase the data in the first similarity matrix, the first similarity matrix may be processed by the above-mentioned interpolation algorithm.

[0085] In one example, the similarity matrix obtained by the first intermediate feature can be directly used as the first similarity matrix. In another example, for each first intermediate feature, the similarity corresponding to the feature point therein can be used as a weight to perform an operation with the feature of the feature point to obtain a new feature of the feature point and use it as part of the first similarity matrix corresponding to the first intermediate feature.

[0086] In step S220, for the preset text matched by the image to be tested, all first similarity matrices of the preset text are fused to obtain a fused similarity matrix of the preset text.

[0087] All first similarity matrices may be fused using methods such as linear averaging and rule-based fusion in the related art to obtain a fused similarity matrix of the preset text. It should be understood that the value range of each similarity in the fused similarity matrix may be 0 to 1. If it exceeds this range after fusion, it may be normalized, and the embodiments of the present invention will not be described in detail here.

[0088] In step S230, for the preset text matched with the image to be tested, based on the target position information of each target similarity in the fusion similarity matrix of the preset text, the pixels corresponding to each target position in the image to be tested are marked as part of the defect area of ​​the image to be tested.

[0089] The target similarity is greater than the preset similarity. For example, the fused similarity matrix contains similarity a, similarity b, similarity c, similarity d, similarity e, similarity f, similarity g and similarity h. Among them, if the similarity of similarity b, similarity c and similarity d is greater than the preset similarity, similarity b, similarity c and similarity d are used as target similarities. Determine the target position information of each target similarity in the fused similarity matrix. According to the correspondence between the position information of each similarity in the fused similarity matrix and the position information of each feature point in the first intermediate feature of the image to be tested, determine the feature points corresponding to each target similarity. The feature point may represent a feature point in the first intermediate feature of the image to be tested that conforms to the semantics of the preset text. According to the semantics of the preset text, the above feature point may be regarded as a feature point that may have defects. The pixels corresponding to the above feature points in the image to be tested may be marked as part of the defect area of ​​the image to be tested.

[0090] According to the above scheme of the embodiment of the present invention, the similarity between the features of each feature point in the first intermediate feature and the preset text feature can be determined for each first intermediate feature output by each image feature extraction submodule, so as to obtain a first similarity matrix between the first intermediate feature and the preset text. Then, all the first similarity matrices are fused to obtain a fused similarity matrix of the preset text. Finally, based on the target position information of each target similarity in the fused similarity matrix of the preset text, the pixels corresponding to each target position are marked in the image to be tested as part of the defect area of ​​the image to be tested. The above scheme can improve the information breadth of defect detection by determining the defect area of ​​the image to be tested, which is conducive to improving the comprehensiveness of defect detection.

[0091] Exemplarily, the above defect detection method further includes steps S310 to S320.

[0092] In step S310, for each preset text, similarity is calculated between the preset text feature of the preset text and each feature point in each first intermediate feature, so as to obtain a second similarity matrix between the preset text and each first intermediate feature.

[0093] The preset text features of the preset text can be used to calculate the similarity with each feature point in the first intermediate features output by each image feature extraction submodule. The generation process of the first similarity matrix mentioned above can be referred to, and the embodiment of the present invention will not be described in detail here.

[0094] The above scheme can calculate the similarity between each feature point in the first intermediate feature and the preset text feature, obtain the similarity corresponding to each feature point in the first intermediate feature, and then obtain the second similarity matrix of the first intermediate feature. In this way, it is determined whether it matches the preset text feature. If there is a similarity with a higher value in the second similarity matrix, it can be proved that the feature point corresponding to the similarity is more consistent with the semantics of the preset text. Combined with the actual scene, the feature points corresponding to the similarity with a higher value can be regarded as feature points that may have defects. It should be understood that the feature sizes of the first intermediate features output by different image feature extraction submodules may be different. Through the relevant process of step S210 above, multiple first intermediate features can be converted into first intermediate features with the same feature size to facilitate subsequent calculations.

[0095] In step S320, for each preset text, all second similarity matrices of the preset text are fused to obtain a fused similarity matrix of the preset text.

[0096] All the second similarity matrices may be fused using methods such as linear averaging and rule-based fusion in related technologies to obtain a fused similarity matrix of the preset text.

[0097] In this example, step S130 determines the preset text matching the image to be tested based on the similarity between each preset text feature and the global image feature, as a text description related to the defects in the wafer to be tested, and may include steps S131 to S132.

[0098] In step S131, for each preset text, the matching degree between the preset text and the image to be tested is determined based on the similarity between the preset text feature and the global image feature of the preset text and the fusion similarity matrix of the preset text.

[0099] The degree of matching between the preset text and the image to be tested can be determined based on the similarity between the preset text features and the global image features of the preset text and the fused similarity matrix of the preset text. For example, the degree of matching between the preset text and the image to be tested is determined based on the maximum similarity value in the fused similarity matrix of the preset text and the similarity between the preset text features and the global image features. For example, the maximum similarity value in the fused similarity matrix of the preset text is 0.8, and the similarity between the preset text features and the global image features is 0.9. The average value (i.e., 0.85) of the maximum similarity value in the fused similarity matrix of the preset text and the similarity between the preset text features and the global image features can be calculated, and the calculation result can be used as the degree of matching between the preset text and the image to be tested. It should be understood that the weighted average value can also be calculated according to actual needs, and the embodiment of the present invention is not limited here.

[0100] In step S132 , based on the matching degree between each preset text and the image to be tested, the preset text matching the image to be tested is determined as a text description related to the defect existing in the wafer to be tested.

[0101] The preset text with the highest matching degree among the matching degrees between each preset text and the image to be tested can be used as the preset text matching the image to be tested. The preset text is used as the text description related to the defects existing in the wafer grain to be tested. For example, the matching degree between the preset text T5 and the image to be tested is 0.8, the matching degree between the preset text T6 and the image to be tested is 0.3, the matching degree between the preset text T7 and the image to be tested is 0.6, and the matching degree between the preset text T8 and the image to be tested is 0.2, then the preset text T5 is used as the preset text matching the image to be tested. If the preset text T5 is "In the [non-pad] area, [there is] defect, there is [transparent] [foreign matter]", then the text description related to the defect existing in the wafer grain to be tested can be "In the [non-pad] area, [there is] defect, there is [transparent] [foreign matter]".

[0102] According to the above scheme of the embodiment of the present invention, for each preset text, the preset text feature of the preset text can be respectively calculated with each feature point in each first intermediate feature to obtain the second similarity matrix of the preset text and each first intermediate feature. Then, all the second similarity matrices of the preset text are fused to obtain the fused similarity matrix of the preset text. In addition, for each preset text, the matching degree of the preset text and the image to be tested can be determined based on the similarity between the preset text feature of the preset text and the global image feature and the fused similarity matrix of the preset text. Then, based on the matching degree of each preset text with the image to be tested, the preset text matching the image to be tested is determined as a text description related to the defects existing in the wafer grain to be tested. The above scheme can comprehensively determine the preset text matching the image to be tested based on the fused similarity matrix of the preset text and the similarity between the preset text feature of the preset text and the global image feature, so that the representativeness of the matched preset text can be improved, which is conducive to improving the detection accuracy of the image to be tested.

[0103] Exemplarily, the trained image feature extraction module in the above defect detection method is obtained at least through steps S410 to S430.

[0104] In step S410 , the training image of the training wafer die is input into an untrained image feature extraction module to obtain the global image features of the training image.

[0105] The training image corresponds to a training text, which is a text description related to the defects present in the training wafer. For example, the above training text can be used to describe the relevant information of the defects of the training object, such as whether the training object has defects, the defect area of ​​the defects present in the training object, the defect type of the defects present in the training object, etc. The specific content of the training text can be set by the developer, and the embodiment of the present invention does not limit this. In one example, the training text can be the preset text mentioned above.

[0106] In step S420, the training text corresponding to the training image is input into a text feature extraction module to obtain training text features corresponding to the training image.

[0107] The training text may be input into a text encoder in a language model to obtain a text embedding of the training text and the text embedding may be used as a training text feature corresponding to the training image. The training text may be related to defects in the training image. It should be understood that the training image may include images with defects or images without defects. The training text related to the defect type of the image without defects may include a text description for describing that the image does not have defects.

[0108] In step S430, the predicted similarity is determined based on the global image features of the training image and the training text features, and the module parameters of the untrained image feature extraction module are adjusted based on the difference between the predicted similarity and the preset highest similarity.

[0109] Based on the global image features and training text features of the training image, the predicted similarity can be determined through the relevant steps of step S120 above. Since the training text features are obtained through feature extraction operations of the training text related to the training text, the similarity between the training text features and the global image features should be the highest. Therefore, the difference between the predicted similarity and the preset highest similarity can be used as the adjustment benchmark for the module parameters of the untrained image feature extraction module.

[0110] Based on the difference between the predicted similarity and the preset highest similarity, the module parameters of the untrained image feature extraction module are adjusted. For example, the preset highest similarity is 1 and the predicted similarity is 0.8, then according to the difference between the predicted similarities 0.8 and 1, the module parameters of the untrained image feature extraction module are adjusted. For example, the difference between the predicted similarity and the preset highest similarity can be used as part of the loss value of the untrained image feature extraction module, and the training goal of the image feature extraction module may include minimizing the loss value. Therefore, the image feature extraction module trained by the above scheme can continuously reduce the difference between the predicted similarity and the preset highest similarity, thereby narrowing the matching degree between the training image and the training text. Therefore, after training, the text related to the defects in the image to be tested can be accurately matched. In an example, the difference between the predicted similarity and the preset highest similarity can be represented by a mean square error, a cross entropy, and the like. It should be understood that the difference between the predicted similarity and the preset highest similarity can be used as part of the loss value, and the loss value can also include the loss value obtained by other loss functions in the related art, and the embodiment of the present invention is not limited here. The module parameters of the untrained image feature extraction module can be adjusted by using methods such as gradient descent and back propagation to obtain a trained image feature extraction module. It should be understood that the training image can also correspond to a defect area marking matrix, and the defect area in the training image can be marked as 1 and the non-defect area can be marked as 0. The difference between the defect area marking matrix and the similarity matrix can then be used as part of the loss value.

[0111] According to the above scheme of the embodiment of the present invention, the training image of the training wafer grain can be input into the untrained image feature extraction module to obtain the global image feature of the training image. Then, the training text corresponding to the training image is input into the text feature extraction module to obtain the training text feature corresponding to the training image. Finally, based on the global image feature of the training image and the training text feature, the predicted similarity is determined, and based on the difference between the predicted similarity and the preset highest similarity, the module parameters of the untrained image feature extraction module are adjusted. The above scheme can adjust the module parameters of the untrained image feature extraction module based on the difference between the predicted similarity and the preset highest similarity, so that the trained image feature extraction module can focus on the features of the wafer grain to be tested related to the preset text in the image to be tested, which is beneficial to improve the accuracy of the preset text matching the image to be tested.

[0112] Exemplarily, the untrained image feature extraction module includes a plurality of image feature extraction submodules connected in sequence.

[0113] The above step S410 inputs the training image of the training wafer die into the untrained image feature extraction module to obtain the global image feature of the training image, which may include step S411.

[0114] In step S411, for each untrained image feature extraction submodule, feature extraction processing is performed on the second intermediate features output by the training image or other untrained image feature extraction submodules connected to the untrained image feature extraction submodule through the untrained image feature extraction submodule to obtain the second intermediate features output by the untrained image feature extraction submodule, and the second intermediate features output by the last untrained image feature extraction submodule are used as the global image features of the training image.

[0115] See also Figure 3 As shown, Figure 3 FIG. 1 is a schematic diagram showing a training process of an image feature extraction module according to an embodiment of the present invention. Figure 3 , the training text can be input into the text encoder to obtain the training text features. The training image can be input into the image feature extraction module to output the second intermediate features through the image feature extraction submodule. The second intermediate features output by the last image feature extraction submodule are used as the global image features of the training image, and the predicted similarity is generated with the training text features. The predicted similarity can be calculated by difference with the annotation of the training image (which may include the preset highest similarity mentioned above) to obtain the global loss.

[0116] According to the above scheme of the embodiment of the present invention, the training image can be input into the image feature extraction submodules connected in sequence that have been trained, so that the last image feature extraction submodule that has been trained outputs the global image feature. In the above scheme, multiple image feature extraction submodules are used to extract features from the training image, which can enrich the feature level of the obtained global image feature, thereby improving the representativeness of the global image feature, which is conducive to improving the accuracy of the detection result.

[0117] Exemplarily, in step S430, based on the difference between the predicted similarity and the preset highest similarity, the module parameters of the untrained image feature extraction module are adjusted, which may include steps S431 to S434.

[0118] In step S431, a reference similarity matrix is ​​generated based on the training image.

[0119] In the benchmark similarity matrix, the benchmark similarity corresponding to the defective area may be a preset highest similarity, and the benchmark similarity corresponding to other areas may be a preset lowest similarity. For example, determine the defective area and the non-defective area in the training image. Mark the pixels in the defective area of ​​the training image as 1, and mark the pixels in the non-defective area of ​​the training image as 0. Use the marked training image as the benchmark similarity matrix. For another example, perform an image feature extraction operation on the training image to obtain the training image features. Determine the feature points in the training image features that correspond to the defective area and the feature points in the non-defective area. Mark the feature points in the training image features that are located in the defective area as 1, and mark the feature points in the training image features that are located in the non-defective area as 0. Use the marked training image features as the benchmark similarity matrix.

[0120] In step S432, similarity is calculated between the training text feature and each feature point in each second intermediate feature to obtain a third similarity matrix between the training text and each second intermediate feature.

[0121] The above similarity calculation can be expressed by Euclidean distance, cosine similarity, etc. The above scheme can calculate the similarity between each feature point in the second intermediate feature and the training text feature, obtain the similarity corresponding to each feature point in the second intermediate feature, and then obtain the third similarity matrix of the second intermediate feature. Based on this, it can be determined whether the second intermediate feature matches the training text feature. If the similarity is high, it can be regarded that the feature point is more consistent with the semantics of the training text. Combined with actual scenarios, feature points with higher similarity can be regarded as feature points that may have defects in the wafer.

[0122] In step S433, all third similarity matrices of the training texts are fused to obtain a fused similarity matrix of the training image.

[0123] All third similarity matrices may be fused using methods such as linear averaging and rule-based fusion in related technologies to obtain a fused similarity matrix of the training image.

[0124] In step S434, the parameters are adjusted based on the difference between the predicted similarity and the preset highest similarity and the difference between the fused similarity matrix of the training image and the reference similarity matrix.

[0125] The difference between the fused similarity matrix and the reference similarity matrix can be determined based on the correspondence between the position of each similarity in the fused similarity of the training image and the position of each similarity in the reference similarity matrix. For example, when the reference similarity matrix is ​​the training image marked in step S431, the size of the reference similarity matrix can be adjusted to the same size as the fused similarity matrix. Then, the adjusted reference similarity matrix (which can be Figure 3 The difference between the predicted similarity and the preset highest similarity can be calculated by referring to the relevant process of step S430 above, and the present disclosure will not elaborate on this.

[0126] According to the above scheme of the embodiment of the present invention, a reference similarity matrix can be generated according to the defective area and the non-defective area in the training image. Then, the training text feature is respectively calculated for similarity with each feature point in each second intermediate feature to obtain the third similarity matrix of the training text and each second intermediate feature. Then, all the third similarity matrices of the training text are fused to obtain the fused similarity matrix of the training image. Finally, the parameters are adjusted based on the difference between the predicted similarity and the preset highest similarity and the difference between the fused similarity matrix of the training image and the reference similarity matrix. In the above scheme, the module parameters of the trained image extraction module are adjusted according to the difference between the predicted similarity and the preset highest similarity and the difference between the fused similarity matrix of the training image and the reference similarity matrix, so that the trained image feature extraction module can comprehensively consider the matching degree between the image to be tested and the preset text according to the similarity between the defective area in the image to be tested and the preset text and the similarity between the global image to be tested and the preset text, which is conducive to improving the accuracy of the matched preset text.

[0127] The embodiment of the present invention also provides a defect detection device. Figure 4 FIG. 5 is a schematic block diagram of a defect detection device 500 according to an embodiment of the present invention. Figure 4 As shown, the detection device 500 may include: a first feature extraction module 510 , a matching module 520 and a defect description module 530 .

[0128] The first feature extraction module 510 is used to perform feature extraction on the image to be tested of the wafer grain to be tested to obtain the global image feature of the image to be tested. The matching module 520 is used to perform feature extraction on each preset text to obtain the preset text feature of the preset text, and perform similarity calculation between the preset text feature of the preset text and the global image feature to obtain the similarity between the preset text feature and the global image feature, wherein the preset text is a text description related to the defects existing in the reference wafer grain. The defect description module 530 is used to determine the preset text that matches the image to be tested based on the similarity between each preset text feature and the global image feature, as a text description related to the defects existing in the wafer grain to be tested.

[0129] Exemplarily, the first feature extraction module 510 may include a global image feature generation module.

[0130] The global image feature generation module is used to input the image to be tested into the trained image feature extraction module to obtain the global image features of the image to be tested.

[0131] Exemplarily, the trained image feature extraction module may include a plurality of image feature extraction submodules connected in sequence, and the global image feature generation module may include a first intermediate feature generation module.

[0132] The first intermediate feature generation module is used to perform feature extraction processing on the first intermediate features output by the image to be tested or other image feature extraction submodules connected to the image feature extraction submodule for each image feature extraction submodule, so as to obtain the first intermediate features output by the image feature extraction submodule, and use the first intermediate features output by the last image feature extraction submodule as the global image feature.

[0133] Exemplarily, the detection device 500 may further include: a first similarity matrix generating module, a first similarity matrix fusion module and a defect area determining module.

[0134] The first similarity matrix generating module is used to calculate the similarity between the preset text features of the preset text and each feature point in each first intermediate feature for the preset text matched with the image to be tested, so as to obtain a first similarity matrix between the preset text and each first intermediate feature;

[0135] The first similarity matrix fusion module is used to fuse all first similarity matrices of the preset text matched by the image to be tested to obtain a fused similarity matrix of the preset text;

[0136] The defect area determination module is used to match the preset text with the image to be tested, and based on the target position information of each target similarity in the fusion similarity matrix of the preset text, mark the pixels corresponding to each target position in the image to be tested as part of the defect area of ​​the image to be tested, wherein the target similarity is greater than the preset similarity.

[0137] Exemplarily, the detection device 500 may further include: a second similarity matrix generating module and a second similarity matrix fusion module.

[0138] For each preset text, similarity calculation is performed between the preset text feature of the preset text and each feature point in each first intermediate feature, so as to obtain a second similarity matrix between the preset text and each first intermediate feature.

[0139] For each preset text, all second similarity matrices of the preset text are fused to obtain a fused similarity matrix of the preset text.

[0140] The defect description module 530 may include: a first matching module and a second matching module.

[0141] The first matching module is used to determine the matching degree between each preset text and the image to be tested based on the similarity between the preset text feature of the preset text and the global image feature and the fusion similarity matrix of the preset text. The second matching module is used to determine the preset text matching the image to be tested based on the matching degree between each preset text and the image to be tested, so as to serve as a text description related to the defects existing in the wafer grain to be tested.

[0142] Exemplarily, the detection device 500 may further include: a training image feature generation module, a training text feature generation module and a first difference module.

[0143] The training image feature generation module is used to input the training image of the training wafer grain into the untrained image feature extraction module to obtain the global image feature of the training image, wherein the training image corresponds to a training text, and the training text is a text description related to the defects existing in the training wafer grain. The training text feature generation module is used to input the training text corresponding to the training image into the text feature extraction module to obtain the training text feature corresponding to the training image. The first difference module is used to determine the predicted similarity based on the global image feature of the training image and the training text feature, and adjust the module parameters of the untrained image feature extraction module based on the difference between the predicted similarity and the preset maximum similarity.

[0144] Exemplarily, the untrained image feature extraction module includes a plurality of image feature extraction submodules connected in sequence, and the trained image feature generation module may include a second intermediate feature generation module.

[0145] The second intermediate feature generation module is used to perform feature extraction processing on the second intermediate features output by the training image or other untrained image feature extraction submodules connected to the untrained image feature extraction submodule for each untrained image feature extraction submodule, so as to obtain the second intermediate features output by the untrained image feature extraction submodule, and use the second intermediate features output by the last untrained image feature extraction submodule as the global image feature of the training image.

[0146] Exemplarily, the first difference module further includes: a reference similarity matrix generation module, a third similarity matrix generation module, a third similarity matrix fusion module and an adjustment module.

[0147] The benchmark similarity matrix generation module is used to generate a benchmark similarity matrix based on the training image, wherein the benchmark similarity corresponding to the defective area in the benchmark similarity matrix is ​​the preset highest similarity, and the benchmark similarity corresponding to other areas is the preset lowest similarity. The third similarity matrix generation module is used to calculate the similarity between the training text features and each feature point in each second intermediate feature, so as to obtain the third similarity matrix between the training text and each second intermediate feature. The third similarity matrix fusion module is used to fuse all the third similarity matrices of the training text to obtain the fused similarity matrix of the training image. The adjustment module is used to adjust the parameters based on the difference between the predicted similarity and the preset highest similarity and the difference between the fused similarity matrix of the training image and the benchmark similarity matrix.

[0148] Exemplarily, the preset text includes: a text description related to whether there is a defect in the reference wafer die, a text description related to a region in which a defect exists in the reference wafer die, and a text description related to a type of defect existing in the reference wafer die.

[0149] According to another aspect of the present invention, an electronic device is also provided. Figure 5 FIG. 6 shows a schematic block diagram of an electronic device 600 according to an embodiment of the present invention. Figure 5 As shown, the electronic device 600 includes a processor 610 and a memory 620. The memory 620 stores a computer program. When the computer program instructions are executed by the processor 610, they are used to execute the above-mentioned defect detection method.

[0150] In addition, according to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned defect detection method of the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned defect detection device or the corresponding modules in the above-mentioned electronic device according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above-mentioned storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media. According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, and when the above-mentioned computer program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned defect detection method.

[0151] A person skilled in the art may understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the above-mentioned description of the defect detection method, which will not be described here for the sake of brevity.

[0152] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0153] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0155] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0156] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0157] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0158] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0159] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the defect detection device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0160] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0161] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A defect detection method, characterized in that: The method comprises: Performing feature extraction on the image to be tested of the wafer to be tested to obtain global image features of the image to be tested; For each preset text, feature extraction is performed on the preset text to obtain a preset text feature of the preset text, and similarity calculation is performed between the preset text feature of the preset text and the global image feature to obtain the similarity between the preset text feature and the global image feature, wherein the preset text is a text description related to a defect existing in a reference wafer grain; Based on the similarity between each preset text feature and the global image feature, a preset text matching the image to be tested is determined to serve as a text description related to the defect existing in the wafer to be tested.

2. The method according to claim 1, characterized in that The feature extraction of the image to be tested of the wafer to be tested to obtain the global image feature of the image to be tested includes: The image to be tested is input into the trained image feature extraction module to obtain the global image features of the image to be tested.

3. The method according to claim 2, characterized in that The trained image feature extraction module includes a plurality of image feature extraction submodules connected in sequence, and the inputting the image to be tested into the trained image feature extraction module to obtain the global image features of the image to be tested includes: For each image feature extraction submodule, the image feature extraction submodule performs feature extraction processing on the first intermediate feature output by the image to be tested or other image feature extraction submodules connected to the image feature extraction submodule to obtain the first intermediate feature output by the image feature extraction submodule, and the first intermediate feature output by the last image feature extraction submodule is used as the global image feature.

4. The method according to claim 3, characterized in that The method further comprises: Preset text matched to the image to be tested, Calculate the similarity between the preset text feature of the preset text and each feature point in each first intermediate feature to obtain a first similarity matrix between the preset text and each first intermediate feature; Fusing all first similarity matrices of the preset text to obtain a fused similarity matrix of the preset text; Based on the target position information of each target similarity in the fusion similarity matrix of the preset text, the pixels corresponding to each target position are marked in the image to be tested as part of the defect area of ​​the image to be tested, wherein the target similarity is greater than the preset similarity.

5. The method according to claim 3, characterized in that The method further comprises: For each preset text, Calculate the similarity between the preset text feature of the preset text and each feature point in each first intermediate feature to obtain a second similarity matrix between the preset text and each first intermediate feature; Fusing all the second similarity matrices of the preset text to obtain a fused similarity matrix of the preset text; The step of determining the preset text matched with the image to be tested based on the similarity between each preset text feature and the global image feature, so as to serve as a text description related to the defect existing in the wafer to be tested, comprises: For each preset text, based on the similarity between the preset text feature of the preset text and the global image feature and the fusion similarity matrix of the preset text, determine the matching degree between the preset text and the image to be tested; Based on the matching degree between each preset text and the image to be tested, the preset text matching the image to be tested is determined as a text description related to the defect existing in the wafer to be tested.

6. The method according to claim 2, characterized in that The trained image feature extraction module is obtained by at least the following steps: Inputting a training image of a training wafer die into an untrained image feature extraction module to obtain a global image feature of the training image, wherein the training image corresponds to a training text, and the training text is a text description related to defects present in the training wafer die; Inputting the training text corresponding to the training image into a text feature extraction module to obtain the training text features corresponding to the training image; Based on the global image features of the training image and the training text features, the predicted similarity is determined, and based on the difference between the predicted similarity and a preset maximum similarity, the module parameters of the untrained image feature extraction module are adjusted.

7. The method according to claim 6, characterized in that The untrained image feature extraction module includes a plurality of image feature extraction submodules connected in sequence, and the training image of the training wafer grain is input into the untrained image feature extraction module to obtain the global image feature of the training image, including: For each untrained image feature extraction submodule, the untrained image feature extraction submodule performs feature extraction processing on the second intermediate features output by the training image or other untrained image feature extraction submodules connected to the untrained image feature extraction submodule to obtain the second intermediate features output by the untrained image feature extraction submodule, and the second intermediate features output by the last untrained image feature extraction submodule are used as the global image features of the training image.

8. The method according to claim 7, characterized in that The module parameters of the untrained image feature extraction module are adjusted based on the difference between the predicted similarity and the preset highest similarity, and are obtained by the following steps: Based on the training image, a reference similarity matrix is ​​generated, wherein the reference similarity corresponding to the defective area in the reference similarity matrix is ​​a preset highest similarity, and the reference similarities corresponding to other areas are a preset lowest similarity; Calculating the similarity between the training text feature and each feature point in each second intermediate feature to obtain a third similarity matrix between the training text and each second intermediate feature; Fusing all third similarity matrices of the training text to obtain a fused similarity matrix of the training image; The parameters are adjusted based on the difference between the predicted similarity and the preset highest similarity and the difference between the fused similarity matrix of the training image and the reference similarity matrix.

9. The method according to claim 1, characterized in that The preset text includes: a text description related to whether there is a defect in the reference wafer die, a text description related to a region in the reference wafer die where the defect exists, and a text description related to a type of defect in the reference wafer die.

10. A defect detection device, characterized in that: The device comprises: A first feature extraction module is used to extract features from the image to be tested of the wafer to be tested, so as to obtain global image features of the image to be tested; A matching module, for each preset text, performing feature extraction on the preset text to obtain a preset text feature of the preset text, and performing similarity calculation between the preset text feature of the preset text and the global image feature to obtain a similarity between the preset text feature and the global image feature, wherein the preset text is a text description related to a defect existing in a reference wafer grain; The defect description module is used to determine the preset text matching the image to be tested based on the similarity between each preset text feature and the global image feature, so as to serve as a text description related to the defect existing in the wafer to be tested.

11. An electronic device, characterized in that: It comprises a memory and a processor, wherein: the memory is used to store a computer program; the processor is used to execute the computer program to implement the defect detection method according to any one of claims 1 to 9.

12. A storage medium storing a computer program / instruction, characterized in that: The computer program / instructions are used to execute the defect detection method according to any one of claims 1 to 9 when running.

13. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the defect detection method according to any one of claims 1 to 9 when running.

Images