Image defect type detection method and device, electronic equipment and storage medium

Abstract

The invention provides an image defect type detection method and device, electronic equipment and a storage medium. The detection method comprises the following steps: determining a first defect area in a reference image of a reference object, and determining a second defect area in a to-be-detected image of a to-be-detected object; determining a plurality of first key points located in the first defect area in the reference image, and determining a plurality of second key points located in the second defect area in the to-be-detected image; based on a matching result of the first key point and the second key point, determining the similarity between the reference image and the to-be-detected image; and if the similarity between the reference image and the to-be-detected image is higher than the preset similarity, determining that the defect type label corresponding to the to-be-detected image is the defect type label corresponding to the reference image. On one hand, the acquisition cost of the training image can be reduced; and on the other hand, the scheme can be directly applied to the detection process of the defect type label of the image, so that the generation cost of the defect type label can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a method for detecting defect types of images, a device for detecting defect types of images, an electronic device, a storage medium and a computer program product. Background Art

[0002] In the industrial field, if defective industrial components (such as wafers) are packaged and tested, it will waste manpower and material resources. Therefore, it is necessary to inspect the industrial components so that unqualified industrial components can be detected and eliminated.

[0003] For example, in the wafer inspection scenario, on the one hand, due to the confidentiality of the wafer manufacturing process, there are relatively few training images; on the other hand, due to the differences in the acquisition scenarios of the training images and the images to be tested, such as different lighting, different resolutions of the image acquisition equipment, different shooting angles, etc., the model trained based on the training images is difficult to match the actual inspection scenario. Therefore, the accuracy of the defect detection model obtained through training is low. Therefore, how to accurately determine the defect type of a wafer or wafer grain has always been a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0004] The present invention is proposed in view of the above problems. The present invention provides a method for detecting defect types of an image, a device for detecting defect types of an image, an electronic device, a storage medium and a computer program product.

[0005] According to one aspect of the present invention, a method for detecting defect types of an image is provided, the method comprising:

[0006] Determine a first defect region in a reference image of a reference object, and determine a second defect region in an image to be tested of an object to be tested; wherein the reference object and the object to be tested are both wafers or wafer dies, and the reference image corresponds to a defect type label of the reference object;

[0007] Determine a plurality of first key points located in a first defect region in the reference image, and determine a plurality of second key points located in a second defect region in the image to be tested;

[0008] Determine the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point;

[0009] If the similarity between the reference image and the image to be tested is higher than a preset similarity, it is determined that the defect type label corresponding to the image to be tested is the defect type label corresponding to the reference image.

[0010] Exemplarily, determining the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point includes:

[0011] For each first key point, inputting a first key point feature of the first key point into a first attention module to obtain a first enhanced feature of the first key point, and for each second key point, inputting a second key point feature of the second key point into a second attention module to obtain a second enhanced feature of the second key point, wherein the first key point feature is obtained based on an image feature of the first key point in a reference image, and the second key point feature is obtained based on an image feature of the second key point in a test image;

[0012] The first enhanced feature and the second enhanced feature are matched to determine the similarity between the reference image and the image to be tested.

[0013] Exemplarily, the first attention module includes a first self-attention module and a first cross-attention module, the second attention module includes a second self-attention module and a second cross-attention module, the first cross-attention module is connected to the first self-attention module and the second self-attention module, and the second cross-attention module is connected to the first self-attention module and the second self-attention module.

[0014] For each first key point, inputting the first key point feature of the first key point into the first attention module to obtain a first enhanced feature of the first key point, including:

[0015] For each first key point,

[0016] Determine a second key point that matches the first key point;

[0017] Inputting the first key point feature of the first key point into the first self-attention module to obtain a first attention feature of the first key point;

[0018] Inputting a first attention feature of the first key point and a second attention feature of a second key point matching the first key point into a first cross attention module to obtain a first enhanced feature of the first key point;

[0019] For each second key point, inputting the second key point feature of the second key point into the second attention module to obtain a second enhanced feature of the second key point, including:

[0020] For each second key point,

[0021] Determine the first key point that matches the second key point;

[0022] Inputting the second key point feature of the second key point into the second self-attention module to obtain a second attention feature of the second key point;

[0023] The second attention feature of the second key point and the first attention feature of the first key point matching the second key point are input into the second cross-attention module to obtain the second enhanced feature of the second key point.

[0024] Exemplarily, the method further comprises:

[0025] Dividing the reference image and the image to be tested into a plurality of initial image blocks respectively;

[0026] For each first key point, determining a second key point that matches the first key point includes:

[0027] For each first key point,

[0028] Determine, based on the first position information of the first key point, a first image block feature corresponding to the first key point, wherein the first image block feature is an image feature of an initial image block of the reference image where the first key point is located;

[0029] Matching the first image block feature corresponding to the first key point with the second image block feature corresponding to each second key point respectively, to obtain the second key point matched with the first key point;

[0030] For each second key point, determining the first key point that matches the second key point includes:

[0031] For each second key point,

[0032] Based on the second position information of the second key point, determine a second image block feature corresponding to the second key point, wherein the second image block feature is an image feature of an initial image block of the image to be tested where the second key point is located;

[0033] The second image block feature corresponding to the second key point is matched with the first image block feature corresponding to each first key point to obtain the first key point matched by the second key point.

[0034] Exemplarily, the method further comprises:

[0035] For each first key point,

[0036] Determine a first image feature corresponding to the first key point in the reference image and a first position feature of the first key point in the reference image, wherein the first position feature is obtained based on first position information of the first key point;

[0037] Performing feature splicing on the first image feature of the first key point and the first position feature of the first key point to obtain a first key point feature;

[0038] For each second key point,

[0039] Determine a second image feature corresponding to the second key point in the image to be tested and a second position feature of the second key point in the image to be tested, wherein the second position feature is obtained based on second position information of the second key point;

[0040] The second image feature of the second key point and the second position feature of the second key point are feature spliced ​​to obtain the second key point feature.

[0041] Exemplarily, the query vector and key vector input to the first self-attention module in the first attention module are both first key point features, and the value vector is the first image feature; the query vector and key vector input to the second self-attention module in the second attention module are both second key point features, and the value vector is the second image feature.

[0042] Exemplarily, for each first key point, the query vector input to the first cross attention module in the first attention module is the first key point feature of the first key point, the key vector is the second key point feature of the second key point matched by the first key point, and the value vector is the second image feature of the second key point matched by the first key point;

[0043] For each second key point, the query vector input to the second cross-attention module in the second attention module is the second key point feature of the second key point, the key vector is the first key point feature of the first key point matched by the second key point, and the value vector is the first image feature of the first key point matched by the second key point.

[0044] Exemplarily, determining a plurality of first key points located in a first defect region in a reference image, and determining a plurality of second key points located in a second defect region in an image to be tested, includes:

[0045] Inputting the reference image and the first defect area into a key point extraction network to obtain first position information of each first key point and first image features of each first key point;

[0046] The image to be tested and the second defect area are input into the key point extraction network to obtain the second position information of each second key point and the second image feature of each second key point.

[0047] According to another aspect of the present invention, a device for detecting defect types of an image is provided. The device comprises: a defect area determination module, a key point determination module, a similarity determination module and a label determination module.

[0048] The defect area determination module is used to determine a first defect area in a reference image of a reference object, and to determine a second defect area in a test image of a test object; wherein the reference object and the test object are both wafers or wafer dies, and the reference image corresponds to a defect type label of the reference object;

[0049] The key point determination module is used to determine a plurality of first key points located in a first defect area in the reference image, and to determine a plurality of second key points located in a second defect area in the image to be tested;

[0050] The similarity determination module is used to determine the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point;

[0051] The label determination module is used to determine that the defect type label corresponding to the image to be tested is the defect type label corresponding to the reference image if the similarity between the reference image and the image to be tested is higher than a preset similarity.

[0052] According to another aspect of the present invention, an electronic device is provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned image defect type detection method when the processor is running.

[0053] According to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned image defect type detection method when running.

[0054] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which are used to execute the above-mentioned image defect type detection method when running.

[0055] According to the above scheme of the embodiment of the present invention, a first defect area can be determined in the reference image of the reference object, and a second defect area can be determined in the image to be tested of the object to be tested. Then, a plurality of first key points located in the first defect area are determined in the reference image, and a plurality of second key points located in the second defect area are determined in the image to be tested. Based on the matching results of the first key points and the second key points, the similarity between the reference image and the image to be tested is determined. Finally, if the similarity between the reference image and the image to be tested is higher than the preset similarity, the defect type label corresponding to the image to be tested is determined to be the defect type label corresponding to the reference image. The above scheme can determine the defect type label of the image to be tested of the object to be tested by matching the reference image of the reference object with the image to be tested of the object to be tested by key points. On the one hand, the image to be tested with the determined defect type label can be used for the training of the machine learning model to reduce the acquisition cost of the training image; on the other hand, the above scheme can also be directly applied to the detection process of the defect type label of the image to replace or combine the process of determining the defect type label of the image using the machine learning model, which is conducive to reducing the generation cost of the defect type label. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0057] Figure 1 A schematic flow chart of a method for detecting defect types of an image according to an embodiment of the present invention is shown;

[0058] Figure 2 A schematic diagram showing a method for detecting defect types of an image according to an embodiment of the present invention is shown;

[0059] Figure 3 A schematic diagram showing a method for detecting defect types of an image according to an embodiment of the present invention is shown;

[0060] Figure 4 A schematic block diagram showing a device for detecting defect types of an image according to an embodiment of the present invention; and

[0061] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0063] With the continuous development of science and technology, artificial intelligence technology can be applied to multiple links of industrial component manufacturing and testing. Deep learning technology represented by convolutional neural networks has greatly improved the accuracy of industrial component defect detection, which makes it possible to deploy neural network models on industrial quality inspection production lines. Since neural network models have hundreds of millions of parameters and have strong generalization capabilities, they can adapt to very complex scenarios. Especially in complex industrial scenarios, defect detection algorithms represented by deep learning can play a huge role. When training the model, it is necessary to learn the characteristics of industrial component defects in order to locate and classify different types of defects in industrial components.

[0064] However, when training a neural network model, if you need to improve the accuracy of the neural network model, you need a lot of data annotation. The above data annotation is used to mark the defect types in industrial components, and then the labeled data images are sent to the neural network model for learning. When there is less labeled data, the accuracy of the neural network model is usually low, that is, the practicality is poor.

[0065] In order to at least partially solve the above problem, an embodiment of the present invention provides a method for detecting defect types of an image. Figure 1 FIG. 2 shows a schematic flow chart of a method for detecting defect types of an image according to an embodiment of the present invention. Figure 1 As shown, the detection method may include: steps S110 to S140.

[0066] In step S110 , a first defect region is determined in a reference image of a reference object, and a second defect region is determined in an image to be tested of an object to be tested.

[0067] The reference object and the object to be tested are both wafers or wafer grains. The reference object and the object to be tested can be determined according to actual conditions. Specifically, the reference object and the object to be tested can be the same object. For example, when the defect type detection method is used to detect the defect type of a wafer, the reference object and the object to be tested are both wafers. For another example, when the defect type detection method is used to detect the defect type of a wafer grain, the reference object and the object to be tested are both wafer grains.

[0068] The reference image is an image of the reference object. For example, if the reference object is a wafer, the reference image is the image of the wafer. For another example, if the reference object is a wafer grain, the reference image is the image of the wafer grain. The reference image can be a static image or any video frame in a dynamic video. The reference image can be an original image captured by an image acquisition device (for example, an original image captured by an image sensor in a camera), or it can be an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.).

[0069] The image to be tested of the object to be tested is the image of the object to be tested. For example, if the object to be tested is a wafer, the image to be tested is the image of the wafer. For another example, if the object to be tested is a wafer grain, the image to be tested is the image of the wafer grain. Among them, the image to be tested can be a static image, or it can be any video frame in a dynamic video. The image to be tested can be the original image captured by an image acquisition device (for example, the original image captured by an image sensor in a camera), or it can be an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.). Among them, the reference image and the image to be tested can be images of different objects. For example, when the reference object and the object to be tested are both wafer grains, the reference image can be the image of wafer grain A, and the image to be tested can be the image of wafer grain B.

[0070] The reference image corresponds to a defect type label of the reference object. The image to be tested does not correspond to a defect type label of the object to be tested. Among them, the defect type label may include any label of the defect type that may exist on the reference object. For example, if the reference object is a wafer, the defect type label may include a defect type label indicating that there are foreign objects in the pad area, a defect type label indicating that the pin print in the pad area is out of bounds, a defect type label indicating that there are too many pin prints in the pad area, a defect type label indicating that there are scratches in the pad area, a defect type label indicating that there are foreign objects in the surface area, a defect type label indicating that there are scratches in the surface area, a defect type label indicating that there are foreign objects in the trimming area, and a defect type label indicating that there are scratches in the trimming area. For example, when the reference object is a wafer, and there are scratches on the surface area of ​​the wafer, the reference image corresponds to a defect type label indicating that there are scratches on the surface area. Among them, the defect type labels corresponding to different reference objects may be different.

[0071] The first defect area is an area in the reference image that corresponds to the defect of the reference object. For example, when the reference object is wafer A and the reference image is an image of wafer A, and there is a scratch on pad area A of wafer A, the area in the reference image that corresponds to the scratch on pad area A is the first defect area.

[0072] The second defect area is an area in the image to be tested that corresponds to the defect of the object to be tested. For example, the object to be tested is wafer B, and the image to be tested is an image of wafer B. When there is a scratch on pad area B of wafer B, the area in the image to be tested that corresponds to the area with the scratch on pad area B is the second defect area.

[0073] The first defect area in the reference image and the second defect area in the image to be tested can be determined by template matching. For example, the reference image is matched with the reference image template of the reference object. And the image to be tested is matched with the image template to be tested of the object to be tested. The above-mentioned reference image template and the image template to be tested can be image templates without defects. For example, when the reference object and the object to be tested are both wafers, the image template of the reference object and the image template of the object to be tested can be images of wafers without any defects. It should be understood that the reference image template can be the same as the image template to be tested. Then, based on the matching of the reference image with the reference image template and the matching of the image to be tested with the image template to be tested, the first defect area and the second defect area can be determined respectively. For example, the first defect area is an area in the reference image with a difference greater than a threshold value from the reference image template, and the second defect area is an area in the image to be tested with a difference greater than a threshold value from the image template to be tested. Here, the above difference degree is represented by the absolute value of the difference between the pixel values ​​of the pixels as an example. The pixel value of each pixel in the reference image is subtracted from the pixel value of the pixel corresponding to the pixel in the reference image template to obtain the pixel difference corresponding to each pixel in the reference image, and the pixels in the reference image whose absolute value of the pixel difference is greater than the threshold are clustered to obtain at least one cluster area. The cluster area can be used as the first defect area. Similarly, the second defect area can be determined. Among them, the threshold value mentioned above can be determined according to the actual situation.

[0074] In step S120, a plurality of first key points located in a first defect region are determined in the reference image, and a plurality of second key points located in a second defect region are determined in the image to be tested.

[0075] The plurality of first key points can be obtained by sampling the first defect area, and the plurality of second key points can be obtained by sampling the second defect area. The sampling frequency can be determined according to actual conditions. The sampling frequency of the sampling operation can be determined according to actual conditions.

[0076] In some embodiments, the first key point may be more representative in the first defect region than other pixels in the first defect region. The second key point may be more representative in the second defect region than other pixels in the second defect region. For example, the first key point in the first defect region and the second key point in the second region may be determined by algorithms such as Harris corner detection and random sample consistency (RANSAC) in related technologies.

[0077] In step S130 , based on the matching results of the first key point and the second key point, the similarity between the reference image and the image to be tested is determined.

[0078] The similarity between the reference image and the image to be tested can be determined by calculating the similarity between the first key point and the second key point (i.e., the matching result). For example, the Euclidean distance, Minkowski distance, squared Euclidean distance, etc. between the first key point and the second key point can be calculated. The distance between the first key point and the second key point is converted into a similarity score. The similarity between the reference image and the image to be tested can then be determined.

[0079] In step S140 , if the similarity between the reference image and the image to be tested is higher than a preset similarity, the defect type label corresponding to the image to be tested is determined to be the defect type label corresponding to the reference image.

[0080] When the similarity between the reference image and the image to be tested is higher than the preset similarity, the defect type label of the reference image is used as the defect type label of the image to be tested. For example, if the similarity is higher than the preset similarity, the defect type label of the reference image is a defect type label indicating that there are scratches in the pad area, and the defect type label of the image to be tested is also a defect type label indicating that there are scratches in the pad area. For another example, if the similarity is higher than the preset similarity, the defect type label of the reference image is a defect type label indicating that there are foreign objects in the trimming area, and the defect type label of the image to be tested is also a defect type label indicating that there are foreign objects in the trimming area. Among them, the size of the preset similarity can be determined according to the actual situation.

[0081] It should be understood that there can be multiple reference images, that is, the image to be tested can be compared with each reference image for similarity. In one example, for each defect type label, multiple reference images corresponding to the defect type label can be set, or a more representative reference image corresponding to the defect type label can be set to improve the matching efficiency between the image to be tested and the reference image.

[0082] In some embodiments, if the similarity between the reference image and the image to be tested is less than or equal to a preset similarity, the above steps S110 to S140 may be re-executed based on the image to be tested and the reference images of other defect type labels until the defect type label of the image to be tested is determined. In some other embodiments, if the similarity between the image to be tested and any reference image is less than or equal to a preset similarity, the image to be tested may be annotated with a defect type label (e.g., manually annotated), and the image to be tested may be used as the reference image corresponding to the defect type label. In some other embodiments, the defect type label of the reference image with the highest similarity may also be used as the defect type label of the image to be tested.

[0083] According to the above scheme of the embodiment of the present invention, a first defect area can be determined in the reference image of the reference object, and a second defect area can be determined in the image to be tested of the object to be tested. Then, a plurality of first key points located in the first defect area are determined in the reference image, and a plurality of second key points located in the second defect area are determined in the image to be tested. Based on the matching results of the first key points and the second key points, the similarity between the reference image and the image to be tested is determined. Finally, if the similarity between the reference image and the image to be tested is higher than the preset similarity, the defect type label corresponding to the image to be tested is determined to be the defect type label corresponding to the reference image. The above scheme can determine the defect type label of the image to be tested of the object to be tested by matching the reference image of the reference object with the image to be tested of the object to be tested by key points. On the one hand, the image to be tested with the determined defect type label can be used for the training of the machine learning model to reduce the acquisition cost of the training image; on the other hand, the above scheme can also be directly applied to the detection process of the defect type label of the image to replace or combine the process of determining the defect type label of the image using the machine learning model, which is conducive to reducing the generation cost of the defect type label.

[0084] Exemplarily, step S120 determines a plurality of first key points located in a first defect region in the reference image, and determines a plurality of second key points located in a second defect region in the image to be tested, and may include step S121 and step S122.

[0085] In step S121, the reference image and the first defect area are input into a key point extraction network to obtain first position information of each first key point and first image features of each first key point.

[0086] See also Figure 2 As shown, Figure 2 FIG. 1 is a schematic diagram showing a method for detecting defect types of an image according to an embodiment of the present invention. Figure 2, the reference image and the first defect area can be input into the key point extraction network. The key point extraction network outputs each first key point in the first defect area, the first position information of each first key point, and the first image feature of each first key point. When there are m first key points, the key point extraction network outputs first position information 1 to first position information m, and first image feature 1 to first image feature m. Wherein, m is a positive integer.

[0087] For example, the key point extraction network outputs a reference image including multiple key points based on the reference image, and screens out the key points in the reference image that are not located in the first defect area based on the first defect area, and uses the key points in the reference image that are located in the first defect area as the first key points. The reference image including multiple key points can be masked according to the first defect area to screen out the key points that are not located in the first defect area, and retain the key points located in the first defect area. Each first key point output by the key point extraction network includes first position information for indicating the position of the first key point in the reference image, and first image features for indicating the image features of the first key point. It should be understood that only the image corresponding to the first defect area in the reference image can be input into the key point extraction network to directly obtain the first key point in the first defect area.

[0088] The process of extracting multiple key points from the reference image by the key point extraction network can be shown as follows. The key point extraction network can first convert the input reference image into a grayscale image. Then adjust the size of the reference image to a preset image size to meet the requirements of the key point extraction network. The pixel values ​​of the reference image are normalized to the range of [0,1] and converted into a form suitable for neural network input (for example, increasing the batch dimension and channel dimension). The key point extraction network is used for inference to obtain a key point heat map of the reference image. Redundant key point candidates are removed from the key point heat map, and the local maximum value is retained as the final detected key point.

[0089] In step S122, the image to be tested and the second defect area are input into the key point extraction network to obtain the second position information of each second key point and the second image feature of each second key point.

[0090] The process of step S122 can refer to the process of step S121, and the disclosure will not elaborate on it here. The position information of the second key point is used to indicate the position of the second key point in the image to be measured, and the second image feature is used to indicate the image feature of the second key point.

[0091] According to the above scheme of the embodiment of the present invention, the reference image and the first defect area can be input into the key point extraction network to obtain the first position information of each first key point and the first image feature of each first key point. Then, the image to be tested and the second defect area can be input into the key point extraction network to obtain the second position information of each second key point and the second image feature of each second key point. The above scheme can determine the first image feature and the first position information of the first key point, and determine the second image feature and the second position information of the second key point, and match the first key point and the second key point accordingly, which can improve the representativeness of the matching result and help improve the accuracy of the defect type label obtained. In addition, the above scheme is to extract key points for the defect area, which is conducive to further improving the accuracy of the defect type label.

[0092] Exemplarily, step S130 of determining the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point may include: step S131 and step S132.

[0093] In step S131, for each first key point, the first key point feature of the first key point is input into the first attention module to obtain the first enhanced feature of the first key point, and for each second key point, the second key point feature of the second key point is input into the second attention module to obtain the second enhanced feature of the second key point.

[0094] The first key point feature can be obtained based on the first image feature of the first key point in the reference image. The second key point feature can be obtained based on the second image feature of the second key point in the image to be measured. For example, the first image feature of the first key point can be used as the first key point feature, and the second image feature of the second key point can be used as the second key point feature. For another example, the first image feature of the first key point and the feature representing the position of the first key point (i.e., the first position feature) are used as the first key point feature, and the second image feature of the second key point and the feature representing the position of the second key point (i.e., the second position feature) are used as the second key point feature. Among them, the first image feature of the first key point and the second image feature of the second key point can refer to the description in step S121.

[0095] The first attention module can obtain the first enhanced feature of each first key point based on the first key point feature of each first key point in the reference image. For example, the first attention module obtains the first enhanced feature based on the correlation between the first key point features of each first key point in the reference image. For another example, the first attention module can obtain the first enhanced feature of each first key point based on the first key point feature of each first key point in the reference image, and the second key point feature of each second key point in the image to be tested. For example, the first attention module obtains the first enhanced feature based on the correlation between the first key point features of each first key point in the reference image, and the correlation between the first key point features of each first key point and the second key point features of each second key point.

[0096] The second attention module can obtain the second enhanced feature of each second key point based on the second key point feature of each second key point in the image to be tested. For example, the second attention module obtains the second enhanced feature based on the correlation between the second key point features of each second key point in the image to be tested. For another example, the second attention module can obtain the second enhanced feature of each second key point based on the second key point feature of each second key point in the image to be tested, and the first key point feature of each first key point in the reference image. For example, the second attention module obtains the second enhanced feature based on the correlation between the second key point features of each second key point in the image to be tested, and the correlation between the second key point features of each second key point and the first key point features of each first key point.

[0097] In step S132, the first enhanced feature and the second enhanced feature are matched to determine the similarity between the reference image and the image to be tested.

[0098] Optionally, the first enhanced feature can be used to calculate the similarity with the second enhanced feature to obtain the similarity between the reference image and the image to be tested. For example, the Euclidean distance, cosine similarity, etc. between the first enhanced feature and the second enhanced feature can be calculated. Alternatively, the first enhanced feature includes features of multiple first key points, and the second enhanced feature includes features of multiple second key points. For each second key point, based on the features of the second key point in the second enhanced feature and the features of each first key point in the first enhanced feature, the first key point that best matches the second key point can be determined (for example, using Euclidean distance, cosine similarity, etc. to match). The similarity between the reference image and the image to be tested can be determined in combination with the first key point that best matches each second key point. For example, the similarity of the first key points matched by all second key points can be combined to determine the mean similarity of the second key points as the similarity between the reference image and the image to be tested.

[0099] According to the above scheme of the embodiment of the present invention, the first key point feature of the first key point can be input into the first attention module to obtain the first enhanced feature of the first key point, and the second key point feature of the second key point can be input into the second attention module to obtain the second enhanced feature of the second key point. Then, the similarity between the reference image and the image to be tested is jointly determined based on the first enhanced feature and the second enhanced feature, thereby improving the accuracy of the similarity between the reference image and the image to be tested, which is conducive to improving the accuracy of the defect type label corresponding to the image to be tested.

[0100] Exemplarily, the first attention module may include a first self-attention module and a first cross-attention module, the second attention module may include a second self-attention module and a second cross-attention module, the first cross-attention module is connected to the first self-attention module and the second self-attention module, and the second cross-attention module is connected to the first self-attention module and the second self-attention module. Figure 3 As shown, Figure 3 A schematic diagram of a defect type detection method for an image according to an embodiment of the present invention is shown. Specifically, the output end of the first self-attention module is connected to the input end of the first cross-attention module, and the input end of the first cross-attention module is connected to the output end of the second self-attention module. The output end of the second self-attention module is connected to the input end of the second cross-attention module, and the input end of the second cross-attention module is connected to the output end of the first self-attention module. Figure 3 The first set may be a set of first key point features of the first key point, and the second set may be a set of second key point features of the second key point.

[0101] In the above step S131, for each first key point, the first key point feature of the first key point is input into the first attention module to obtain the first enhanced feature of the first key point, which may include: steps S1311a to S1313a.

[0102] In step S1311a, for each first key point, a second key point matching the first key point is determined.

[0103] Combination Figure 3, the fourth set may include the second key point feature of the second key point matched by the first key point. The similarity between each first key point and each second key point can be calculated, and the second key point whose similarity with the first key point is higher than a preset value is used as the second key point matched by the first key point. For example, taking the first key point A1, the first key point B1 and the first key point C1 in the reference image, and the second key point A2, the second key point B2 and the second key point C2 in the image to be tested as an example, the similarity between the first key point A1 and the second key point A2, the second key point B2 and the second key point C2 is calculated in sequence, the similarity between the first key point B1 and the second key point A2, the second key point B2 and the second key point C2 is calculated in sequence, and the similarity between the first key point C1 and the second key point A2, the second key point B2 and the second key point C2 is calculated in sequence. If the similarity between the second key point A2 and the second key point B2 and the first key point A1 is less than the first preset distance (that is, the similarity is greater than the preset value, and the specific value of the first preset distance is not limited here), the second key point A2 and the second key point B2 are used as the second key points matched by the first key point A1. Similarly, the second key points matched by other first key points can be obtained. The above preset values ​​can be determined according to actual conditions.

[0104] In step S1312a, for each first key point, the first key point feature of the first key point is input into the first self-attention module to obtain the first attention feature of the first key point.

[0105] Combination Figure 3 , the first self-attention module is used to output the first attention feature of each first key point according to the first key point feature of the input first key point. For example, if the first key point feature includes the first image feature mentioned above, the first self-attention module is used to output the first attention feature of the first key point according to the first image feature of the input first key point. For another example, if the first key point feature includes the first image feature mentioned above, and a feature used to represent the position of the first key point (such as a first position feature), the first self-attention module outputs the first attention feature of the first key point according to the first image feature of the input first key point and the feature used to represent the position of the first key point. The above-mentioned first attention feature can be used to represent the correlation between the first key point and each first key point (it can also be used to represent the correlation between the first key point and other first key points).

[0106] In step S1313a, for each first key point, the first attention feature of the first key point and the second attention feature of the second key point matching the first key point are input into the first cross-attention module to obtain the first enhanced feature of the first key point.

[0107] The first cross-attention module is used to output the first enhanced feature of the first key point according to the first attention feature of the input first key point and the second attention feature of the second key point matched by the first key point. For example, when the first attention feature includes the first image feature mentioned above and the second attention feature includes the second image feature mentioned above, the first cross-attention module is used to output the first enhanced feature of the first key point according to the first image feature in the input first attention feature and the second image feature in the second attention feature of each second key point matched by the first key point. For another example, when the first attention feature includes the first image feature mentioned above and the feature used to represent the position of the first key point, and the second attention feature also includes the second image feature mentioned above and the feature used to represent the position of the second key point, the first cross-attention module outputs the first enhanced feature of the first key point according to the first image feature in the input first attention feature, the feature used to represent the position of the first key point, the second image feature in the second attention feature of each second key point matched by the first key point, and the feature used to represent the position of each second key point matched by the first key point. Wherein, the first enhanced feature of the first key point can be used to represent the association between the first key point and the second key point matched by the first key point.

[0108] In the above step S131, for each second key point, the second key point feature of the second key point is input into the second attention module to obtain the second enhanced feature of the second key point, which may include: steps S1311b to S1313b.

[0109] In step S1311b, for each second key point, the first key point that matches the second key point is determined.

[0110] Combination Figure 3 , the third set may include the first key point feature of the first key point matched by the second key point. The similarity between each second key point and each first key point can be calculated, and the first key point with a similarity with the second key point higher than a preset value is used as the first key point matched by the second key point. The specific process can be referred to the above step S1311a, which will not be described in detail in this disclosure. Among them, the preset value can be determined according to the actual situation.

[0111] In step S1312b, for each second key point, the second key point feature of the second key point is input into the second self-attention module to obtain the second attention feature of the second key point.

[0112] Combination Figure 3As shown, the second self-attention module is used to output the second attention feature of each second key point according to the second key point feature of the input second key point. For example, if the second key point feature includes the second image feature mentioned above, the second self-attention module is used to output the second attention feature of the second key point according to the second image feature of the input second key point. For another example, if the second key point feature includes the second image feature mentioned above, and a feature for indicating the position of the second key point (such as a second position feature), the second self-attention module outputs the second attention feature of the second key point according to the second image feature of the input second key point and the feature for indicating the position of the second key point. Among them, the second attention feature can be used to represent the correlation between the second key point and each second key point (it can also be used to represent the correlation between the second key point and other second key points).

[0113] In step S1313b, for each second key point, the second attention feature of the second key point and the first attention feature of the first key point matching the second key point are input into the second cross-attention module to obtain the second enhanced feature of the second key point.

[0114] The second cross-attention module is used to output the second enhanced feature of the second key point according to the first attention feature of the input second key point and the first attention feature of the first key point matched by the second key point. For example, when the second attention feature includes the second image feature mentioned above and the first attention feature includes the first image feature mentioned above, the second cross-attention module is used to output the second enhanced feature of the second key point according to the second image feature in the input second attention feature and the first image feature in the first attention feature of each first key point matched by the second key point. For another example, when the second attention feature includes the second image feature mentioned above and the feature used to indicate the position of the second key point, and the first attention feature also includes the first image feature mentioned above and the feature used to indicate the position of the first key point, the second cross-attention module outputs the second enhanced feature of the second key point according to the second image feature in the input second attention feature, the feature used to indicate the position of the second key point, the first image feature in the first attention feature of each first key point matched by the second key point, and the feature used to indicate the position of each first key point matched by the second key point. Wherein, the second enhanced feature of the second key point is used to indicate the association between the second key point and the first key point matched by the second key point.

[0115] Combination Figure 3 As shown, the first self-attention module and the second self-attention module can share model parameters. The first cross-attention module can share model parameters with the second cross-attention module.

[0116] According to the above scheme of the embodiment of the present invention, the first attention feature output by the first self-attention module and the second attention feature of the second key point matched by the first key point can be input into the first cross-attention module to obtain the first enhanced feature of the first key point. And the second attention feature output by the second self-attention module and the first attention feature of the first key point matched by the second key point are input into the second cross-attention module to obtain the second enhanced feature of the second key point. The above scheme can improve the representativeness of the first enhanced feature and the second enhanced feature finally obtained through the self-attention mechanism and the cross-attention mechanism.

[0117] Illustratively, the detection method of the embodiment of the present invention may further include step S210.

[0118] In step S210, the reference image and the image to be tested are respectively divided into a plurality of initial image blocks.

[0119] In some embodiments, the reference image and the image to be tested may be divided into a plurality of initial image blocks respectively by using an image segmentation algorithm or an image segmentation model in the related art, and the embodiment of the present invention does not limit this. Figure 2 As shown, the reference image can be input into the Dinov2 model to obtain a reference image divided into multiple initial image blocks. The image to be tested is input into the Dinov2 model to obtain an image to be tested that is divided into multiple initial image blocks (the image block features can also be directly extracted through the model). Each initial image block may correspond to a position information and an image feature. For example, the position information corresponding to the initial image block in the reference image represents the position of the initial image block in the reference image, and the image feature corresponding to the initial image block represents the visual feature of the initial image block. The position information corresponding to the initial image block in the image to be tested represents the position of the initial image block in the image to be tested, and the image feature corresponding to the initial image block represents the visual feature of the initial image block.

[0120] If the detection method includes step S210, step S1311a may include: step S1311a1 to step S1311a2.

[0121] In step S1311a1, for each first key point, based on the first position information of the first key point, a first image block feature corresponding to the first key point is determined.

[0122] The first image block feature may be an image feature of an initial image block of the reference image where the first key point is located. The initial image block corresponding to the first key point may be determined according to the position of the first key point in the reference image. For example, if a first key point is located in an initial image block, the initial image block may be used as the initial image block corresponding to the first key point. Then, the image feature of the initial image block may be used as the first image block feature of the first key point. Figure 2 , if there are m first key points in total, there are m first image block features in total, that is Figure 2 The first image block 1 to the first image block m in . Wherein, m is a positive integer.

[0123] In step S1311a2, for each first key point, the first image block feature corresponding to the first key point is matched with the second image block feature corresponding to each second key point to obtain the second key point matched with the first key point.

[0124] The second key point matched with the first key point can be obtained by calculating the similarity between the first image block feature corresponding to the first key point and the second image block feature corresponding to each second key point. For example, the Euclidean distance between the first image block feature corresponding to the first key point and the second image block feature corresponding to each second key point can be calculated. The second key point whose corresponding Euclidean distance is less than the second preset distance is used as the second key point matched with the first key point. The specific value of the second preset distance can be determined according to actual conditions.

[0125] In the above example, step S1311b may include: steps S1311b1 to S1311b2.

[0126] In step S1311b1, for each second key point, based on the second position information of the second key point, a second image block feature corresponding to the second key point is determined.

[0127] The second image block feature may be an image feature of an initial image block of the image to be measured where the second key point is located. The initial image block corresponding to the second key point may be determined based on the position of the second key point in the image to be measured. For example, if a second key point is located in an initial image block, the initial image block may be used as the initial image block corresponding to the second key point. Then, the image feature of the initial image block may be used as the second image block feature of the second key point.

[0128] In step S1311b2, for each second key point, the second image block feature corresponding to the second key point is matched with the first image block feature corresponding to each first key point to obtain the first key point matched by the second key point.

[0129] The first key point matched with the second key point can be obtained by calculating the similarity between the second image block feature corresponding to the second key point and the first image block feature corresponding to each first key point. For example, the Euclidean distance between the second image block feature corresponding to the second key point and the first image block feature corresponding to each first key point can be calculated. The first key point whose corresponding Euclidean distance is less than the third preset distance is used as the first key point matched with the second key point. The specific value of the third preset distance can be determined according to actual conditions.

[0130] According to the above scheme of the embodiment of the present invention, the second key point matched by each first key point and the first key point matched by each second key point can be determined according to the first image block feature of the first key point and the second image block feature of the second key point. Since the first image block feature of the first key point integrates the image features around the first key point, and the second image feature of the second key point integrates the image features around the second key point, the first key point and the second key point can be matched more accurately, which is conducive to improving the accuracy of the defect type label corresponding to the image to be tested.

[0131] Illustratively, the detection method of the embodiment of the present invention may further include: step S310 to step S340.

[0132] In step S310, for each first key point, a first image feature corresponding to the first key point in the reference image and a first position feature of the first key point in the reference image are determined.

[0133] The first position feature can be obtained based on the first position information of the first key point. For example, the first position information of the first key point can be processed by fixed position embedding, learning position embedding, sine / cosine position encoding, absolute coordinate encoding, and relative position encoding to convert the first position information into the first position feature.

[0134] In step S320, for each first key point, the first image feature of the first key point and the first position feature of the first key point are feature spliced ​​to obtain the first key point feature.

[0135] A concat operator (concatenation operator) or an add operator (superposition operator) may be used to perform feature concatenation on the first image feature of the first key point and the first position feature of the first key point to obtain the first key point feature.

[0136] In step S330, for each second key point, a second image feature corresponding to the second key point in the image to be tested and a second position feature of the second key point in the image to be tested are determined.

[0137] The second position feature can be obtained based on the second position information of the second key point. For example, the second position information of the second key point can be processed by fixed position embedding, learning position embedding, sine / cosine position encoding, absolute coordinate encoding, and relative position encoding to convert the second position information into the second position feature.

[0138] In step S340, for each second key point, the second image feature of the second key point and the second position feature of the second key point are feature spliced ​​to obtain the second key point feature.

[0139] The concat operator (splicing operator) or the add operator (superposition operator) may be used to perform feature splicing on the second image feature of the second key point and the second position feature of the second key point to obtain the second key point feature.

[0140] According to the above scheme of the embodiment of the present invention, the splicing result of the first image feature of the first key point and the first position feature of the first key point can be used as the first key point feature, and the splicing result of the second image feature of the second key point and the second position feature of the second key point can be used as the second key point feature. Then, according to the first key point feature of the first key point and the second key point feature of the second key point, the similarity between the reference image and the image to be tested is determined. Since the first key point feature of the first key point integrates the position information of the first key point, and the second key point feature of the second key point integrates the position information of the second key point, the matching accuracy of the first key point and the second key point can be improved, which is conducive to improving the accuracy of the defect type label corresponding to the image to be tested.

[0141] Exemplarily, the query vector and key vector input to the first self-attention module in the first attention module are both the first key point features, and the value vector is the first image feature. The query vector and key vector input to the second self-attention module in the second attention module are both the second key point features, and the value vector is the second image feature.

[0142] In the first self-attention module, an attention weight distribution can be obtained by calculating the similarity between the query vector (Query) and the key vector (Key). The attention weight distribution and the value vector (Value) are weighted and summed to generate the first attention feature. For example, if there are 10 first key points in total, in the first self-attention module, for the first key point 1, the similarity score between the first key point feature 1 (query vector) and the first key point features 1 to 10 (key vector) can be calculated to obtain the calculation result. The calculation result is normalized by the softmax function to obtain the attention weight distribution. Then the attention weight distribution is used as the weight to weight the first image features 1 to 10 respectively, and then summed to obtain the first attention feature of the first key point feature 1. The processing flow of the second self-attention module can refer to the processing flow of the first self-attention module, and the embodiment of the present invention will not be repeated here. It should be understood that the processing method of the self-attention module, the query vector (Query), the key vector (Key), and the value vector (Value) can also be adjusted according to actual needs, and the embodiment of the present invention is not limited here. It should be understood that the value vector input to the first self-attention module may also be the first key point feature. The value vector input to the second self-attention module may also be the second key point feature.

[0143] The defect type detection method of the embodiment of the present invention can use the first image feature as the value vector of the first self-attention module, and use the second image feature as the value vector of the second self-attention module, thereby enhancing the feature expression strength of the first image feature of the first key point and the second image feature of the second key point, and reducing the feature expression strength of the remaining features of the first key point (such as the first position feature) and the remaining features of the second key point (such as the second position feature), which is conducive to improving the learning efficiency of the first self-attention module and the second self-attention module.

[0144] Exemplarily, for each first key point, the query vector input to the first cross-attention module in the first attention module is the first key point feature, the key vector is the second key point feature of the second key point matched by the first key point, and the value vector is the second image feature of the second key point matched by the first key point. For each second key point, the query vector input to the second cross-attention module in the second attention module is the second key point feature, the key vector is the first key point feature of the first key point matched by the second key point, and the value vector is the first image feature of the first key point matched by the second key point.

[0145] Regarding the processing method of the query vector, key vector and value vector, reference may be made to the processing method in the self-attention module, and the embodiments of the present invention will not be described in detail here.

[0146] It should be understood that the query vector input by the first cross attention module may also be the value vector of the first self-attention output. The query vector input by the second cross attention module may also be the value vector of the second self-attention output. In one example, for each first key point, the value vector input by the first cross attention module may also be the second key point feature of the second key point that matches the first key point. For each second key point, the value vector input by the second cross attention module may also be the first key point feature of the first key point that matches the second key point.

[0147] According to the above scheme of an embodiment of the present invention, the second image feature of the second key point matching the first key point can be used as the value vector of the first cross-attention module, and the first image feature of the first key point matching the second key point can be used as the value vector of the second cross-attention module, thereby enhancing the feature expression strength of the first image feature of the first key point and the second image feature of the second key point, and reducing the feature expression strength of the remaining features of the first key point (such as the first position feature) and the remaining features of the second key point (such as the second position feature), which is conducive to improving the learning efficiency of the first cross-attention module and the second cross-attention module.

[0148] The embodiment of the present invention also provides a device for detecting defect types of an image. Figure 4 FIG. 4 is a schematic block diagram of an image defect type detection device 400 according to an embodiment of the present invention. Figure 4 As shown, the detection device 400 may include: a defect area determination module 410 , a key point determination module 420 , a similarity determination module 430 and a label determination module 440 .

[0149] The defect area determination module 410 is used to determine the first defect area in the reference image of the reference object, and to determine the second defect area in the image to be tested of the object to be tested; wherein the reference object and the object to be tested are both wafers or wafer grains, and the reference image corresponds to the defect type label of the reference object. The key point determination module 420 is used to determine a plurality of first key points located in the first defect area in the reference image, and to determine a plurality of second key points located in the second defect area in the image to be tested. The similarity determination module 430 is used to determine the similarity between the reference image and the image to be tested based on the matching results of the first key points and the second key points. The label determination module 440 is used to determine that the defect type label corresponding to the image to be tested is the defect type label corresponding to the reference image if the similarity between the reference image and the image to be tested is higher than a preset similarity.

[0150] Exemplarily, the similarity determination module 430 may include: an enhanced feature generation module and an enhanced feature matching module.

[0151] The enhanced feature generation module is used to input the first key point feature of each first key point into the first attention module to obtain the first enhanced feature of the first key point, and input the second key point feature of each second key point into the second attention module to obtain the second enhanced feature of the second key point, wherein the first key point feature is obtained based on the image feature of the first key point in the reference image, and the second key point feature is obtained based on the image feature of the second key point in the image to be tested. The enhanced feature matching module is used to match the first enhanced feature with the second enhanced feature to determine the similarity between the reference image and the image to be tested.

[0152] Exemplarily, the first attention module includes a first self-attention module and a first cross-attention module, the second attention module includes a second self-attention module and a second cross-attention module, the first cross-attention module is connected to the first self-attention module and the second self-attention module, and the second cross-attention module is connected to the first self-attention module and the second self-attention module.

[0153] The enhanced feature generation module may include: a first matching module, a first attention feature generation module, a first enhanced feature generation module, a second matching module, a second attention feature generation module and a second enhanced feature generation module.

[0154] The first matching module is used to determine the second key point matched by the first key point for each first key point. The first attention feature generation module is used to input the first key point feature of the first key point into the first self-attention module for each first key point to obtain the first attention feature of the first key point. The first enhanced feature generation module is used to input the first attention feature of the first key point and the second attention feature of the second key point matched by the first key point into the first cross-attention module for each first key point to obtain the first enhanced feature of the first key point. The second matching module is used to determine the first key point matched by the second key point for each second key point. The second attention feature generation module is used to input the second key point feature of the second key point into the second self-attention module for each second key point to obtain the second attention feature of the second key point. The second enhanced feature generation module is used to input the second attention feature of the second key point and the first attention feature of the first key point matched by the second key point into the second cross-attention module for each second key point to obtain the second enhanced feature of the second key point.

[0155] Exemplarily, the detection device 400 may further include an image segmentation module.

[0156] The image division module is used to divide the reference image and the image to be tested into a plurality of initial image blocks respectively.

[0157] The first matching module may include: a first image block feature determination module and a first image block matching module.

[0158] The first image block feature determination module is used to determine, for each first key point, a first image block feature corresponding to the first key point based on the first position information of the first key point, wherein the first image block feature is an image feature of an initial image block of the reference image where the first key point is located. The first image block matching module is used to match, for each first key point, the first image block feature corresponding to the first key point with the second image block feature corresponding to each second key point, respectively, to obtain a second key point matched with the first key point;

[0159] The second matching module may include: a second image block feature determination module and a second image block matching module.

[0160] The second image block feature determination module is used to determine, for each second key point, the second image block feature corresponding to the second key point based on the second position information of the second key point, wherein the second image block feature is the image feature of the initial image block of the image to be tested where the second key point is located. The second image block matching module is used to match, for each second key point, the second image block feature corresponding to the second key point with the first image block feature corresponding to each first key point, respectively, to obtain the first key point matched by the second key point.

[0161] Exemplarily, the detection device 400 may further include: a first position determination module, a first splicing module, a second position determination module and a second splicing module.

[0162] The first position determination module is used to determine, for each first key point, the first image feature corresponding to the first key point in the reference image and the first position feature of the first key point in the reference image, wherein the first position feature is obtained based on the first position information of the first key point. The first splicing module is used to perform feature splicing on the first image feature of the first key point and the first position feature of the first key point for each first key point to obtain the first key point feature. The second position determination module is used to determine, for each second key point, the second image feature corresponding to the second key point in the image to be tested and the second position feature of the second key point in the image to be tested, wherein the second position feature is obtained based on the second position information of the second key point. The second position determination module is used to perform feature splicing on the second image feature of the second key point and the second position feature of the second key point for each second key point to obtain the second key point feature.

[0163] Exemplarily, the query vector and key vector input to the first self-attention module in the first attention module are both the first key point features, and the value vector is the first image feature. The query vector and key vector input to the second self-attention module in the second attention module are both the second key point features, and the value vector is the second image feature.

[0164] Exemplarily, for each first key point, the query vector input to the first cross-attention module in the first attention module is the first key point feature of the first key point, the key vector is the second key point feature of the second key point matched by the first key point, and the value vector is the second image feature of the second key point matched by the first key point. For each second key point, the query vector input to the second cross-attention module in the second attention module is the second key point feature of the second key point, the key vector is the first key point feature of the first key point matched by the second key point, and the value vector is the first image feature of the first key point matched by the second key point.

[0165] Exemplarily, the key point determination module 420 may include: a first extraction module and a second extraction module.

[0166] The first extraction module is used to input the reference image and the first defect area into the key point extraction network to obtain the first position information of each first key point and the first image feature of each first key point. The second extraction module is used to input the image to be tested and the second defect area into the key point extraction network to obtain the second position information of each second key point and the second image feature of each second key point.

[0167] According to another aspect of the present invention, an electronic device is also provided. Figure 5 FIG. 5 shows a schematic block diagram of an electronic device 500 according to an embodiment of the present invention. Figure 5 As shown, the electronic device 500 includes a processor 510 and a memory 520. The memory 520 stores a computer program. When the computer program instructions are executed by the processor 510, they are used to execute the above-mentioned image defect type detection method.

[0168] In addition, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned image defect type detection method according to the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned image defect type detection device according to the embodiment of the present invention or the corresponding modules in the above-mentioned electronic device. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above-mentioned storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media. According to another aspect of the present invention, a computer program product is provided, including computer program instructions, and when the above-mentioned computer program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned image defect type detection method.

[0169] A person skilled in the art may understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the above-mentioned description of the defect type detection method for an image, and for the sake of brevity, it will not be described in detail here.

[0170] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0173] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0174] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0175] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0176] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0177] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the image defect type detection device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0178] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0179] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting defect types of an image, characterized in that: The method comprises: Determine a first defect area in a reference image of a reference object, and determine a second defect area in an image to be tested of an object to be tested; wherein the reference object and the object to be tested are both wafers or wafer dies, and the reference image corresponds to a defect type label of the reference object; Determine a plurality of first key points located in the first defect region in the reference image, and determine a plurality of second key points located in the second defect region in the image to be tested; Determining the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point; If the similarity between the reference image and the image to be tested is higher than a preset similarity, it is determined that the defect type label corresponding to the image to be tested is the defect type label corresponding to the reference image.

2. The method according to claim 1, characterized in that The determining the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point includes: For each first key point, inputting a first key point feature of the first key point into a first attention module to obtain a first enhanced feature of the first key point, and for each second key point, inputting a second key point feature of the second key point into a second attention module to obtain a second enhanced feature of the second key point, wherein the first key point feature is obtained based on an image feature of the first key point in the reference image, and the second key point feature is obtained based on an image feature of the second key point in the image to be tested; The first enhanced feature and the second enhanced feature are matched to determine the similarity between the reference image and the image to be tested.

3. The method according to claim 2, characterized in that The first attention module includes a first self-attention module and a first cross-attention module, the second attention module includes a second self-attention module and a second cross-attention module, the first cross-attention module is connected to the first self-attention module and the second self-attention module, and the second cross-attention module is connected to the first self-attention module and the second self-attention module, For each first key point, inputting the first key point feature of the first key point into the first attention module to obtain the first enhanced feature of the first key point, including: For each first key point, Determine a second key point that matches the first key point; Inputting the first key point feature of the first key point into the first self-attention module to obtain a first attention feature of the first key point; Inputting a first attention feature of the first key point and a second attention feature of a second key point matching the first key point into the first cross attention module to obtain a first enhanced feature of the first key point; For each second key point, inputting the second key point feature of the second key point into the second attention module to obtain the second enhanced feature of the second key point, including: For each second key point, Determine the first key point that matches the second key point; Inputting the second key point feature of the second key point into the second self-attention module to obtain a second attention feature of the second key point; The second attention feature of the second key point and the first attention feature of the first key point matching the second key point are input into the second cross-attention module to obtain the second enhanced feature of the second key point.

4. The method according to claim 3, characterized in that The method further comprises: Dividing the reference image and the image to be tested into a plurality of initial image blocks respectively; The step of determining, for each first key point, a second key point that matches the first key point comprises: For each first key point, Determine, based on the first position information of the first key point, a first image block feature corresponding to the first key point, wherein the first image block feature is an image feature of an initial image block of the reference image where the first key point is located; Matching the first image block feature corresponding to the first key point with the second image block feature corresponding to each second key point respectively, to obtain the second key point matched with the first key point; The step of determining, for each second key point, the first key point that matches the second key point comprises: For each second key point, Based on the second position information of the second key point, determining a second image block feature corresponding to the second key point, wherein the second image block feature is an image feature of an initial image block of the image to be tested where the second key point is located; The second image block feature corresponding to the second key point is matched with the first image block feature corresponding to each first key point to obtain the first key point matched by the second key point.

5. The method according to claim 3, characterized in that The method further comprises: For each first key point, Determine a first image feature corresponding to the first key point in the reference image and a first position feature of the first key point in the reference image, wherein the first position feature is obtained based on first position information of the first key point; Performing feature splicing on the first image feature of the first key point and the first position feature of the first key point to obtain a first key point feature; For each second key point, Determine a second image feature corresponding to the second key point in the image to be tested and a second position feature of the second key point in the image to be tested, wherein the second position feature is obtained based on second position information of the second key point; The second image feature of the second key point and the second position feature of the second key point are feature spliced ​​to obtain the second key point feature.

6. The method according to claim 5, characterized in that The query vector and key vector input to the first self-attention module in the first attention module are both first key point features, and the value vector is the first image feature; the query vector and key vector input to the second self-attention module in the second attention module are both second key point features, and the value vector is the second image feature.

7. The method according to claim 5, characterized in that For each first key point, the query vector input by the first cross attention module in the first attention module is the first key point feature of the first key point, the key vector is the second key point feature of the second key point matched by the first key point, and the value vector is the second image feature of the second key point matched by the first key point; For each second key point, the query vector input to the second cross-attention module in the second attention module is the second key point feature of the second key point, the key vector is the first key point feature of the first key point matched by the second key point, and the value vector is the first image feature of the first key point matched by the second key point.

8. The method according to claim 1, characterized in that The step of determining a plurality of first key points located in the first defect region in the reference image, and determining a plurality of second key points located in the second defect region in the image to be tested, comprises: Inputting the reference image and the first defect area into a key point extraction network to obtain first position information of each first key point and first image features of each first key point; The image to be tested and the second defect area are input into the key point extraction network to obtain the second position information of each second key point and the second image feature of each second key point.

9. A device for detecting defect types of an image, characterized in that: The device comprises: A defect area determination module, used to determine a first defect area in a reference image of a reference object, and to determine a second defect area in an image to be tested of an object to be tested; wherein the reference object and the object to be tested are both wafers or wafer dies, and the reference image corresponds to a defect type label of the reference object; a key point determination module, configured to determine a plurality of first key points located in the first defect region in the reference image, and to determine a plurality of second key points located in the second defect region in the image to be tested; A similarity determination module, used to determine the similarity between the reference image and the image to be tested based on the matching result of the first key point and the second key point; The label determination module is used to determine that the defect type label corresponding to the image to be tested is the defect type label corresponding to the reference image if the similarity between the reference image and the image to be tested is higher than a preset similarity.

10. An electronic device, characterized in that: It comprises a memory and a processor, wherein: the memory is used to store a computer program; the processor is used to execute the computer program to implement the image defect type detection method as described in any one of claims 1-8.

11. A storage medium storing a computer program / instruction, characterized in that: The computer program / instructions are used to execute the image defect type detection method as claimed in any one of claims 1 to 8 when running.

12. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the image defect type detection method according to any one of claims 1 to 8 when being run.

Images