Defect classification method and apparatus, and electronic device and storage medium

Abstract

Provided in the embodiments of the present application are a defect classification method and apparatus, and an electronic device and a storage medium. The method comprises: inputting an image to be subjected to detection and first position prompt information into a feature extraction network of a defect classification model, so as to obtain an image feature of said image and a position identification feature, wherein the first position prompt information is used for indicating the position of a defect area in said image; and inputting the image feature and the position identification feature into a fully connected neural network of the defect classification model, so as to obtain a defect classification result, wherein the defect classification result is used for indicating the category to which the defect area in said image belongs, and the defect classification model is an open-world semi-supervised learning model. By means of the solution, the interference of other areas in said image except the defect area on the defect classification result can be reduced, thereby improving the detection efficiency and the accuracy of the defect classification result.

Description

Defect classification method and device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on December 26, 2023, with application number 202311802922.X and application name "Defect Classification Method and Device, Electronic Device and Storage Medium", all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and more specifically to a defect classification method, a defect classification device, an electronic device, and a storage medium. Background Art

[0003] During the semiconductor manufacturing process, semiconductors need to be defect-detected to determine whether the semiconductors have defects and the types of defects. Taking semiconductor wafers as an example, in the prior art, wafers are usually inspected using automated optical inspection, and the manufacturing process is optimized based on the types of defects detected. However, there are many types of wafer defects, and for different types of wafers or for wafers in different process sections, the morphology of the defects is often quite different. At this time, it is necessary to use manual inspection or a deep learning classification algorithm with fixed categories to determine whether there are defects in the wafer and the types of defects. The cost of manual inspection is not only relatively high, but also relatively low in efficiency. The deep learning classification algorithm with fixed categories requires re-acquisition of labeled data and training of the classification model. This method cannot effectively detect new defect types. Summary of the Invention

[0004] In view of the above problems, the present application is proposed. The present application provides a defect classification method, a defect classification device, an electronic device and a storage medium.

[0005] A first aspect of the present application provides a defect classification method, the method comprising:

[0006] The image to be tested and the first position prompt information are input into the feature extraction network of the defect classification model to obtain image features and position identification features of the image to be tested, wherein the first position prompt information is used to indicate the location of the defect area in the image to be tested; the image features and position identification features are input into the fully connected neural network of the defect classification model to obtain a defect classification result, and the defect classification result is used to indicate the category to which the defect area in the image to be tested belongs; wherein the defect classification model is an open-world semi-supervised classification model.

[0007] In one possible implementation, the defect classification model further includes an attention network. Before inputting the image features and the location identification features into the fully connected neural network of the defect classification model to obtain the defect classification result, the method further includes:

[0008] Inputting the image features and the location identification features into the attention network for feature fusion to obtain fusion features; inputting the image features and the location identification features into the fully connected neural network of the defect classification model to obtain defect classification results, including: inputting the fusion features into the fully connected neural network to obtain defect classification results.

[0009] In a possible implementation, the first position prompt information includes: target frame prompt information corresponding to a target detection frame containing the defect area, and / or mask prompt information corresponding to a mask of the defect area, and / or marker point prompt information corresponding to one or more marker points within the defect area;

[0010] The position identification features include: target frame coding features corresponding to the target detection frame containing the defect area, and / or mask coding features corresponding to the mask of the defect area, and / or identification point coding features corresponding to one or more identification points in the defect area.

[0011] In one possible implementation, the feature extraction network includes an image encoding module and a position encoding module, and inputs the image to be tested and the first position prompt information into the feature extraction network of the defect classification model to obtain image features and position identification features of the image to be tested, including:

[0012] The image to be tested is input into the image coding module to obtain image features; the first position prompt information is input into the position coding module to obtain position identification features.

[0013] In a possible implementation, the first position prompt information includes: target frame prompt information corresponding to a target detection frame containing the defect area, and / or mask prompt information corresponding to a mask of the defect area, and / or marker point prompt information corresponding to one or more marker points within the defect area;

[0014] The position identification features include: target frame coding features corresponding to the target detection frame containing the defect area, and / or mask coding features corresponding to the mask of the defect area, and / or identification point coding features corresponding to one or more identification points in the defect area;

[0015] The position encoding module includes: a target frame encoder, and / or a mask encoder, and / or a marker encoder;

[0016] Inputting the first position prompt information into a position encoding module to obtain a position identification feature, including: inputting the target frame prompt information into a target frame encoder to obtain a target frame encoding feature;

[0017] and / or, inputting the mask prompt information into a mask encoder to obtain mask encoding features;

[0018] And / or, inputting the marker point prompt information into the marker point encoder to obtain the marker point coding feature.

[0019] In a possible implementation, the image encoding module is trained using an unsupervised training method. When training the defect classification model, the parameters of the image encoding module remain fixed, and the parameters of the position encoding module are trained.

[0020] In one possible implementation, the defect classification model is obtained through the following training operations:

[0021] Obtaining a sample data set, where the sample data set includes labeled data and unlabeled data, the labeled data including a plurality of first sample images and second position prompt information and a label corresponding to each first sample image, the label being used to indicate a category to which a defective area in the corresponding first sample image belongs, and the unlabeled data including a plurality of second sample images and third position prompt information corresponding to each second sample image;

[0022] Perform the following model training operations based on the sample data set: train the defect classification model using the labeled data to obtain an initially trained defect classification model;

[0023] Inputting the plurality of second sample images and the third position prompt information corresponding to each of the plurality of second sample images into the initially trained defect classification model to obtain a first prediction classification result corresponding to each of the plurality of second sample images, the first prediction classification result being used to indicate the category to which the defect area in the corresponding second sample image belongs;

[0024] Obtaining pseudo labels corresponding to at least some of the second sample images in the plurality of second sample images based on the first prediction classification results corresponding to the plurality of second sample images;

[0025] Inputting the plurality of first sample images and the second position prompt information corresponding to each of the plurality of first sample images into the initially trained defect classification model to obtain a second prediction classification result corresponding to each of the plurality of first sample images, the second prediction classification result being used to indicate the category to which the defect area in the corresponding first sample image belongs;

[0026] inputting at least a portion of the second sample image and third position prompt information corresponding to at least a portion of the second sample image into the initially trained defect classification model to obtain a third predicted classification result corresponding to at least a portion of the second sample image, the third predicted classification result being used to indicate a category to which the defect area in the corresponding second sample image belongs;

[0027] Calculating a prediction loss value based on a difference between the annotation labels corresponding to each of the plurality of first sample images and the second predicted classification result, and a difference between the pseudo labels corresponding to at least some of the second sample images and the third predicted classification result;

[0028] The parameters of the initially trained defect classification model are optimized based on the predicted loss value to obtain a trained defect classification model corresponding to the current round of training operations.

[0029] In a possible implementation, the image to be tested includes a plurality of first images to be tested, and the training operation further includes:

[0030] After the current round of training operation is completed, a fourth predicted classification result corresponding to each of the at least one test image is obtained by prediction of the trained defect classification model, wherein the fourth predicted classification result is used to indicate a category to which a defect region in the corresponding test image belongs, and the at least one test image includes one or more of the following: at least a portion of the plurality of second sample images; an image to be tested; or a new image other than the plurality of second sample images and the image to be tested;

[0031] Outputting the fourth predicted classification result corresponding to each of the at least one test image; and performing corresponding operations on the fourth predicted classification result and / or the test image in response to verification information input by the user.

[0032] In one possible implementation, in response to the verification information input by the user, performing corresponding operations on the fourth predicted classification result and / or the test image includes one or more of the following:

[0033] For any test image classified as a known category based on the fourth predicted classification result, if the verification information includes deletion information about the test image, deleting the test image, wherein the known category is the category indicated by the annotation label in the annotated data;

[0034] For any new category divided in the fourth predicted classification result, if the verification information includes deletion information about the new category, deleting the new category from the fourth predicted classification result, wherein the new category is a category different from the known category;

[0035] For any new category divided in the fourth predicted classification result, if the verification information includes merging information about the new category, then the new category and the known category specified by the merging information are merged into the same category;

[0036] For any new category divided in the fourth predicted classification result, if the verification information includes additional information about the new category, the new category and the test image corresponding to the new category are added to the labeled data.

[0037] In a possible implementation, the image to be tested is a wafer image including a wafer, and the defective area is a defective area on the wafer.

[0038] A second aspect of the present application further provides a defect classification device, the device comprising:

[0039] The first input module is used to input the image to be tested and the first position prompt information into the feature extraction network of the defect classification model to obtain the image features and position identification features of the image to be tested, wherein the first position prompt information is used to indicate the location of the defect area in the image to be tested; the second input module is used to input the image features and position identification features into the fully connected neural network of the defect classification model to obtain the defect classification result, and the defect classification result is used to indicate the category to which the defect area in the image to be tested belongs; wherein the defect classification model is an open-world semi-supervised classification model.

[0040] The third aspect of the present application further provides an electronic device, which includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned defect classification method when the processor is executed.

[0041] A fourth aspect of the present application further provides a storage medium storing a computer program / instruction, wherein the computer program / instruction is used to execute the above-mentioned defect classification method when running.

[0042] According to the defect classification method, defect classification device, electronic device and storage medium of the embodiments of the present application, the image features and position identification features of the image to be tested can be obtained by inputting the image to be tested into the feature extraction network of the defect classification model. The obtained image features and position identification features are input into the fully connected neural network of the defect classification model to obtain the defect classification result. By obtaining the position identification features of the image to be tested, this scheme can reduce the interference of other areas in the image to be tested except the defect area on the defect classification result, thereby improving the detection efficiency and the accuracy of the defect classification result. At the same time, by detecting the category to which the defect area in the image to be tested belongs through the defect classification model in this scheme, unknown new categories can also be discovered. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0044] FIG1 shows a schematic flow chart of a defect classification method according to an embodiment of the present application;

[0045] FIG2 shows a schematic diagram of obtaining fusion features according to an embodiment of the present application;

[0046] FIG3 is a schematic diagram showing a defect classification model training process according to an embodiment of the present application;

[0047] FIG4 is a schematic diagram showing a defect classification model training process according to another embodiment of the present application;

[0048] FIG5 shows a schematic block diagram of a defect classification device according to an embodiment of the present application;

[0049] FIG6 shows a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application more apparent, the following is a detailed description of example embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0051] To at least partially address the above issues, an embodiment of the present application provides a defect classification method. FIG1 shows a schematic flow chart of a defect classification method 100 according to an embodiment of the present application. As shown in FIG1 , the method 100 may include the following steps S110 and S120.

[0052] In step S110, the image to be tested and the first position prompt information are input into a feature extraction network of a defect classification model to obtain image features and position identification features of the image to be tested, wherein the defect classification model is an open-world semi-supervised classification model, and the first position prompt information is used to indicate the location of the defect area in the image to be tested.

[0053] For example, the image to be tested can be any type of image containing a target object. The target object can be any type of item, such as a wafer, a character, an electronic component, etc. The image to be tested can be a static image or any frame in a dynamic video. The image to be tested can be an original image captured by an image acquisition device (such as an image sensor in a camera), or an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.).

[0054] For example, the image to be tested is a wafer image containing a wafer, and the defect region is a defect region on the wafer. In one embodiment of the present application, the target object may be a wafer. The image to be tested may be an image containing the wafer. The defect region may indicate the location of the defect on the wafer. This allows defects on the wafer to be classified, so that corresponding operations can be performed for different defect categories.

[0055] The first position prompt information can indicate the location of the defect area in the image to be tested. The first position prompt information can be manually marked, or it can be obtained after the defect area in the image to be tested is detected by a defect detection model. For example, the first position prompt information can be any type of prompt information such as a detection box (Bounding Box), a mask (mask), a position point (point) corresponding to the defect area, and this application is not limited to this. The image to be tested and the first position prompt information are input into the feature extraction network of the defect classification model to obtain the image features and position identification features of the image to be tested. The defect classification model is an open-world semi-supervised classification (Open-World Semi-Supervised Learning, OWSSL) model, for example, Towards Realistic Semi-Supervised Learning (TRSSL). The image to be tested and the first position prompt information are input into the feature extraction network of the defect classification model to obtain the image features and position identification features of the image to be tested. Exemplarily and non-restrictively, the feature extraction network can be implemented using a vision transformer (ViT) or a convolutional neural network backbone (CNN backbone).

[0056] Exemplarily, the first position prompt information may include: target frame prompt information corresponding to the target detection frame containing the defect area, and / or mask prompt information corresponding to the mask of the defect area, and / or identification point prompt information corresponding to one or more identification points within the defect area; the position identification feature may include: target frame coding features corresponding to the target detection frame containing the defect area, and / or mask coding features corresponding to the mask of the defect area, and / or identification point coding features corresponding to one or more identification points within the defect area.

[0057] In one embodiment, the first position prompt information may include: any one or more of: target frame prompt information corresponding to the target detection frame containing the defect area, mask prompt information corresponding to the mask of the defect area, and identification point prompt information corresponding to one or more identification points within the defect area.

[0058] For example, the first position prompt information may include target frame prompt information corresponding to the target detection frame containing the defect area, or the first position prompt information may include mask prompt information corresponding to the mask of the defect area, or the first position prompt information may include identification point prompt information corresponding to one or more identification points within the defect area.

[0059] For another example, the first position prompt information may include target frame prompt information corresponding to the target detection frame containing the defect area, mask prompt information corresponding to the mask of the defect area, and identification point prompt information corresponding to one or more identification points in the defect area. Exemplarily, the target frame prompt information can be presented as an circumscribed rectangular frame of the defect area. Exemplarily, the mask prompt information can be presented through a binary image. In the binary image, each pixel in the defect area can be highlighted. The identification point prompt information can be presented through the image coordinates of one or more identification points in the image to be tested. The identification point can be any representative pixel point in the defect area.

[0060] The position identification feature may include: any one or more of: a target frame coding feature corresponding to a target detection frame containing a defective area, a mask coding feature corresponding to a mask of a defective area, and a marking point coding feature corresponding to one or more marking points within the defective area. The type of the position identification feature corresponds to the type of the first position prompt information. That is, when the first position prompt information includes target frame prompt information corresponding to a target detection frame containing a defective area, mask prompt information corresponding to a mask of a defective area, and marking point prompt information corresponding to one or more marking points within the defective area, the position identification feature may include a target frame coding feature corresponding to a target detection frame containing a defective area, a mask coding feature corresponding to a mask of a defective area, and a marking point coding feature corresponding to one or more marking points within the defective area.

[0061] According to the above technical solution, through the target frame prompt information corresponding to the target detection frame containing the defect area, the mask prompt information corresponding to the mask of the defect area, and any one or more first position prompt information in the identification point prompt information corresponding to one or more identification points in the defect area and the corresponding position identification features, supervision information that is helpful for the training of the defect classification model can be provided to reduce the interference of irrelevant areas on the training of the defect classification model and improve the training efficiency of the defect classification model.

[0062] In step S120 , the image features and the position identification features are input into a fully connected neural network of a defect classification model to obtain a defect classification result. The defect classification result is used to indicate the category to which the defect area in the image to be tested belongs.

[0063] Exemplarily, the defect classification model may include a fully connected neural network. By inputting the image features and the position identification features into the fully connected neural network of the defect classification model, a defect classification result can be obtained. The defect classification result may indicate the category to which the defective area in the image to be tested belongs. The category to which the defective area belongs may include but is not limited to scratches, missing parts or particulate foreign matter. For example, if the position identification feature is a target frame coding feature, then the image features and the target frame coding features are input into the fully connected neural network to obtain a defect classification result. The defect classification result may include the target frame and confidence corresponding to the defective area in the image to be tested. Through the confidence corresponding to different categories, the category corresponding to the maximum confidence can be determined as the category to which the defective area belongs.

[0064] According to the defect classification method of the embodiment of the present application, the image features and position identification features of the image to be tested can be obtained by inputting the image to be tested into the feature extraction network of the defect classification model. The obtained image features and position identification features are input into the fully connected neural network of the defect classification model to obtain the defect classification result. By obtaining the position identification features of the image to be tested, this solution can reduce the interference of other areas in the image to be tested except the defect area on the defect classification result, thereby improving the detection efficiency and the accuracy of the defect classification result. At the same time, by detecting the category to which the defect area in the image to be tested belongs through the defect classification model in this solution, unknown new categories can also be discovered.

[0065] Exemplarily, the defect classification model may further include an attention network. Before inputting the image features and location identification features into the fully connected neural network of the defect classification model to obtain a defect classification result, the method may further include: inputting the image features and location identification features into the attention network for feature fusion to obtain a fusion feature; inputting the image features and location identification features into the fully connected neural network of the defect classification model to obtain a defect classification result, including: inputting the fusion feature into the fully connected neural network to obtain a defect classification result.

[0066] In one embodiment, the defect classification model may also include an attention network. Image features and location marker features are input into the attention network for feature fusion. Through the attention mechanism, fused features corresponding to the image and location marker features are obtained. The obtained fused features are then input into a fully connected neural network to obtain the defect classification result.

[0067] According to the above technical solution, image features and location marker features can be fused through an attention network, so that the obtained fused features contain both image features and location marker features. This can prompt the defect classification model to focus on the area corresponding to the location marker feature, reducing the interference of the defect classification results in irrelevant areas.

[0068] Exemplarily, the feature extraction network may include an image encoding module and a position encoding module. Inputting the image to be tested and the first position prompt information into the feature extraction network of the defect classification model to obtain image features and position identification features of the image to be tested may include: inputting the image to be tested into the image encoding module to obtain image features; inputting the first position prompt information into the position encoding module to obtain position identification features.

[0069] In one embodiment, the feature extraction network may include an image encoding module and a position encoding module. The image to be tested is input into the image encoding module to obtain image features. By way of example and not limitation, image encoding may be implemented using methods such as convolutional neural networks (CNNs) or vision transformers (ViTs).

[0070] In one embodiment of the present application, a convolutional neural network can be used to obtain image features corresponding to the image to be tested. The first position prompt information is input into a position encoding module to obtain a position identification feature. By way of example and not limitation, position encoding can include any one of conditional position encoding, learnable absolute position encoding, sine and cosine function encoding, and relative position encoding. In one embodiment, the two-dimensional position coordinates in the first position prompt information can be position-encoded using sine and cosine functions to obtain their corresponding position identification features.

[0071] According to the above technical solution, the image to be tested is input into the image encoding module to obtain image features. The first position prompt information is input into the position encoding module to obtain position identification features. This ensures the efficiency and accuracy of obtaining image features and position coding features.

[0072] Exemplarily, the first position prompt information includes: target frame prompt information corresponding to the target detection frame containing the defect area, and / or mask prompt information corresponding to the mask of the defect area, and / or identification point prompt information corresponding to one or more identification points in the defect area; the position identification feature includes: target frame coding feature corresponding to the target detection frame containing the defect area, and / or mask coding feature corresponding to the mask of the defect area, and / or identification point coding feature corresponding to one or more identification points in the defect area; the position coding module includes: a target frame encoder, and / or a mask encoder, and / or an identification point encoder; inputting the first position prompt information into the position coding module to obtain the position identification feature includes: inputting the target frame prompt information into the target frame encoder to obtain the target frame coding feature; inputting the mask prompt information into the mask encoder to obtain the mask coding feature; inputting the identification point prompt information into the identification point encoder to obtain the identification point coding feature.

[0073] In one embodiment, the first position prompt information and the position identification feature have been described in detail in the previous embodiment and will not be repeated here for the sake of brevity. The position encoding module may include any one or more of a target frame encoder, a mask encoder, and an identification point encoder. For example, when the first position prompt information includes target frame prompt information and mask prompt information, the position encoding module may include a target frame encoder and a mask encoder. For another example, when the first position prompt information only includes target frame prompt information, the position encoding module includes a target frame encoder.

[0074] In one embodiment of the present application, the target frame encoder can encode the target frame position information by using a fixed sin-cos position embedding method. The mask encoder can use convolution to embed the mask information. By inputting different types of position prompt information into the corresponding encoder, the corresponding position identification features can be obtained. By inputting the target frame prompt information into the target frame encoder, the target frame coding features can be obtained. Similarly, by inputting the mask prompt information into the mask encoder, the mask coding features can be obtained; by inputting the marker point prompt information into the marker point encoder, the marker point coding features can be obtained.

[0075] According to the above technical solution, the coding features corresponding to different types of position prompt information are obtained through different encoders, which can ensure the accuracy of the obtained coding features.

[0076] FIG2 shows a schematic diagram of obtaining fusion features according to an embodiment of the present application. As shown in FIG2 , the image coding features corresponding to the image to be tested can be obtained by inputting the image to be tested into the image coding module. The target frame prompt information in the first position prompt information corresponding to the image to be tested is input into the target frame encoder to obtain the target frame coding features. The mask prompt information in the first position prompt information corresponding to the image to be tested is input into the mask encoder to obtain the mask coding features. The obtained image coding features, target frame coding features, and mask coding features are input into the attention network to obtain fusion features.

[0077] Exemplarily, the image encoding module is trained using an unsupervised training method. When training the defect classification model, the parameters of the image encoding module remain fixed, and the parameters of the position encoding module are trained.

[0078] In one embodiment, the image encoding module can be trained using an unsupervised training method. For example, Emerging Properties in Self-Supervised Vision Transformers (DINO) and Learning Robust Visual Features without Supervision (DINOv2) are used. When training the defect classification model, the parameters of the image encoding module remain fixed, and only branches such as the position encoding module and the fully connected neural network are trained. This reduces the training overhead when training the defect classification model, which can improve the training efficiency of the defect classification model.

[0079] Exemplarily, the defect classification model is obtained by the following training operation: obtaining a sample data set, the sample data set including labeled data and unlabeled data, the labeled data including a plurality of first sample images and second position prompt information and a label corresponding to each first sample image, the label being used to indicate the category to which the defect area in the corresponding first sample image belongs, and the unlabeled data including a plurality of second sample images and third position prompt information corresponding to each second sample image;

[0080] Performing the following model training operations based on the sample data set: training the defect classification model using the labeled data to obtain an initially trained defect classification model; inputting the plurality of second sample images and the third position prompt information corresponding to each of the plurality of second sample images into the initially trained defect classification model to obtain a first prediction classification result corresponding to each of the plurality of second sample images, the first prediction classification result being used to indicate the category to which the defect area in the corresponding second sample image belongs;

[0081] obtaining pseudo labels corresponding to at least some of the second sample images in the plurality of second sample images based on the first predicted classification results corresponding to the plurality of second sample images; inputting the plurality of first sample images and the second position prompt information corresponding to the plurality of first sample images into the initially trained defect classification model to obtain second predicted classification results corresponding to the plurality of first sample images, the second predicted classification results being used to indicate the category to which the defect area in the corresponding first sample image belongs;

[0082] Input at least part of the second sample image and the third position prompt information corresponding to at least part of the second sample image into the initially trained defect classification model to obtain a third predicted classification result corresponding to at least part of the second sample image, and the third predicted classification result is used to indicate the category to which the defect area in the corresponding second sample image belongs; based on the difference between the labeled labels corresponding to each of the multiple first sample images and the second predicted classification result and the difference between the pseudo labels corresponding to at least part of the second sample image and the third predicted classification result, calculate the prediction loss value; based on the prediction loss value, optimize the parameters in the initially trained defect classification model to obtain a trained defect classification model corresponding to this round of training operation.

[0083] In one embodiment, the sample data set may include labeled data and unlabeled data. The labeled data may include multiple first sample images and the second position prompt information and label corresponding to each first sample image. The method for acquiring the first sample images is similar to the method for acquiring the image to be tested. The method for acquiring the image to be tested has been described in detail in step S110 and will not be repeated here for the sake of brevity.

[0084] Each first sample image has corresponding second location information and a label. Similar to the first location information, the second location information may include any one or more of label box location information, mask information, and marker information corresponding to the defect area contained in the first sample image. The label may be used to indicate the category of the defect area in the corresponding first sample image.

[0085] The unlabeled data may include multiple second sample images and third position prompt information corresponding to each second sample image. The method for acquiring the second sample images is similar to the method for acquiring the images to be tested. The method for acquiring the images to be tested has been described in detail in step S110 and will not be repeated here for the sake of brevity. The multiple second sample images may include images obtained by performing multiple data enhancements on the initial second sample image, as well as the initial second sample image.

[0086] Methods for implementing data augmentation include, but are not limited to, flipping, rotating, and cropping the initial second sample images. For example, if the number of initial second sample images U is 100, performing data augmentation twice on the initial second sample images U can yield an initial second sample image U1 after the first data augmentation and an initial second sample image U2 after the second data augmentation. The plurality of second sample images may include the initial second sample image U, the initial second sample image U1 after the first data augmentation, and the initial second sample image U2 after the second data augmentation.

[0087] The third position prompt information is similar to the first position prompt information and the second position prompt information in the previous embodiment. For the sake of brevity, it will not be repeated here. Figure 3 shows a schematic diagram of defect classification model training according to an embodiment of the present application. By training the defect classification model using labeled data, an initially trained defect classification model can be obtained. As shown in Figure 3, multiple second sample images and the third position prompt information corresponding to each second sample image (i.e., unlabeled data) are input into the initially trained defect classification model, and the first predicted classification results corresponding to each of the multiple second sample images can be obtained.

[0088] The first predicted classification result can be used to indicate the category to which the defect area in the corresponding second sample image belongs. Based on the first predicted classification results corresponding to each of the multiple second sample images, the pseudo labels corresponding to all or part of the second sample images are obtained. Constraints are added to the pseudo labels corresponding to each of the obtained second sample images. For example, the Sinkhorn-Knopp algorithm can be used to add constraints to the pseudo labels corresponding to each of the obtained second sample images. Similarly, by inputting the multiple first sample images and the second position prompt information corresponding to each of the multiple first sample images into the initially trained defect classification model, the second predicted classification results corresponding to each of the multiple first sample images can be obtained.

[0089] The second predicted classification result can indicate the category to which the defect region in the corresponding first sample image belongs. By again inputting the obtained pseudo-labeled at least portion of the second sample image and the third position prompt information corresponding to at least portion of the second sample image (i.e., the pseudo-labeled data shown in FIG3 ) into the initially trained defect classification model, a third predicted classification result corresponding to at least portion of the second sample image can be obtained.

[0090] The third predicted classification result can indicate the category to which the defect area in the corresponding second sample image belongs. The predicted classification results shown in Figure 3 may include the third predicted classification result and the second predicted classification result. "ema" represents the sliding average of the defect classification model weights. Based on the differences between the annotation labels corresponding to each of the multiple first sample images and the second predicted classification result, and the differences between the pseudo labels corresponding to at least some of the second sample images and the third predicted classification result, a predicted loss value can be calculated using a preset loss function.

[0091] The preset loss function can be any loss function such as the cross entropy loss function, and this application does not impose any restrictions on this. In one embodiment of the present application, the annotation labels corresponding to each of the multiple first sample images and the pseudo labels corresponding to at least part of the second sample images can also be substituted into the preset loss function to calculate the predicted loss value. Subsequently, the parameters in the initially trained defect classification model can be optimized using the backpropagation and gradient descent algorithms based on the predicted loss value. The optimization operation can be repeatedly performed until the defect classification model reaches a convergence state. When this round of training is completed, a defect classification model that has been trained corresponding to this round of training operations can be obtained.

[0092] According to the above technical solution, the defect classification model is trained using both labeled and unlabeled data. New categories are discovered based on the consistency of multiple data augmentations performed on the same initial sample image and the added constraints. This solution can fully utilize the large amount of unlabeled data to train the defect classification model, reducing the cost of manual labeling. Furthermore, the trained defect classification model can detect defects of unknown categories.

[0093] Exemplarily, the images to be tested include multiple first images to be tested, and the training operation may further include: after the current round of training operation is completed, obtaining a fourth predicted classification result corresponding to each of the at least one test image obtained by prediction of the trained defect classification model, wherein the fourth predicted classification result is used to indicate the category to which the defect area in the corresponding test image belongs, and the at least one test image includes one or more of the following: at least part of the sample images of the multiple second sample images; the image to be tested; a new image other than the multiple second sample images and the image to be tested;

[0094] Outputting the fourth predicted classification result corresponding to each of the at least one test image; and performing corresponding operations on the fourth predicted classification result and / or the test image in response to verification information input by the user.

[0095] In one embodiment, the image to be tested may include multiple first images to be tested. In the above embodiment, the defect classification model after the current round of training is obtained through training, and one or more test images can be respectively input into the defect classification model to obtain the fourth predicted classification result corresponding to each test image. The fourth predicted classification result can indicate the category to which the defect area in the corresponding test image belongs. The test image can be any image containing a target object. For example, the test image may include at least part of the sample images of the multiple second sample images, the image to be tested, and a new image other than the multiple second sample images and the image to be tested. One or more test images may also only include at least part of the sample images of the multiple second sample images, or the image to be tested, or a new image other than the multiple second sample images and the image to be tested.

[0096] In one embodiment, the apparatus for executing the defect classification method in the embodiment of the present application may include an input device and / or an output device. The input device and / or the output device may be communicatively connected to or included in the apparatus for executing the defect classification method in the embodiment of the present application.

[0097] The input device may include, but is not limited to, one or more of a mouse, a keyboard, a microphone, a touch screen, and the like. The output device may include, but is not limited to, one or more of a display device, a speaker, and the like. After obtaining the fourth predicted classification result, the fourth predicted classification result corresponding to each of at least one test image may be output. For example, if the output device is a display device, the fourth predicted classification result may be displayed in a display interface of the display device. The user may also input verification information through the input device. Based on the verification information input by the user, corresponding operations may be performed on the fourth predicted classification result and / or the test image. For example, if the user's verification information indicates that some images in the test image that do not meet the requirements need to be deleted, then in response to the verification information input by the user, the images that do not meet the requirements may be automatically deleted from the test image.

[0098] According to the above technical solution, after the current round of training is completed, the fourth predicted classification result corresponding to each of the at least one test image, predicted by the trained defect classification model, can be obtained. The fourth predicted classification result corresponding to each of the at least one test image is then output. Users can enter verification information to perform corresponding operations on the fourth predicted classification result and / or the test image. This solution facilitates users to view and perform corresponding operations on the fourth predicted classification result and test image, meeting the needs of different users and providing high interactivity.

[0099] Exemplarily, in response to the verification information input by the user, performing corresponding operations on the fourth predicted classification result and / or the test image includes one or more of the following: for any test image classified as a known category based on the fourth predicted classification result, if the verification information includes deletion information about the test image, deleting the test image, wherein the known category is the category indicated by the annotation label in the annotated data;

[0100] For any new category divided in the fourth predicted classification result, if the verification information includes deletion information about the new category, the new category is deleted from the fourth predicted classification result, where the new category is a category different from the known category; for any new category divided in the fourth predicted classification result, if the verification information includes merging information about the new category, the new category and the known category specified by the merging information are merged into the same category; for any new category divided in the fourth predicted classification result, if the verification information includes adding information about the new category, the new category and the test image corresponding to the new category are added to the labeled data.

[0101] In one embodiment, the fourth predicted classification result may indicate a defect region belonging to a known category or a defect region belonging to an unknown category. A known category is a category indicated by an annotated label in the annotated data. For example, if the fourth predicted classification result corresponding to a test image indicates that the defect region in the test image belongs to the category of a scratch, then the category of the defect region indicated by the fourth predicted classification result is a known category.

[0102] In the first embodiment, for any test image classified as a known category based on the fourth predicted classification result, if the verification information includes deletion information regarding the test image, the test image is deleted. For example, for test image A, if the user enters characters such as "Delete test image A" or clicks the "×" control corresponding to test image A, indicating that the user desires to delete the test image, then, in response to the verification information currently entered by the user, test image A may be deleted from at least one test image.

[0103] In the second embodiment, for any new category divided in the fourth prediction classification result, if the verification information includes deletion information about the new category, the new category is deleted from the fourth prediction classification result. The new category can represent other categories in addition to the known categories. For example, the fourth prediction classification result corresponding to the test image B indicates that the defect area contained in the current test image B belongs to category b. Category b is not included in the known categories, so category b can be used as a new category. If the verification information input by the user includes deletion information, such as characters such as "delete category b", it can indicate that category b is an invalid category and the user expects to delete the category. In response to the verification information currently input by the user, category b can be deleted from the fourth prediction classification result.

[0104] In the third embodiment, for any new category identified in the fourth predicted classification result, if the verification information includes merge information regarding the new category, the new category can be merged with the known category specified by the merge information to form the same category. For example, if the fourth predicted classification result corresponding to test image C indicates that the defect area contained in the current test image C belongs to category C, and category C has a high similarity to scratches, the user can enter merge information to merge category C with the scratch category. After the new category and the known category specified by the merge information are merged into the same category, the category corresponding to the defect area contained in test image C will be scratches.

[0105] In the fourth embodiment, for any new category identified in the fourth predicted classification result, if the verification information includes additional information regarding the new category, the new category and the test image corresponding to the new category are added to the already labeled data. For example, if the fourth predicted classification result for test image D indicates that the defect area contained in the current test image D belongs to category d, and category d is a valid new category, the user can add information to add category d to the known categories. Simultaneously, based on the additional information entered by the user, test image D corresponding to category d is also added to the already labeled data.

[0106] According to the above technical solution, for any test image that is classified into a known category based on the fourth prediction classification result, different operations can be performed for different verification information. When the verification information includes deletion information about any test image, the test image can be deleted. When the verification information includes deletion information about any new category, the new category can be deleted from the fourth prediction classification result. When the verification information includes merging information about any new category, the new category and the known category specified by the merging information can be merged into the same category. When the verification information includes adding information about any new category, the new category and the test image corresponding to the new category can be added to the labeled data. This solution can perform operations such as deleting, adding to known categories, or merging with known categories on new categories through manual verification, and can also delete test images. In this way, the labeled data can be iteratively optimized to obtain more new categories.

[0107] Figure 4 shows a schematic diagram of the defect classification model training process according to another embodiment of the present application. As shown in Figure 4, when the defect classification model is initially trained, the unlabeled data and labeled data in the sample data set can be input into the defect classification model to obtain the corresponding predicted classification results.

[0108] As shown in Figure 4, the predicted classification results can include 8 defect categories (each square can represent a defect category). The 4 defect categories in the dotted box represent existing categories, and the other 4 defect categories represent new categories. Based on the obtained predicted classification results, the user can enter different verification information to verify the predicted classification results. After the verification is completed, some new categories that do not meet the requirements in the predicted classification results can be deleted (for example, the two new categories indicated by the arrows in Figure 4 do not meet the requirements and can be deleted from the new categories), or some new categories in the predicted classification results can be merged with known categories in the labeled data, and the new categories in the predicted classification results can be added to the known categories. For the merged or added new categories, the test images corresponding to this part of the predicted classification results can be re-divided into known data to continue training the defect classification model. It can be understood that when the defect classification model is first trained, the training sample data set does not contain "new category labeled data."

[0109] According to another aspect of the present application, a defect classification device is further provided. FIG5 shows a schematic block diagram of a defect classification device 500 according to an embodiment of the present application. As shown in FIG5 , the defect classification device 500 includes a first input module 510 and a second input module 520 .

[0110] The first input module 510 is used to input the image to be tested and the first position prompt information into the feature extraction network of the defect classification model to obtain image features and position identification features of the image to be tested, wherein the defect classification model is an open-world semi-supervised classification model, and the first position prompt information is used to indicate the location of the defect area in the image to be tested.

[0111] The second input module 520 is used to input the image features and the location identification features into the fully connected neural network of the defect classification model to obtain a defect classification result, wherein the defect classification result is used to indicate the category to which the defect area in the image to be tested belongs.

[0112] A person skilled in the art can understand the specific implementation scheme and beneficial effects of the above-mentioned defect classification device by reading the above-mentioned description of the defect classification method 100. For the sake of brevity, they will not be described in detail here.

[0113] According to another aspect of the present application, an electronic device is also provided. Figure 6 shows a schematic block diagram of an electronic device according to an embodiment of the present application. As shown in Figure 6, the electronic device 600 includes a processor 610 and a memory 620. The memory 620 stores a computer program. When the computer program instructions are executed by the processor 610, they are used to execute the above-mentioned defect classification method.

[0114] According to another aspect of the present application, a storage medium storing a computer program / instructions is provided. The storage medium may include, for example, a tablet computer storage component, a personal computer hard drive, an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory device, or any combination thereof. The storage medium may be any combination of one or more computer-readable storage media. The computer program / instructions, when executed by a processor, are used to execute the aforementioned defect classification method.

[0115] A person skilled in the art can understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the above-mentioned description of the defect classification method. For the sake of brevity, it will not be repeated here.

[0116] Example:

[0117] Embodiment 1: A defect classification method, wherein the method comprises:

[0118] Inputting the image to be tested and the first position prompt information into a feature extraction network of a defect classification model to obtain image features and position identification features of the image to be tested, wherein the first position prompt information is used to indicate the position of the defect area in the image to be tested;

[0119] Inputting the image features and the location identification features into a fully connected neural network of the defect classification model to obtain a defect classification result, wherein the defect classification result is used to indicate the category to which the defect area in the image to be tested belongs;

[0120] Wherein, the defect classification model is an open-world semi-supervised classification model.

[0121] Embodiment 2: The method according to embodiment 1, wherein the defect classification model further includes an attention network, and before inputting the image features and the location identification features into the fully connected neural network of the defect classification model to obtain the defect classification result, the method further includes:

[0122] Inputting the image feature and the location identification feature into the attention network for feature fusion to obtain a fusion feature;

[0123] Inputting the image features and the location identification features into the fully connected neural network of the defect classification model to obtain a defect classification result includes:

[0124] The fused features are input into the fully connected neural network to obtain the defect classification result.

[0125] Embodiment 3: The method according to embodiment 1 or 2, wherein

[0126] The first position prompt information includes: target frame prompt information corresponding to the target detection frame containing the defect area, and / or mask prompt information corresponding to the mask of the defect area, and / or marker point prompt information corresponding to one or more marker points in the defect area;

[0127] The position identification features include: target frame coding features corresponding to the target detection frame containing the defect area, and / or mask coding features corresponding to the mask of the defect area, and / or identification point coding features corresponding to one or more identification points within the defect area.

[0128] Embodiment 4: According to the method described in any one of Embodiments 1-3, wherein the feature extraction network includes an image encoding module and a position encoding module, inputting the image to be tested and the first position prompt information into the feature extraction network of the defect classification model to obtain the image features and position identification features of the image to be tested, including:

[0129] Inputting the image to be tested into the image encoding module to obtain the image features;

[0130] The first position prompt information is input into the position encoding module to obtain the position identification feature.

[0131] Embodiment 5: The method described in any one of Embodiments 1-4, wherein the first position prompt information includes: target frame prompt information corresponding to the target detection frame containing the defect area, and / or mask prompt information corresponding to the mask of the defect area, and / or identification point prompt information corresponding to one or more identification points in the defect area; the position identification feature includes: target frame coding features corresponding to the target detection frame containing the defect area, and / or mask coding features corresponding to the mask of the defect area, and / or identification point coding features corresponding to one or more identification points in the defect area; the position encoding module includes: a target frame encoder, and / or a mask encoder, and / or an identification point encoder;

[0132] The step of inputting the first position prompt information into the position encoding module to obtain the position identification feature includes:

[0133] Inputting the target frame prompt information into the target frame encoder to obtain the target frame encoding feature; and / or,

[0134] Inputting the mask prompt information into the mask encoder to obtain the mask encoding feature; and / or,

[0135] The identification point prompt information is input into the identification point encoder to obtain the identification point coding feature.

[0136] Example 6: According to the method introduced in any one of Examples 1-5, the image encoding module is trained using an unsupervised training method. When training the defect classification model, the parameters of the image encoding module remain fixed and the parameters of the position encoding module are trained.

[0137] Embodiment 7: According to the method described in any one of embodiments 1-6, the defect classification model is obtained by the following training operations:

[0138] Acquire a sample data set, the sample data set including labeled data and unlabeled data, the labeled data including a plurality of first sample images and second position prompt information and a label corresponding to each first sample image, the label being used to indicate a category to which a defective area in the corresponding first sample image belongs, and the unlabeled data including a plurality of second sample images and third position prompt information corresponding to each second sample image;

[0139] Perform the following model training operations based on the sample dataset:

[0140] Using the labeled data to train the defect classification model to obtain the initially trained defect classification model;

[0141] inputting the plurality of second sample images and the third position prompt information corresponding to each of the plurality of second sample images into the initially trained defect classification model to obtain a first prediction classification result corresponding to each of the plurality of second sample images, wherein the first prediction classification result is used to indicate the category to which the defect area in the corresponding second sample image belongs;

[0142] Obtaining pseudo labels corresponding to at least some of the second sample images in the plurality of second sample images based on the first prediction classification results corresponding to the plurality of second sample images;

[0143] inputting the plurality of first sample images and the second position prompt information corresponding to each of the plurality of first sample images into the initially trained defect classification model to obtain a second prediction classification result corresponding to each of the plurality of first sample images, wherein the second prediction classification result is used to indicate the category to which the defect area in the corresponding first sample image belongs;

[0144] inputting the at least part of the second sample image and the third position prompt information corresponding to each of the at least part of the second sample image into the initially trained defect classification model to obtain a third predicted classification result corresponding to each of the at least part of the second sample image, the third predicted classification result being used to indicate the category to which the defect area in the corresponding second sample image belongs;

[0145] Calculating a prediction loss value based on a difference between the annotation labels corresponding to each of the plurality of first sample images and the second predicted classification result and a difference between the pseudo labels corresponding to each of at least some of the second sample images and the third predicted classification result;

[0146] Parameters in the initially trained defect classification model are optimized based on the predicted loss value to obtain the trained defect classification model corresponding to this round of training operation.

[0147] Embodiment 8: According to the method described in any one of Embodiments 1-7, the image to be tested includes a plurality of first images to be tested, and the training operation further includes:

[0148] After the current round of training operation is completed, a fourth predicted classification result corresponding to each of the at least one test image is obtained by the defect classification model through training, wherein the fourth predicted classification result is used to indicate the category to which the defect area in the corresponding test image belongs, and the at least one test image includes one or more of the following: at least part of the plurality of second sample images; the image to be tested; a new image other than the plurality of second sample images and the image to be tested;

[0149] Outputting the fourth prediction classification result corresponding to each of the at least one test images;

[0150] In response to the verification information input by the user, a corresponding operation is performed on the fourth predicted classification result and / or the test image.

[0151] Embodiment 9: According to the method described in any one of embodiments 1-8, in response to the verification information input by the user, performing corresponding operations on the fourth predicted classification result and / or the test image includes one or more of the following:

[0152] For any test image classified as a known category based on the fourth predicted classification result, if the verification information includes deletion information about the test image, deleting the test image, wherein the known category is the category indicated by the annotation label in the annotated data;

[0153] For any new category divided in the fourth predicted classification result, if the verification information includes deletion information about the new category, deleting the new category from the fourth predicted classification result, wherein the new category is a category different from the known category;

[0154] For any new category divided in the fourth predicted classification result, if the verification information includes merging information about the new category, merging the new category and the known category specified by the merging information into the same category;

[0155] For any new category divided in the fourth predicted classification result, if the verification information includes additional information about the new category, the new category and the test image corresponding to the new category are added to the labeled data.

[0156] Example 10: The method described in any one of Examples 1-9, wherein the image to be tested is a wafer image including a wafer, and the defective area is a defective area on the wafer.

[0157] Embodiment 11: A defect classification device, wherein the device comprises:

[0158] A first input module is configured to input the image to be tested and first position prompt information into a feature extraction network of a defect classification model to obtain image features and position identification features of the image to be tested, wherein the first position prompt information is used to indicate the position of the defect area in the image to be tested;

[0159] a second input module, configured to input the image features and the location identification features into a fully connected neural network of the defect classification model to obtain a defect classification result, wherein the defect classification result is used to indicate the category to which the defect area in the image to be tested belongs;

[0160] Wherein, the defect classification model is an open-world semi-supervised classification model.

[0161] Embodiment 12: An electronic device comprises a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the defect classification method described in any one of embodiments 1-10 when the processor is running.

[0162] Embodiment 13: A storage medium storing a computer program / instruction, wherein the computer program / instruction is used to execute the defect classification method described in any one of embodiments 1-10 when running.

[0163] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present application. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0166] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0167] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach of the present application should not be interpreted as reflecting the intention that the application claimed for protection requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0168] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0169] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0170] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the defect classification device according to the embodiments of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0171] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols between brackets should not be construed as limiting the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0172] The above description is merely a specific embodiment or illustration of a specific embodiment of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A defect classification method, characterized in that, The method includes: Inputting the image to be measured and the first position hint information into the feature extraction network of the defect classification model to obtain the image feature and the position identification feature of the image to be measured, where the first position hint information is used to indicate the position of the defect area in the image to be measured; Inputting the image feature and the position identification feature into the fully connected neural network of the defect classification model to obtain a defect classification result, where the defect classification result is used to indicate the category to which the defect area in the image to be measured belongs; Wherein, the defect classification model is an open-world semi-supervised classification model.

2. The method according to claim 1, wherein The defect classification model further includes an attention network. Before inputting the image feature and the position identification feature into the fully connected neural network of the defect classification model to obtain a defect classification result, the method further includes: Inputting the image feature and the position identification feature into the attention network for feature fusion to obtain a fused feature; The step of inputting the image feature and the position identification feature into the fully connected neural network of the defect classification model to obtain a defect classification result includes: Inputting the fused feature into the fully connected neural network to obtain the defect classification result.

3. The method according to claim 1, wherein The first position hint information includes: target box hint information corresponding to the target detection box containing the defect area, and / or mask hint information corresponding to the mask of the defect area, and / or landmark point hint information corresponding to one or more landmark points in the defect area; The position identification feature includes: target box encoding feature corresponding to the target detection box containing the defect area, and / or mask encoding feature corresponding to the mask of the defect area, and / or landmark point encoding feature corresponding to one or more landmark points in the defect area.

4. The method according to claim 1, characterized in that The feature extraction network includes an image encoding module and a position encoding module. The step of inputting the image to be measured and the first position hint information into the feature extraction network of the defect classification model to obtain the image feature and the position identification feature of the image to be measured includes: Inputting the image to be measured into the image encoding module to obtain the image feature; Inputting the first position hint information into the position encoding module to obtain the position identification feature.

5. The method according to claim 4, characterized in that The first position hint information includes: target box hint information corresponding to the target detection box containing the defect area, and / or mask hint information corresponding to the mask of the defect area, and / or landmark point hint information corresponding to one or more landmark points in the defect area; the position identification feature includes: target box encoding feature corresponding to the target detection box containing the defect area, and / or mask encoding feature corresponding to the mask of the defect area, and / or landmark point encoding feature corresponding to one or more landmark points in the defect area; the position encoding module includes: a target box encoder, and / or a mask encoder, and / or a landmark point encoder; The step of inputting the first position hint information into the position encoding module to obtain the position identification feature includes: Input the target box prompt information into the target box encoder to obtain the target box encoded feature; and / or, Input the mask prompt information into the mask encoder to obtain the mask encoded feature; and / or, Input the landmark point prompt information into the landmark point encoder to obtain the landmark point encoded feature.

6. The method according to claim 4, characterized in that The image encoding module is trained using an unsupervised training method. When training the defect classification model, the parameters of the image encoding module are kept fixed, and the parameters of the position encoding module are trained.

7. The method according to any one of claims 1 to 6, characterized in that, The defect classification model is obtained through the following training operations: Obtain a sample data set, which includes labeled data and unlabeled data. The labeled data includes a plurality of first sample images and second position prompt information and annotation labels corresponding to each first sample image. The annotation labels are used to indicate the category to which the defect area in the corresponding first sample image belongs. The unlabeled data includes a plurality of second sample images and third position prompt information corresponding to each second sample image; Perform the following model training operations based on the sample data set: Use the labeled data to train the defect classification model to obtain the initially trained defect classification model; Input the plurality of second sample images and the third position prompt information corresponding to each of the plurality of second sample images into the initially trained defect classification model to obtain the first predicted classification result corresponding to each of the plurality of second sample images. The first predicted classification result is used to indicate the category to which the defect area in the corresponding second sample image belongs; Obtain the pseudo-labels corresponding to at least some of the second sample images among the plurality of second sample images based on the first predicted classification results corresponding to the plurality of second sample images; Input the plurality of first sample images and the second position prompt information corresponding to each of the plurality of first sample images into the initially trained defect classification model to obtain the second predicted classification result corresponding to each of the plurality of first sample images. The second predicted classification result is used to indicate the category to which the defect area in the corresponding first sample image belongs; Input the at least some of the second sample images and the third position prompt information corresponding to each of the at least some of the second sample images into the initially trained defect classification model to obtain the third predicted classification result corresponding to each of the at least some of the second sample images. The third predicted classification result is used to indicate the category to which the defect area in the corresponding second sample image belongs; Calculate a prediction loss value based on the difference between the annotation labels and the second predicted classification results corresponding to the plurality of first sample images and the difference between the pseudo-labels and the third predicted classification results corresponding to at least some of the second sample images; Optimize the parameters in the initially trained defect classification model based on the prediction loss value to obtain the defect classification model that is trained and completed corresponding to this round of training operations.

8. The method according to claim 7, wherein The image to be measured includes a plurality of first images to be measured, and the training operation further includes: After this round of training operations is completed, obtain the fourth prediction classification results respectively corresponding to at least one test image obtained by predicting with the defect classification model completed by training, where the fourth prediction classification results are used to indicate the categories to which the defect regions in the corresponding test images belong, and the at least one test image includes one or more of the following: at least some sample images among the multiple second sample images; the image to be tested; new images other than the multiple second sample images and the image to be tested; Output the fourth prediction classification results respectively corresponding to the at least one test image; In response to verification information input by the user, perform corresponding operations on the fourth prediction classification results and / or the test images.

9. The method according to claim 8, wherein The performing corresponding operations on the fourth prediction classification results and / or the test images in response to verification information input by the user includes one or more of the following: For any test image classified as a known category based on the fourth prediction classification results, if the verification information includes deletion information about this test image, then delete this test image, where the known category is the category indicated by the annotation label in the annotated data; For any new category classified from the fourth prediction classification results, if the verification information includes deletion information about this new category, then delete this new category from the fourth prediction classification results, where the new category is a category different from the known category; For any new category classified from the fourth prediction classification results, if the verification information includes merging information about this new category, then merge this new category with the known category specified by the merging information into the same category; For any new category classified from the fourth prediction classification results, if the verification information includes addition information about this new category, then add this new category and the test image corresponding to this new category to the annotated data.

10. The method according to any one of claims 1-6, characterized in that, The image to be tested is a wafer image containing a wafer, and the defect region is a defect region on the wafer.

11. A defect classification device, characterized in that, The device includes: A first input module, configured to input the image to be tested and first position prompt information into the feature extraction network of the defect classification model to obtain the image feature and position identification feature of the image to be tested, where the first position prompt information is used to indicate the position where the defect region in the image to be tested is located; A second input module, configured to input the image feature and the position identification feature into the fully connected neural network of the defect classification model to obtain a defect classification result, where the defect classification result is used to indicate the category to which the defect region in the image to be tested belongs; Wherein, the defect classification model is an open-world semi-supervised classification model.

12. An electronic device, comprising a processor and a memory, characterized in that, The memory stores computer program instructions, and when the computer program instructions are run by the processor, they are used to execute the defect classification method according to any one of claims 1-10.

13. A storage medium stores computer programs / instructions, characterized in that, The computer program / instructions are used to execute the defect classification method according to any one of claims 1-10 when running.

Images