Visual identification method and device based on computer processing

Abstract

The invention provides a visual identification method and device based on computer processing, and the method comprises the steps: S1, collecting image data based on a wafer generation line, carrying out the preprocessing and label judgment of the collected image data, and obtaining a wafer defect detection data set; s2, aiming at the wafer defect detection data set obtained in S1, constructing a feature extraction and feature enhancement network to obtain a multi-level feature map; s3, designing a wafer defect detection model based on the multi-level feature map obtained in the S2, thereby obtaining a semiconductor wafer defect detection model; s4, based on the semiconductor wafer defect detection model obtained in S3, constructing a training strategy combined with antagonism collaborative learning, and training to obtain a final wafer defect detection model; and S5, model deployment: deploying the wafer defect detection model obtained in the step S4 to a production process equipment vision module, and detecting and feeding back defect problems in real time. The method has the following beneficial effects that the detection accuracy and robustness are remarkably improved while the precision is maintained.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a multi-task collaborative semiconductor wafer defect visual detection method. Background Art

[0002] With the rapid development of semiconductor technology, the types of defects that occur in the wafer manufacturing process have become more and more complex and diverse. These defects not only affect the production yield of wafers, but also have a profound impact on the performance and reliability of the final product. In order to ensure high-quality wafer manufacturing, accurate defect detection has become a crucial link. However, with the continuous advancement of process technology, traditional detection methods have gradually exposed the problem of being unable to meet modern manufacturing needs when facing tiny and complex defects. Therefore, it is urgent to develop more advanced detection technologies to meet the growing detection needs and ensure production efficiency and product quality.

[0003] The existing PCB semiconductor wafer defect detection mainly covers the following three methods: template matching-based method, image segmentation-based method and machine learning-based method:

[0004] (1) Template matching based method:

[0005] This method detects defects by comparing the image of the wafer to be inspected with the image of a standard defect-free wafer at the pixel level. Although the implementation is simple, it has the following disadvantages:

[0006] a) The image alignment requirement is extremely high, and even a slight deviation may lead to false detection;

[0007] b) Poor detection capability for non-repetitive defects;

[0008] c) The computational complexity is high and it is difficult to meet the real-time requirements of online detection.

[0009] (2) Methods based on image segmentation:

[0010] This method first segments the wafer image and then analyzes the segmented regional features to identify defects. Its main disadvantages include:

[0011] a) The segmentation algorithm is not robust enough and is easily affected by image noise and illumination changes;

[0012] b) The detection effect of tiny defects and low-contrast defects is poor;

[0013] c) It is difficult to effectively distinguish defects from normal structural changes.

[0014] (3) Machine learning-based methods:

[0015] This type of method uses traditional machine learning algorithms (such as SVM, random forest, etc.) to classify the extracted image features. Although it is an improvement over the previous two methods, it still has the following problems:

[0016] a) Feature engineering relies on manual design and is difficult to fully capture complex defect features;

[0017] b) Limited generalization ability and poor adaptability to unseen defect types;

[0018] c) It is difficult to achieve high-precision and high-efficiency detection at the same time.

[0019] In view of the limitations of the above existing semiconductor wafer defect detection methods, it can be seen that they have problems such as insufficient accuracy, low efficiency, and weak generalization ability when dealing with complex and diverse wafer defects. Especially in advanced processes, traditional methods are difficult to cope with the increasingly miniaturized and diverse defect types, and cannot meet the needs of modern semiconductor manufacturing for high-precision and high-efficiency defect detection. Therefore, it is necessary to develop a new type of semiconductor wafer defect detection method to overcome these shortcomings and achieve high-precision, high-efficiency, and strong generalization ability of semiconductor wafer defect detection. Summary of the invention

[0020] To solve the above problems, the present invention proposes a visual recognition method based on computer processing, which effectively realizes high-precision and high-efficiency wafer defect detection, especially showing strong adaptability and robustness in the face of complex and diverse defect types.

[0021] A visual recognition method based on computer processing, characterized in that the method comprises:

[0022] S1 collects original semiconductor wafer image data based on the real semiconductor wafer production line, designs data preprocessing methods such as center cropping, denoising and data enhancement for the collected original image data to ensure the smoothness of the image data, and performs label determination to obtain a complete wafer defect detection data set;

[0023] S2 constructs a dynamic multi-scale feature extraction and feature enhancement network for wafer defect image based on the wafer defect detection dataset obtained in S1, and obtains a multi-level feature map containing local details and global semantic information of the wafer defect image;

[0024] S3 designs a multi-task collaborative wafer defect detection model based on the multi-level feature map of wafer defects, obtains the classification, regression and circular boundary prediction results of wafer defects, and thus realizes the visual detection of semiconductor wafer defects;

[0025] S4 is based on the semiconductor wafer defect detection model, constructs a training strategy combined with adversarial collaborative learning, and performs multi-stage progressive training to obtain the final wafer defect detection model;

[0026] S5 model deployment deploys the wafer defect detection model obtained in S4 to the vision module of the process equipment for semiconductor wafer production, and detects and provides feedback in real time whether there are defects in the produced semiconductor wafers.

[0027] In the visual recognition method described above, the process of collecting and making the data set in S1 includes:

[0028] (1) Raw data collection

[0029] Source and acquisition method of semiconductor wafer defect image data: Raw data acquisition is carried out by installing high-precision industrial cameras on the semiconductor wafer production line. The ace series industrial camera (model: acA4112-20um) from Basler, Germany, is used. The camera has a resolution of 4112×3008 pixels and a collection speed of up to 20 frames per second. The camera is installed above the wafer transport track and works in conjunction with the light source system to ensure the capture of clear wafer surface images.

[0030] To ensure image quality, an LED ring light source (model: CCS LDR2-70SW2-LA1) is used to provide uniform and stable lighting conditions. During image acquisition, the triggering of the camera and the light source is precisely controlled by a PLC (programmable logic controller) to ensure that each wafer can be fully captured.

[0031] The original image data is saved in 16-bit TIFF format, with each pixel containing 65536 gray levels to ensure high dynamic range and detail retention of the image. The file size of each image is about 24MB, and the naming convention is "wafer_YYYYMMDD_HHMMSS_Serial Number.tiff", where YYYYMMDD represents the date, HHMMSS represents the time, and the serial number is the sequential number of the acquisition on that day.

[0032] Data diversity and representativeness analysis: To ensure the diversity and representativeness of the data set, wafer samples from different production batches and different process stages are collected. Specifically:

[0033] Wafers of different sizes: 6 inches, 8 inches and 12 inches;

[0034] Wafers at different process nodes: 28nm, 14nm and 7nm;

[0035] Wafers at different stages of the process: after lithography, after etching, and after polishing;

[0036] Different defect types: scratches, cracks, peeling and foreign matter adhesion;

[0037] Different degrees of defects: from mild to severe.

[0038] A total of 100,000 original wafer images were collected, including various defect types and non-defective samples. Through data analysis, it is ensured that the distribution of various defects in the data set is basically balanced, and the proportion of non-defective samples is about 60%, in order to simulate the frequency of defects in actual production. The collected original image data is recorded as dataset.

[0039] (2) Data preprocessing

[0040] ROI area cropping and its impact on model accuracy: Considering the circular characteristics of the wafer, a circular ROI (region of interest) cropping algorithm is designed:

[0041] First, the wafer edge is detected using Hough circle transform to obtain the wafer center coordinates and radius; secondly, the circular area with the detected center as the center and a radius slightly smaller than the detection radius (take 98%) is used as the ROI; finally, the pixel values ​​of the area outside the ROI are set to 0, and the original image information within the ROI is retained. This cropping method can effectively remove the interference information of the wafer edge and increase the model's attention to defects in the central area.

[0042] Image denoising: In wafer images, due to the thermal noise of the sensor readout circuit in the ace series industrial camera and the high-frequency components of the image, there is usually Gaussian white noise and speckle noise. In order to reduce the impact of these noises on image quality and subsequent processing, image denoising is required.

[0043] First, Gaussian filtering is used to remove Gaussian white noise. Gaussian filtering can effectively smooth the image and remove noise by convolving with the Gaussian kernel. Specifically, the calculation formula of the Gaussian filter is:

[0044]

[0045] Where σ is the standard deviation of the Gaussian kernel, x and y represent the pixel positions in the image, and are used to represent the offset of each pixel to the center of the filter. This offset is used to calculate the value of the Gaussian distribution function to determine the pixel weight. By performing a convolution operation with the image, Gaussian filtering can effectively reduce the Gaussian white noise in the image.

[0046] Data enhancement techniques: In order to increase the diversity of data and the generalization ability of the model, the following data enhancement techniques are used:

[0047] Random rotation: randomly rotate the image in the range of [-10°, 10°];

[0048] Random scaling: Randomly scale the image in the range [0.9, 1.1];

[0049] Random translation: Random translation within ±5% of the image width and height;

[0050] Random brightness and contrast adjustment: the brightness adjustment range is [0.8, 1.2], and the contrast adjustment range is [0.8, 1.2];

[0051] Random Gaussian noise: Add Gaussian noise with a mean of 0 and a standard deviation in the range of [0, 0.05];

[0052] Random horizontal and vertical flip: 50% chance of flipping each;

[0053] These data augmentation techniques are implemented through the Albumentations library and applied in real time during the training process. Five augmented images are generated for each original image, thus expanding the dataset to 500,000 images. This augmentation strategy significantly improves the model's adaptability to various lighting conditions and defect morphologies.

[0054] The image data after data preprocessing is recorded as data.

[0055] (3) Label determination

[0056] Manually annotating the collected raw image data is to provide accurate training data for subsequent defect detection tasks. The industrial-grade annotation software LabelImg is used, which has an intuitive user interface and rich functions. Users can draw bounding boxes on the image to annotate the target object through simple operations, and add corresponding label information to each annotated box. The annotation process follows the following rules:

[0057] Strict pixel-level annotation: Annotators need to accurately draw the bounding box of the defect area on the image to ensure the accuracy and precision of the annotation;

[0058] Defect classification, regression and circular boundaries as labels: the classification label is denoted as label cls , corresponding to five defect types (no defect, scratch, crack, peeling and foreign matter attachment), the value of each dimension represents the confidence score of the defect type; the regression label is recorded as label reg Contains the bounding box coordinate information of the predicted target, which is used to accurately locate the location of the defect; the circular boundary label is recorded as label cir The circular parameters closely related to the defect boundary are given, which can further improve the accuracy of boundary positioning.

[0059] Multiple defect annotations: Each image may contain 0 or more defect annotations. The annotator needs to mark all defect areas in the image according to the actual situation.

[0060] Saving annotation results: The annotation results are saved in the PASCAL VOC format, which can be easily integrated and processed with the wafer defect detection model.

[0061] The annotation process requires the annotator to have certain professional knowledge and experience to ensure the accuracy and usability of the annotation results.

[0062] (4) Dataset creation

[0063] In order to divide the labeled data into training set, validation set and test set, and prepare for model training, evaluation and testing, it is necessary to construct a data set: the labeled data set is recorded as (data, label), and the training set, validation set and test set are divided into 8:1:1 ratio.

[0064] In order to divide the labeled data into training set, validation set and test set in preparation for model training, evaluation and testing, the data set (data, label) needs to be divided in a ratio of 8:1:1. The specific steps are as follows:

[0065] The labeled data set (data, label) is divided into the following proportions: 80% of the data is used as the training set (Training Set), 10% of the data is used as the validation set (Validation Set), and 10% of the data is used as the test set (Test Set).

[0066] Such a division can ensure that there is sufficient data for parameter adjustment and optimization during the model training process, and perform evaluation of the validation set and final performance verification of the test set after model training to ensure the generalization ability and effectiveness of the model.

[0067] In the visual recognition method described above, the image dynamic multi-scale feature extraction and feature enhancement network construction process in S2 includes:

[0068] This solution innovatively proposes a dynamic multi-scale feature enhancement network for wafer images, breaking through the limitations of traditional single-scale feature extraction. Through multi-scale feature extraction and enhancement, combined with a dynamic weight allocation mechanism, the network can capture local details and global semantics at the same time, significantly improving the richness and robustness of feature representation. In particular, the module that dynamically adjusts feature weights effectively highlights key information, suppresses irrelevant noise, and enhances the adaptability of the model in complex production line environments. In addition, the network design focuses on computational efficiency, and through feature reuse and lightweight structure, it takes into account both performance and real-time performance, and is suitable for semiconductor wafer production line equipment. This method not only improves the accuracy of defect detection, but also enhances the robustness of the system under various visual interference conditions.

[0069] The input wafer image is processed from the bottom layer to the top layer. First, feature maps {C2, C3, C4, C5} of different scales are extracted, and then a multi-scale feature enhancement network is constructed according to the following steps:

[0070] Starting from the deepest P5 layer, the upper-level wafer feature map is generated by upsampling, and this high-level semantic information is passed to the top layer of the multi-scale feature enhancement network:

[0071] M5=C5

[0072] Among them, M5 is the top-level feature map in the multi-scale feature enhancement network, which comes from the deepest C5 feature map, and then the number of 1×1 convolution channels of M5 is adjusted:

[0073] P5=Conv1×1(M5)

[0074] Then build a multi-scale feature enhancement network from top to bottom:

[0075] M4=C4+Upsample(P5)

[0076] P4=Conv3×3(M4)

[0077] Among them, Conv3×3 is a 3×3 convolution, which is used to fuse features of different scales. Upsample is an upsampling operation implemented by deconvolution. Repeat the above steps to gradually generate {P3, P4, P5}. Finally, P3, P4, P5 are multi-scale feature enhancement networks that fuse multi-scale information.

[0078] A feature weight dynamic allocation module is incorporated into the network to enhance the adaptability to multi-scale features and learn the correlation between features at different positions in the wafer feature map. By highlighting important features and suppressing irrelevant features, the mechanism first generates a spatial attention submodule, that is, the attention weight corresponding to each position, and then combines the spatial attention submodule with the original feature map to obtain an enhanced feature representation. The specific steps are as follows:

[0079] The input of the feature weight dynamic allocation module is a wafer feature map X with a dimension of (C×H×W). Two 1×1 convolution kernels are used to perform channel compression on X to generate compressed wafer feature maps A and B. The dimensions of the compressed wafer feature map are (C′×H×W):

[0080] A=X·W a (C′×H×W)

[0081] B=X·W b (C′×H×W)

[0082] Among them, W a and W b are the weights of two 1×1 convolution kernels respectively. A and B are used to calculate the similarity between positions and generate the spatial attention submodule map M with a dimension of (H×W×H×W):

[0083] M=Softmax(A T B)

[0084] Then, the original input X and M are element-wise multiplied to obtain the wafer feature map X′ weighted by the spatial attention submodule:

[0085] X′=X·M

[0086] Finally, X′ and X are added and then output through a convolutional layer:

[0087] O=γ·X′+X·W c

[0088] Among them, γ is a learnable scaling factor used to scale the effect of the spatial attention submodule enhancement, W c is a 1×1 convolution kernel weight, and O is the final output wafer feature map of the feature weight dynamic allocation module.

[0089] As described above, the visual recognition method, the multi-task collaborative wafer defect detection model building process in S3 includes:

[0090] In the defect detection task, classification and regression are two interrelated but often separately processed subtasks. Traditional methods model and optimize the two tasks separately, lacking knowledge fusion and mutual promotion between the two. To solve this problem, the present invention designs a multi-task collaborative detection algorithm (MCDA).

[0091] The key to MCDA is the introduction of a multi-task collaborative encoding mechanism to fuse classification and regression information. The model cascades the feature maps of the two subtasks through the convolution layer and encodes each other's information to obtain classification and regression features that fuse bidirectional relationships. In addition, MCDA also introduces a circular boundary prediction branch. By cascading classification and regression prediction results, it can more finely predict the circular parameters related to the defect boundary and improve the accuracy of boundary positioning. Compared with the traditional separate modeling method, MCDA can better explore the intrinsic connection between classification, regression and shape, thereby improving the consistency and accuracy of overall detection. Specifically, it includes the following processes:

[0092] The feature O output by the dynamic multi-scale feature extraction and feature enhancement network is fed into MCDA to complete classification, regression and circular boundary prediction at the same time:

[0093] cls pred ,reg pred ,cir pred =MCDA(O)

[0094] Among them, cls pred ,reg pred ,cir pred They are the classification prediction results, regression prediction results, and circular boundary prediction results of wafer defect detection.

[0095] The specific calculation process of MCDA is as follows:

[0096] 1) Perform basic convolution on O to obtain the initial prediction value {cls init , reg init}:

[0097] cls init ,reg init =Conv(O)

[0098] 2) reg init With cls init Perform relationship encoding to obtain the regression feature reg that integrates the relationship between the two fused :

[0099] reg fused =reg init +Conv(Concat(reg init ,cls init ))

[0100] 3) At the same time, cls init With reg init Perform relationship encoding to obtain the classification feature cls that integrates the relationship between the two fused :

[0101] cls fused =cls init +Conv(Concat(cls init , reg init ))

[0102] 4) reg fused and cls fused Perform convolution to obtain the final classification prediction cls pred and regression prediction reg pred :

[0103]

[0104] 5) Set cls pred and reg pred Cascaded, through an additional convolution branch, predicts the defect boundary circular parameter cir which is closely related to the defect boundary pred :

[0105] cir pred =Conv(Concat(cls pred , reg pred ))

[0106] Through multi-task collaborative encoding, classification and regression knowledge can be fully integrated to promote each other. At the same time, a circular boundary prediction branch is introduced to more finely describe the shape and location of wafer defects.

[0107] In existing target detection methods, common loss functions such as FocalLoss, CIoULoss, etc., only focus on optimizing the indicators of a single task, and lack consideration of the mutual influence and knowledge guidance between multiple tasks. In order to solve this problem, the present invention proposes a multi-task collaborative loss function.

[0108] The innovation of this loss function lies in the integration of classification, regression and circular boundaries to achieve joint optimization of three closely related subtasks: classification, regression and shape. Compared with the traditional single loss function, the multi-task collaborative loss function makes full use of the correlation between subtasks, achieves mutual promotion between tasks, and also incorporates the guidance of prior knowledge, so that the model benefits from additional knowledge transfer during the optimization process, thereby improving the overall performance and robustness of detection.

[0109] Therefore, the present invention adopts a multi-task collaborative loss function to simultaneously optimize the three tasks of classification, regression and circular boundary prediction, namely:

[0110] L=λ1L cls +λ2L reg +λ3L cir

[0111] Among them, L cls For classification loss, use FocalLoss; L reg For regression loss, CIoULoss is used; L cir For the annular boundary loss, PolygonLoss is used; λ1, λ2, and λ3 are the hyperparameters for balancing the loss terms.

[0112] Under the optimization of the multi-task collaborative loss function, the target detection network can simultaneously output three prediction results: classification prediction result cls pred , regression prediction result reg pred And the circular boundary prediction result cir pred .

[0113] Among them, the classification prediction result cls pred is a vector corresponding to four defect types (scratches, cracks, peeling, and foreign matter attachment). The value of each dimension represents the confidence score of the defect type. By obtaining the dimension index with the highest score, the defect type in the current prediction box can be determined;

[0114] Regression prediction result reg pred Contains the bounding box coordinate information of the predicted target, which is used to accurately locate the location of the defect;

[0115] Circular boundary prediction result cir pred The circular parameters closely related to the defect boundary are given, which further improves the accuracy of boundary positioning.

[0116] After obtaining the above three prediction results, the defect type can be determined, and combined with precise position and shape information, accurate detection of various wafer defects can be achieved, providing an important basis for subsequent quality control and defect repair.

[0117] In the visual recognition method described above, the adversarial collaborative learning training strategy in S4 includes:

[0118] The labeled training data set is input into the model, and this scheme proposes an innovative adversarial collaborative learning strategy to train the model. This strategy is an innovative multi-stage progressive training, which fully considers the local details and global semantic information in the wafer image data, as well as their dynamic relationship in the fusion process, and improves the performance of the model through multi-scale temporal consistency and deep collaborative optimization.

[0119] (1) Multi-scale temporal consistency loss:

[0120] In order to enhance the model's learning of different time batch data information, this proposal proposes a multi-scale temporal consistency loss. This loss function measures the consistency of predictions at different time scales:

[0121] L tc =∑ s λ s *D KL (P(Y|X t ),P(Y|X{ts}))

[0122] Among them, s represents different time scales, λ s is the corresponding weight, D KL is the KL divergence, P(Y|X t ) represents the detection distribution at time step t, including classification detection, regression detection and circular boundary detection, which is a summary of the total detection result. This loss encourages the model to maintain consistency in predictions at different time scales, thereby improving the modeling ability of different time batch data.

[0123] (2) Adversarial training strategy:

[0124] In order to improve the robustness of training, this proposal introduces an adversarial training strategy. Specifically, a discriminator D is designed to try to distinguish whether the features come from local details or global semantic information. The fusion network F is trained to deceive the discriminator:

[0125] L adv =E[log(D(F(X v ,X a )))]+E[log(1-D(X v ))+log(1-D(X a ))]

[0126] Among them, X v and X a They are local details and global semantic information inputs respectively. This adversarial training forces the network to generate feature representations that are more difficult to distinguish and more closely integrated.

[0127] (3) Course learning and difficulty adaptation

[0128] According to (2) adversarial strategy, a curriculum learning strategy based on sample difficulty is proposed. Sample difficulty D(x) is defined as:

[0129] D(x)=1-exp(-γ(w v L v +w a L a ))

[0130] Among them, L v and La are the loss of local details and global semantic information respectively, and γ is an adjustable parameter. During the training process, the proportion of difficult samples is gradually increased:

[0131]

[0132] Among them, t is the current training step, T is the total training step, p0 is the initial difficult sample ratio, and μ controls the rate of difficulty increase.

[0133] (4) Dynamic Batch Normalization

[0134] Considering the variability of illumination and noise conditions in semiconductor wafer production lines, this proposal proposes Dynamic Batch Normalization (DBN) technology. DBN dynamically adjusts the normalization parameters according to the statistical characteristics of the input:

[0135] y=γ(x)(x-μ(x)) / (σ(x)+ε)+β(x)

[0136] Among them, γ(x) and β(x) are input-dependent scaling and offset parameters. This method can better adapt to the changes in feature distribution under different input conditions.

[0137] (5) Model training

[0138] Based on the above method, the specific process of model training is as follows:

[0139] 1) Initializing the model parameters θ, where θ represents the parameters to be updated, including all weight matrices and bias vectors in the wafer defect detection model;

[0140] 2) For each training step t:

[0141] a. Sample a batch from the dataset, press p hard The proportion of (t) contains difficult samples;

[0142] b. Perform forward propagation to obtain multi-scale features and prediction results;

[0143] c. Calculate the multi-scale temporal consistency loss L tc ;

[0144] d. Perform adversarial training strategy to update the discriminator D and fusion network F;

[0145] e. Apply dynamic batch normalization;

[0146] f. Calculate the total loss: L total =λ tc *L tc +λ adv *L adv;

[0147] g. Back propagation, update model parameters: in is the learning rate.

[0148] 3) Repeat step 2) and terminate the training process when the loss cannot be reduced for N consecutive epochs. Save the final model parameters at the end of the training as the deployment model for wafer defect detection.

[0149] In the visual inspection method described above, the wafer defect inspection model deployment process in S5 includes:

[0150] (1) Hardware platform selection and configuration:

[0151] a. According to the actual needs of the semiconductor wafer production line, select suitable edge computing devices, such as NVIDIA Jetson series or Intel NUC high-performance embedded systems;

[0152] b. Configure the necessary deep learning frameworks and dependent libraries on the selected hardware platform: CUDA, cuDNN, etc. to support efficient operation of the model;

[0153] c. Optimize hardware resource allocation and reasonably set GPU memory usage limits and CPU thread numbers to balance detection performance and system stability.

[0154] (2) Model integration and interface development:

[0155] a. Design and implement the model reasoning interface, including image preprocessing, model reasoning and result post-processing functional modules;

[0156] b. Develop a communication interface with the production line control system to achieve real-time transmission and feedback of test results;

[0157] c. Build a cache mechanism, optimize data stream processing, and reduce the impact of I / O operations on detection speed.

[0158] (3) Real-time image acquisition and preprocessing:

[0159] a. Use high-speed industrial cameras to collect wafer images in real time to ensure that image quality and acquisition frequency meet inspection requirements;

[0160] b. Implement an image preprocessing pipeline, including center cropping, denoising, and data augmentation operations, to improve the quality of input images;

[0161] c. Use an asynchronous processing mechanism to perform preprocessing while image acquisition to maximize the use of computing resources.

[0162] (4) Defect detection and result output:

[0163] a. Input the preprocessed image into the deployed wafer defect detection model to perform inference operations;

[0164] b. Parse the model output to extract defect category, bounding box coordinate information, and circle parameter information;

[0165] c. Screen and grade the detection results according to preset thresholds to reduce false positives and false negatives.

[0166] (5) Visualization and storage of test results:

[0167] a. Develop a real-time visualization interface to intuitively display the inspection results, including defect categories, bounding box coordinate information, and circle parameter information;

[0168] b. Realize the local storage function of the test results, including original images, test results and related data;

[0169] (6) Production line linkage and feedback:

[0170] a. Transmit the test results to the production line control system in real time for automated decision-making;

[0171] b. Develop an alarm mechanism to promptly notify relevant personnel and trigger an emergency response on the production line when a serious defect is detected;

[0172] c. Realize the correlation analysis between test results and production parameters to provide data support for process optimization.

[0173] (7) Performance evaluation and continuous optimization:

[0174] a. Establish a regular performance evaluation mechanism, including statistical analysis of key indicators of detection accuracy, recall rate and processing speed;

[0175] b. Develop an automated testing process and use standard test sets to evaluate the performance of the model in the actual production environment;

[0176] c. Based on performance evaluation results and production feedback, continuously optimize model parameters and deployment strategies to continuously improve the overall performance of the detection system.

[0177] The present invention also provides a visual recognition device based on computer processing, which uses the above visual recognition method.

[0178] Compared with the prior art, the present invention has the following beneficial effects:

[0179] (1) Multi-task collaborative optimization:

[0180] The present invention introduces a multi-task collaborative optimization strategy in wafer defect detection, integrating classification, regression and circular boundary prediction tasks into an overall framework. Traditional detection methods often focus on a single task, such as only defect classification or region segmentation, resulting in limitations and deviations in the detection results. Through multi-task collaborative optimization, the present invention can utilize the correlation and complementarity between different tasks to improve the overall performance of the model. Specifically, this collaborative mechanism enables the model to improve the detection capability of tiny and edge defects while maintaining accuracy when dealing with complex and diverse defects, significantly enhancing the comprehensiveness and robustness of detection.

[0181] (2) Dynamic multi-scale feature extraction and enhancement:

[0182] The present invention innovatively proposes a dynamic multi-scale feature extraction and enhancement network architecture to deal with defects of different scales on semiconductor wafers. Traditional detection methods usually use fixed-scale feature extraction, which is prone to losing key information when facing complex wafer defects, especially when dealing with defects of different sizes and shapes. The present invention introduces a dynamic multi-scale feature extraction network, which can dynamically adjust the extraction scale according to the geometric characteristics and location of the defect, ensuring that the global structure and local details of the defect are fully captured in the process of multi-scale feature fusion. At the same time, the feature enhancement mechanism further optimizes the expression ability of key features, enabling the model to accurately identify subtle defects in complex backgrounds and improve the accuracy and robustness of detection.

[0183] (3) Adversarial collaborative learning training strategy:

[0184] In order to further improve the generalization ability of the model, the present invention adopts a unique adversarial collaborative learning training strategy. In this strategy, the model performs collaborative optimization of multiple tasks by simulating various complex situations in the actual production environment during the training process. By introducing adversarial factors in multi-stage progressive training, the model can effectively deal with different types of wafer defects, especially when facing unknown or extremely challenging defects, it can still maintain a high level of detection accuracy. This strategy significantly enhances the model's resistance to various interference factors, making it more robust and reliable in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0185] Figure 1 Dynamic multi-scale feature enhancement network structure diagram for wafer images;

[0186] Figure 2 The overall block diagram of semiconductor wafer defect detection. DETAILED DESCRIPTION

[0187] Figure 1The structure diagram of the dynamic multi-scale feature enhancement network for wafer images is shown in Figure 1. A dynamic feature weight allocation module is incorporated into the network to enhance the adaptability to multi-scale features and learn the correlation between features at different positions in the wafer feature map. By highlighting important features and suppressing irrelevant features, the mechanism first generates a spatial attention submodule, that is, the attention weight corresponding to each position, and then combines the spatial attention submodule with the original feature map to obtain an enhanced feature representation.

[0188] Figure 2 The overall block diagram of semiconductor wafer defect detection is as follows: first, based on the real semiconductor wafer generation line, the original semiconductor wafer image data is collected, and data preprocessing is performed, including center cropping, denoising and data enhancement, to ensure the smoothness of the image data, and finally a complete wafer defect detection data set is obtained; secondly, for this data set, a dynamic multi-scale feature extraction and feature enhancement network for wafer defect images is constructed to generate a multi-level feature map containing local details and global semantic information; then, based on the multi-level feature map, a multi-task collaborative wafer defect detection model is designed to output the classification, regression and circular boundary prediction results of wafer defects, and realize the visual detection of semiconductor wafer defects; finally, by combining the training strategy of adversarial collaborative learning, multi-stage progressive training is performed, and finally an efficient wafer defect detection model is obtained, which is deployed in the visual module of semiconductor wafer production equipment to realize real-time defect detection and feedback.

[0189] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0190] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A computer-based visual recognition method, characterized in that: The method includes: S1. Collect original semiconductor wafer image data based on a real semiconductor wafer production line, design a data preprocessing method of center cropping, denoising and data enhancement for the collected original image data, and perform label determination to obtain a complete wafer defect detection data set; S2. For the wafer defect detection dataset obtained in S1, a dynamic multi-scale feature extraction and feature enhancement network for wafer defect images is constructed to obtain a multi-level feature map containing local details and global semantic information of the wafer defect image; S3. Based on the multi-level feature map of wafer defects obtained in S2, a multi-task collaborative wafer defect detection model is designed to obtain the classification, regression and circular boundary prediction results of wafer defects, thereby obtaining a semiconductor wafer defect detection model; S4, based on the semiconductor wafer defect detection model obtained in S3, construct a training strategy combined with adversarial collaborative learning, and perform multi-stage progressive training to obtain the final wafer defect detection model; S5, model deployment, deploying the wafer defect detection model obtained in S4 to the vision module of the process equipment for semiconductor wafer production, to detect and provide feedback in real time whether there are defects in the produced semiconductor wafers.

2. The visual recognition method according to claim 1, characterized in that: The collection and production process of the data set in S1 includes: (1) Raw data collection: A high-precision industrial camera is installed above the wafer transport track and works with the light source system to ensure the capture of clear wafer surface images; the original image data is saved in 16-bit TIFF format, and each pixel contains 65536 gray levels to ensure the high dynamic range and detail retention of the image; wafer samples from different production batches and different process stages are collected; specifically, they include: different sizes, process nodes, process stages, defect types, and defect degrees; the total collected original image data is recorded as dataset; (2) Data preprocessing: ROI area cropping and its impact on model accuracy: Considering the circular characteristics of the wafer, a circular ROI (region of interest) cropping algorithm is designed: First, the wafer edge is detected using Hough circle transform to obtain the wafer center coordinates and radius; secondly, a circular area with the detected center as the center and a radius slightly smaller than the detection radius (98%) is used as the ROI; finally, the pixel values ​​of the area outside the ROI are set to 0, and the original image information within the ROI is retained; this cropping method can effectively remove the interference information of the wafer edge and increase the model's attention to defects in the central area; Image denoising, first, Gaussian filtering is used to remove Gaussian white noise; Gaussian filtering can effectively smooth the image and remove noise by convolving with the Gaussian kernel; specifically, the calculation formula of the Gaussian filter is: Where σ is the standard deviation of the Gaussian kernel, x and y represent the pixel positions in the image, which are used to represent the offset of each pixel to the center of the filter; this offset is used to calculate the value of the Gaussian distribution function to determine the weight of the pixel; by performing convolution operations with the image, Gaussian filtering can effectively reduce the Gaussian white noise in the image; Data augmentation techniques are implemented through the Albumentations library and applied in real time during training: Random rotation: randomly rotate the image in the range of [-10°, 10°]; Random scaling: randomly scale the image in the range of [0.9, 1.1]; Random translation: randomly translate the image within the range of ±5% of the image width and height; Random brightness and contrast adjustment: the brightness adjustment range is [0.8, 1.2], and the contrast adjustment range is [0.8, 1.2]; Random Gaussian noise: add Gaussian noise with a mean of 0 and a standard deviation in the range of [0, 0.05]; Random horizontal and vertical flip: flip with a probability of 50% each; Each original image will generate 5 enhanced images, thus expanding the data set to 500,000 images; the image data after data preprocessing is recorded as data; (3) Labeling ruling: The industrial-grade labeling software LabelImg is used, and the labeling process follows the following rules: Strict pixel-level annotation: Annotators need to accurately draw the bounding box of the defect area on the image to ensure the accuracy and precision of the annotation; Defect classification, regression and circular boundaries as labels: the classification label is denoted as label cls , corresponding to five defect types (no defect, scratch, crack, peeling and foreign matter attachment), the value of each dimension represents the confidence score of the defect type; the regression label is recorded as label reg Contains the bounding box coordinate information of the predicted target, which is used to accurately locate the location of the defect; the circular boundary label is recorded as label cir The circular parameters closely related to the defect boundary are given, which can further improve the accuracy of boundary positioning; Multiple defect annotations: Each image may contain 0 or more defect annotations. The annotator needs to mark all defect areas in the image according to the actual situation. Annotation result saving: Annotation results are saved in PASCAL VOC format; The annotation process requires the annotator to have certain professional knowledge and experience to ensure the accuracy and usability of the annotation results; (4) Dataset preparation: The labeled data set is recorded as (data, label), and the training set, validation set, and test set are divided into 8:1:1 ratios, specifically: 80% of the data is used as a training set, 10% of the data is used as a validation set, and 10% of the data is used as a test set.

3. The visual recognition method according to claim 1, characterized in that: The image dynamic multi-scale feature extraction and feature enhancement network construction process in S2 includes: The input wafer image is processed from the bottom layer to the top layer. First, feature maps {C2, C3, C4, C5} of different scales are extracted, and then a multi-scale feature enhancement network is constructed according to the following steps: Starting from the deepest P5 layer, the upper-layer wafer feature map is generated by upsampling, and this high-level semantic information is passed to the top layer of the multi-scale feature enhancement network: M5=C5 Among them, M5 is the top-level feature map in the multi-scale feature enhancement network, which comes from the deepest C5 feature map, and then M5 is adjusted by 1×1 convolution channel number: P5=Conv1×1(M5) Then build a multi-scale feature enhancement network from top to bottom: M4=C4+Upsample(P5) P4=Conv3×3(M4) Among them, Conv3×3 is a 3×3 convolution, which is used to fuse features of different scales. Upsample is an upsampling operation implemented by deconvolution. Repeat the above steps to gradually generate {P3, P4, P5}. Finally, P3, P4, P5 are multi-scale feature enhancement networks that fuse multi-scale information. The network incorporates a feature weight dynamic allocation module and learns the correlation between features at different positions in the wafer feature map. By highlighting important features and suppressing irrelevant features, the mechanism first generates a spatial attention submodule, that is, the attention weight corresponding to each position, and then combines the spatial attention submodule with the original feature map to obtain an enhanced feature representation. The specific steps are as follows: The input of the feature weight dynamic allocation module is a wafer feature map X with a dimension of (C×H×W). Two 1×1 convolution kernels are used to perform channel compression on X to generate compressed wafer feature maps A and B. The dimensions of the compressed wafer feature map are (C′×H×W): A=X·W a (C′×H×W) B=X·W b (C′×H×W) Among them, W a and W b are the weights of two 1×1 convolution kernels respectively. A and B are used to calculate the similarity between positions and generate the spatial attention submodule map M with a dimension of (H×W×H×W): M=Softmax(A T ·B) Then, the original input X and M are element-wise multiplied to obtain the wafer feature map X′ weighted by the spatial attention submodule: X′=X·M Finally, X′ and X are added and then output through a convolutional layer: O=γ·X′+X·W c Among them, γ is a learnable scaling factor used to scale the effect of the spatial attention submodule enhancement, W c is a 1×1 convolution kernel weight, and O is the final output wafer feature map of the feature weight dynamic allocation module.

4. The visual recognition method according to claim 1, characterized in that: The process of constructing the multi-task collaborative wafer defect detection model in S3 includes: The model cascades the feature maps of the two subtasks through the convolution layer and encodes each other's information to obtain classification and regression features that integrate bidirectional relationships; the multi-task collaborative detection algorithm (MCDA) is used to introduce a circular boundary prediction branch, and the classification and regression prediction results are cascaded. The specific process includes the following: The feature O output by the dynamic multi-scale feature extraction and feature enhancement network is fed into MCDA to complete classification, regression and circular boundary prediction at the same time: cls pred ,reg pred ,surgery pred =MCDA(O) Among them, cls pred ,reg pred ,cir pred They are the classification prediction results, regression prediction results, and circular boundary prediction results of wafer defect detection; The specific calculation process of MCDA is as follows: 1) Perform basic convolution on 0 to obtain the initial prediction value for classification and regression {cls init , reg init }: cls init ,reg init =Conv(O) 2) reg init With cls init Perform relationship encoding to obtain the regression feature reg that integrates the relationship between the two fused : reg fused =reg init +Conv(Concat(reg init ,cls init )) 3) At the same time, cls init With reg init Perform relationship encoding to obtain the classification feature cls that integrates the relationship between the two fused : cls fused =cls init +Conv(Concat(cls init ,reg init )) 4) reg fused and cls fused Perform convolution to obtain the final classification prediction cls pred and regression prediction reg pred : 5) Set cls pred and reg pred Cascaded, through an additional convolution branch, predicts the defect boundary circular parameter cir which is closely related to the defect boundary pred : cir pred =Conv(Concat(cls pred ,reg pred )) A multi-task collaborative loss function is used to simultaneously optimize the three tasks of classification, regression, and circular boundary prediction, namely: L=λ1L cls +λ2L reg +λ3L cir Among them, L cls For classification loss, use FocalLoss; L reg For regression loss, CIoULoss is used; L cir For the annular boundary loss, PolygonLoss is used; λ1, λ2, and λ3 are hyperparameters for balancing the loss terms; Under the optimization of multi-task collaborative loss function, the object detection network can simultaneously output three prediction results: classification prediction result clspred, regression prediction result regpred and circular boundary prediction result cirpred; Among them, the classification prediction result clspred is a vector, corresponding to four defect types (scratches, cracks, peeling, and foreign matter attachment), and the value of each dimension represents the confidence score of the defect type; By obtaining the dimension index with the highest score, the defect type in the current prediction box can be determined; the regression prediction result regpred contains the bounding box coordinate information of the predicted target; the circular boundary prediction result cirpred gives the circular parameters that are closely related to the defect boundary; After obtaining the above three prediction results, the defect type can be determined, and combined with precise position and shape information, accurate detection of various wafer defects can be achieved, providing an important basis for subsequent quality control and defect repair.

5. The visual recognition method according to claim 1, characterized in that: The adversarial collaborative learning training strategy process in S4 includes: (1) Calculate multi-scale temporal consistency loss: This paper proposes a multi-scale temporal consistency loss, which measures the consistency of predictions at different time scales: L tc =∑ s λ s *D KL (P(Y|X t ),P(Y|X{t-s})) Among them, s represents different time scales, λ s is the corresponding weight, D KL is the KL divergence, P(Y|X t ) represents the detection distribution at time step t, including classification detection, regression detection and circular boundary detection, which is a summary of the total detection results; this loss encourages the model to maintain consistency in predictions at different time scales, thereby improving the modeling ability of different time batch data; (2) Adversarial training strategy: An adversarial training strategy is introduced, specifically: a discriminator D is designed to try to distinguish whether the features come from local details or global semantic information; the fusion network F is trained to deceive the discriminator: L adv =E[log(D(F(X v ,X a )))]+E[log(1-D(X v ))+log(1-D(X a ))] Among them, X v and X a They are local details and global semantic information input respectively; (3) Course learning and difficulty adaptation: According to (2) adversarial strategy, a curriculum learning strategy based on sample difficulty is proposed; the sample difficulty D(x) is defined as: D(x)=1-exp(-γ(w v L v +w a L a )) Among them, L v and L a are the loss of local details and global semantic information respectively, and γ is an adjustable parameter; during the training process, the proportion of difficult samples is gradually increased: Among them, t is the current training step number, T is the total training step number, p0 is the initial difficult sample ratio, and μ controls the rate of difficulty increase; (4) Dynamic Batch Normalization: Using Dynamic Batch Normalization (DBN) technology, DBN dynamically adjusts the normalization parameters according to the statistical characteristics of the input: y=γ(x)(x-μ(x)) / (σ(x)+ε)+β(x) Where γ(x) and β(x) are input-dependent scaling and offset parameters; (5) Model training: Based on the above method, the specific process of model training is as follows: 1) Initializing the model parameters θ, where θ represents the parameters to be updated, including all weight matrices and bias vectors in the wafer defect detection model; 2) For each training step t: a. Sample a batch from the dataset, press p hard The proportion of (t) contains difficult samples; b. Perform forward propagation to obtain multi-scale features and prediction results; c. Calculate the multi-scale temporal consistency loss L tc ; d. Perform adversarial training strategy to update the discriminator D and fusion network F; e. Apply dynamic batch normalization; f. Calculate the total loss: L total =λ tc *L tc +λ adv *L adv ; g. Back propagation, update model parameters: in is the learning rate; 3) Repeat step 2) and terminate the training process when the loss cannot be reduced after N consecutive epochs; save the final model parameters at the end of the training as the deployment model for wafer defect detection.

6. The visual recognition method according to claim 1, characterized in that: The wafer defect detection model deployment process in S5 includes: (1) Hardware platform selection and configuration: a. Select suitable edge computing devices, such as NVIDIA Jetson series or Intel NUC high-performance embedded systems, based on the actual needs of the semiconductor wafer production line; b. Configure the necessary deep learning frameworks and dependent libraries on the selected hardware platform: CUDA, cuDNN, etc.; c. Optimize hardware resource allocation and reasonably set GPU memory usage limits and CPU thread numbers; (2) Model integration and interface development: a. Design and implement the model reasoning interface, including image preprocessing, model reasoning and result post-processing functional modules; b. Develop a communication interface with the production line control system to achieve real-time transmission and feedback of test results; c. Build a cache mechanism, optimize data stream processing, and reduce the impact of I / O operations on detection speed; (3) Real-time image acquisition and preprocessing: a. Real-time acquisition of wafer images through high-speed industrial cameras; b. Implement the image preprocessing pipeline, including center cropping, denoising, and data augmentation operations; c. Use asynchronous processing mechanism to perform preprocessing while image acquisition; (4) Defect detection and result output: a. Input the preprocessed image into the deployed wafer defect detection model to perform inference operations; b. Parse the model output to extract defect category, bounding box coordinate information, and circle parameter information; c. Screen and classify the test results according to the preset threshold; (5) Visualization and storage of test results: a. Develop a real-time visualization interface to intuitively display the inspection results, including defect categories, bounding box coordinate information, and circle parameter information; b. Realize the local storage function of the test results, including original images, test results and related data; (6) Production line linkage and feedback: a. Transmit the test results to the production line control system in real time for automated decision-making; b. Develop an alarm mechanism to promptly notify relevant personnel and trigger an emergency response on the production line when a serious defect is detected; c. Realize the correlation analysis between test results and production parameters to provide data support for process optimization; (7) Performance evaluation and continuous optimization: a. Establish a regular performance evaluation mechanism, including statistical analysis of key indicators of detection accuracy, recall rate and processing speed; b. Develop an automated testing process and use standard test sets to evaluate the performance of the model in the actual production environment; c. Based on performance evaluation results and production feedback, continuously optimize model parameters and deployment strategies to continuously improve the overall performance of the detection system.

7. A visual recognition device based on computer processing, characterized in that: The visual recognition device uses any one of the visual detection methods of claims 1 to 6.

Images