Defect classification method and system for semiconductor chip and readable storage medium

Abstract

The invention provides a defect classification method and system for a semiconductor chip and a readable storage medium. The training method comprises the following steps: acquiring training set data, wherein the training set data comprises different types of defect images of the semiconductor chip; inputting the training set data into an initial defect classification model based on a deep convolutional neural network so as to train the initial defect classification model; in the training process, an attention mechanism module is introduced to the tail of at least part of convolution blocks of the initial defect classification model, and residual connection is introduced to at least part of the convolution blocks; and a target defect classification model is obtained after multiple rounds of iterative training. The method effectively balances model performance and computing resource requirements, and ensures that higher efficiency is kept while the classification precision is improved.

Description

Technical Field

[0001] The present application relates to the field of semiconductor manufacturing technology, and in particular to a defect classification method, system and readable storage medium for semiconductor chips. Background Art

[0002] As the core component of modern electronic devices, the quality of integrated circuits directly affects the performance and reliability of the entire product. Tiny chip defects, such as microcracks, uneven doping, or metal shorts, may cause device performance degradation or even cause system failures. Therefore, the classification of surface defects becomes an indispensable part of the manufacturing process of semiconductor chips after packaging. Accurate defect classification not only helps to distinguish between normal and abnormal chips, but also plays a vital role in the quality control of factory chips, and can provide timely feedback on chip production status, thereby optimizing the manufacturing process.

[0003] Traditional defect detection and classification methods mainly rely on manual inspection and simple automated tools. Although manual inspection can find some obvious defects, its accuracy and consistency are low due to the subjectivity and fatigue of manual operation. In addition, traditional automated inspection technology often performs poorly when dealing with complex backgrounds and tiny defects, making it difficult to meet the needs of high-precision, high-efficiency inspection and classification in modern semiconductor manufacturing processes. Summary of the invention

[0004] In order to solve the above problems, according to a first aspect of the present invention, a method for training a defect classification model for a semiconductor chip is provided, comprising the following steps:

[0005] Acquire training set data, wherein the training set data includes different types of defect images of semiconductor chips;

[0006] Inputting the training set data into an initial defect classification model based on a deep convolutional neural network to train the initial defect classification model;

[0007] During the training process, an attention mechanism module is introduced at the end of at least part of the convolution blocks of the initial defect classification model, and a residual connection is introduced in at least part of the convolution blocks;

[0008] After multiple rounds of iterative training, the target defect classification model is obtained.

[0009] Optionally, during the training process, an attention mechanism module is introduced at the end of at least part of the convolution blocks of the initial defect classification model, and a residual connection is introduced at least part of the convolution blocks, comprising the following steps:

[0010] Obtaining importance index values ​​of each convolution block in the initial defect classification module;

[0011] Compare the importance index value in each convolution block with the preset threshold in each convolution block;

[0012] The convolution block whose importance index value is greater than the preset threshold is used as the target convolution block, and the attention mechanism module is introduced at the end of the target convolution block.

[0013] Optionally, the method for determining the preset threshold comprises the following steps:

[0014] Define the performance indicator function P(V);

[0015] According to the performance index P(V t-1 ) and the performance index P(V t ) Adjust the threshold V for the next round of training t+1 ;

[0016] When the threshold value changes by less than a preset range in multiple rounds of iterations, the adjustment of the threshold value is stopped, and the current threshold value is determined to be the preset threshold value.

[0017] Optionally, the performance indicator P(V t-1 ) and the performance index P(V t ) Adjust the threshold V for the next round of training t+1 , including the following steps:

[0018] The weight parameter λ of the current round of training is calculated according to the following formula: t and γ t , the weight parameter λ of the previous round of training t-1 and γ t-1 ,

[0019]

[0020] Among them, i represents the i-th round of training, α and β are learning rates, and λ i is the first weight parameter of the i-th round of training, γ i is the second weight parameter of the i-th round of training, V i-1 is the threshold of the model in the i-1th round of training, A(V i-1 ) is the classification accuracy of the model in the i-1th round of training, ΔA(V i-1 ) is the difference in classification accuracy between the i-1th round of training and the i-2th round of training, C(V i-1 ) is the computational cost of the model in the i-1th round of training, ΔC(V i-1 ) is the difference in computational cost between the i-1th round of training and the i-2th round of training;

[0021] The performance index P(Vt-1 ) and the performance index P(V t ),

[0022] P(V i )=λ i ×A(V i )-γ i ×C(V i );

[0023] The threshold V for the next round of training is calculated according to the following formula t+1 , V t+1 =V t +η×sign(P(V t )-P(V t-1 )), where η represents the step size parameter.

[0024] Optionally, during the training process, an attention mechanism module is introduced at the end of at least part of the convolution blocks of the initial defect classification model, and in the step of introducing residual connections in at least part of the convolution blocks, residual connections are introduced in the last N convolution blocks, where N is a positive integer greater than or equal to 3.

[0025] According to a second aspect of the present invention, a method for classifying defects in a semiconductor chip is provided, comprising the following steps:

[0026] Acquire a target semiconductor chip image to be classified;

[0027] The target semiconductor chip image is input into a target defect classification model constructed using the training method described in any one of claims 1 to 5 to obtain a defect classification result corresponding to the target semiconductor chip image.

[0028] Optionally, the step of acquiring the target semiconductor chip image to be classified comprises the following steps:

[0029] Acquire an original semiconductor chip image;

[0030] Selecting windows of different scales for each pixel in the original semiconductor chip image;

[0031] Perform discrete Fourier transform on each window of selected scale, convert the spatial domain data in the window into frequency domain data, and obtain the corresponding frequency domain feature map;

[0032] The frequency domain feature maps obtained at all scales are fused to generate a comprehensive frequency domain feature map;

[0033] The comprehensive frequency domain feature map is combined with the spatial domain feature map of the original image to generate a multi-scale joint feature map as a target semiconductor chip image to be classified.

[0034] Optionally, in the step of performing a discrete Fourier transform on each window of selected scale, converting the spatial domain data in the window into frequency domain data, and obtaining a corresponding frequency domain feature map, after converting the spatial domain data in the window into frequency domain data, it also includes applying a frequency filter to the frequency domain data to respectively extract high-frequency information and low-frequency information.

[0035] Optionally, inputting the target semiconductor chip image into a target defect classification model constructed by the aforementioned training method to obtain a defect classification result corresponding to the target semiconductor chip image comprises the following steps:

[0036] When the target semiconductor chip image includes a target defect, the defect classification result is a defect type corresponding to the target semiconductor chip image, and the target defect is a defect included in a training set;

[0037] When the target semiconductor chip image contains an unknown defect, the defect classification result is an unclassified result, a low confidence classification result or an abnormal detection result, and the unknown defect is a defect not included in the training set;

[0038] When the target semiconductor chip image contains no defects, the defect classification result is no defect.

[0039] According to a third aspect of the present invention, there is provided a semiconductor chip defect classification system, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0040] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the steps corresponding to the aforementioned defect classification method.

[0041] According to a fourth aspect of the present invention, a readable storage medium is provided, wherein the readable storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement steps corresponding to the aforementioned defect classification method.

[0042] According to the scheme of the embodiment of the present invention, the introduction of the attention mechanism module in some convolution blocks can selectively focus on the important feature areas in the image and filter out irrelevant information. For some convolution blocks that introduce the attention mechanism module, it can significantly enhance the sensitivity of these convolution blocks to different types of defects and improve the effectiveness of feature extraction. The introduction of residual connections in some convolution blocks can help the model train deep networks more effectively, avoid the gradient vanishing problem, and accelerate the convergence speed. The existence of residual connections enables these convolution blocks to directly transmit key information through jump connections, thereby improving the stability and efficiency of training. The local application of the attention mechanism module and the residual connection enables the model to have different feature extraction capabilities in different convolution blocks. This selective application not only reduces the computational complexity, but also improves the defect classification accuracy of the model in local areas, especially when there are complex and diverse defect types in the defect image of semiconductor chips. Therefore, this method effectively balances the model performance and computing resource requirements, ensuring that high efficiency is maintained while improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic flow chart of a method for training a defect classification model for a semiconductor chip according to an embodiment of the present invention is shown;

[0044] Figure 2 Shows Figure 1 A schematic flow chart of a method for acquiring a training set in step S100 is shown;

[0045] Figure 3 Shows Figure 2 A schematic flowchart of a method for determining at which ends of convolution blocks attention mechanism modules are introduced in step S300;

[0046] Figure 4 A schematic flow chart of a method for determining a preset threshold according to an embodiment of the present invention is shown;

[0047] Figure 5 Shows Figure 4 The threshold value V in the next round of training in step S322 is shown as t+1 A schematic flow chart of an adjustment method;

[0048] Figure 6 FIG. 1 is a schematic flow chart showing a method for classifying semiconductor chip defects according to an embodiment of the present invention;

[0049] Figure 7 Shows Figure 6 A schematic flow chart of a method for generating a defect type corresponding to a target semiconductor chip image in step S120 is shown;

[0050] Figure 8A schematic structural diagram of a semiconductor chip defect classification system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0053] In addition, in order to better illustrate the present invention, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0054] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0055] The "multiple" in the embodiments of the present invention refers to two or more. The first, second, etc. descriptions in the embodiments of the present invention are only used for illustration and distinction of the described objects, and have no order, nor do they represent a special limitation on the number in the embodiments of the present invention, and cannot constitute any limitation on the embodiments of the present invention.

[0056] Figure 1 FIG. 1 is a schematic flow chart of a method for training a defect classification model for a semiconductor chip according to an embodiment of the present invention. Figure 1 As shown, the training method includes:

[0057] Step S100, acquiring training set data, where the training set data includes different types of defect images of semiconductor chips;

[0058] Step S200, inputting the training set data into an initial defect classification model based on a deep convolutional neural network to train the initial defect classification model;

[0059] Step S300, during the training process, introducing an attention mechanism module at the end of at least part of the convolution blocks of the initial defect classification model, and introducing a residual connection at least part of the convolution blocks;

[0060] Step S400: After multiple rounds of iterative training, a target defect classification model is obtained.

[0061] According to the scheme of the embodiment of the present invention, the introduction of the attention mechanism module in some convolution blocks can selectively focus on the important feature areas in the image and filter out irrelevant information. For some convolution blocks that introduce the attention mechanism module, it can significantly enhance the sensitivity of these convolution blocks to different types of defects and improve the effectiveness of feature extraction. The introduction of residual connections in some convolution blocks can help the model train deep networks more effectively, avoid the gradient vanishing problem, and accelerate the convergence speed. The existence of residual connections enables these convolution blocks to directly transmit key information through jump connections, thereby improving the stability and efficiency of training. The local application of the attention mechanism module and the residual connection enables the model to have different feature extraction capabilities in different convolution blocks. This selective application not only reduces the computational complexity, but also improves the defect classification accuracy of the model in local areas, especially when there are complex and diverse defect types in the defect image of semiconductor chips. Therefore, this method effectively balances the model performance and computing resource requirements, ensuring that high efficiency is maintained while improving classification accuracy.

[0062] Figure 2 Shows Figure 1 The schematic flow chart of the method for obtaining the training set in step S100 is shown. The method for obtaining the training set may include:

[0063] Step S101, obtaining raw data. The raw data is image data of semiconductor chips obtained from the semiconductor chip manufacturing process, detection equipment or database. These data come from scanned images on the actual production line, images taken by detection instruments or annotated public data sets.

[0064] Step S102: Capture the semiconductor chip image, which may be captured using a high-resolution camera or other image acquisition equipment.

[0065] Step S103, defect annotation is performed on the captured image. Defect annotation can include annotation of different types of defect locations and categories. Annotation can be performed by manual review or using automatic annotation tools. Common defect types include cracks, bubbles, impurities, defects, etc. Images without defects are annotated as defect-free.

[0066] Step S104, preprocess the defect-annotated images and apply data enhancement technology to increase the diversity of training data. The preprocessing process may include image denoising, normalization, cropping, resizing, etc. Data enhancement technology may include rotation, translation, scaling, and flipping operations.

[0067] Step S105, divide the processed data into a training set, a validation set and a test set. The training set is used for model training, the validation set is used to adjust model parameters and select the best model, and the test set is used to evaluate the final performance of the model.

[0068] Through the above steps, it is ensured that the training set data can be used completely and accurately to train the semiconductor chip defect classification model, laying a solid foundation for subsequent model training and optimization.

[0069] In step S200, the training set data input into the initial defect classification model is the training set data after format conversion. The format conversion includes image size adjustment and channel number (such as RGB) setting to ensure that the data can be correctly input into the model.

[0070] Figure 3 Shows Figure 2 FIG. 4 is a schematic flow chart of a method for determining at which end of a convolutional block the attention mechanism module is introduced in step S300. Figure 3 As shown, the method for determining which convolution blocks to introduce the attention mechanism module at the end includes:

[0071] Step S301, obtaining the importance index value of each convolution block in the initial defect classification module;

[0072] Step S302, comparing the importance index value in each convolution block with a preset threshold in each convolution block;

[0073] Step S303, taking the convolution block whose importance index value is greater than the preset threshold as the target convolution block, and introducing the attention mechanism module at the end of the target convolution block.

[0074] In step S301, the importance index can reflect the contribution of the convolution block to the final model output. The importance index includes gradient information, feature map activation intensity or parameter sensitivity. The importance index can be calculated by the existing gradient backpropagation method, feature map activation method and parameter sensitivity method. In the gradient backpropagation method, the gradient of each convolution block is calculated by backpropagation. The larger the gradient, the greater the influence of the convolution block on the model decision. In the feature map activation method, the activation of the feature map output by the convolution block is analyzed. The convolution block with a high activation value has a more obvious response to a specific category of defects. In the parameter sensitivity method, the degree of influence on the model output is observed by changing the parameters of the convolution block. The convolution block with a larger change is more important. The importance index is normalized to obtain the importance index value, which can be between 0 and 1, for example.

[0075] In step S302, the preset threshold is defined in the model training process to distinguish the importance of the convolutional blocks. It can be determined in the following ways.

[0076] The first method is based on experience. In the early stage when there is no ready-made data, a preliminary judgment can be made based on the depth or position of the convolution block. For example, the convolution blocks close to the input layer usually capture low-level features such as edges, textures, etc., so the threshold can be set lower, while the convolution blocks close to the output layer usually capture high-level features such as object shape, category, etc., and the threshold can be set higher. This method does not require additional computing resources and directly relies on expert experience or intuitive understanding of the network structure. This is especially useful when there is insufficient data or time is limited in the early stage.

[0077] The second method is based on data statistics. For example, based on the activation value statistics, the output feature map of the convolution block will have different activation value distributions. By counting the activation values ​​of the output feature map of the convolution block, the overall impact of the feature map on the model output can be obtained. The higher the activation value of the convolution block, the more important the features extracted by the convolution block are for classification. The threshold is set to a certain percentile value (for example, the 75th percentile), and convolution blocks above the threshold are considered important convolution blocks. For example, it can also be set based on gradient statistics. By calculating the gradient information in the convolution block, the impact of each convolution block on the loss function can be understood. A convolution block with a large gradient value means that it contributes more to the loss optimization of the model and can be set as a high-importance convolution block. The threshold is set according to the average or standard deviation of the convolution block gradient, for example, it is set to 1.5 times the standard deviation of the average gradient. In this way, the activation value or gradient statistics are based on the performance of the model on a specific data set, which can provide a more accurate contribution assessment for each convolution block. As the training data changes, the statistical indicators can dynamically reflect the importance of the convolution block, avoiding relying solely on fixed experience. Through precise statistics, the attention mechanism is introduced only for important convolution blocks, reducing the computational complexity of the network.

[0078] The third method is based on performance improvement settings. For example, it can be a step-by-step optimization method. In the initial training stage of the model, multiple different thresholds are set to analyze the role of each convolution block. For example, after some training rounds, observe the impact of different thresholds on the overall performance of the model, and then gradually adjust these thresholds to select those threshold ranges that can significantly improve the classification accuracy of the model. Another example is a dynamic adjustment method. As the training progresses, the threshold can be dynamically adjusted according to the performance of the convolution block after each round of training. In this method, as the training process progresses, the dynamic adjustment of the threshold can effectively improve the performance of the model, especially when the amount of data is large or the problem is complex. Moreover, through multiple rounds of iterations and performance feedback after training, the importance evaluation and threshold setting of the convolution block can be gradually optimized to maximize the model accuracy.

[0079] The preset threshold is standardized to a value between 0 and 1 to facilitate comparison with the normalized importance index. The benefit of this normalization is that it makes the comparison between different convolution blocks intuitive and unified, avoiding unfair comparisons due to different scales. If the preset threshold is set to 0, it means that all convolution blocks will be introduced into the attention mechanism, which will make the model too complicated. If the threshold is set to 1, it means that only the most important convolution blocks (importance index reaches the maximum value) will be introduced into the attention mechanism, which will ignore some equally important convolution blocks whose indicators have not reached the maximum value.

[0080] Figure 4 FIG. 1 is a schematic flow chart of a method for determining a preset threshold according to an embodiment of the present invention. The method for determining the preset threshold is the aforementioned dynamic adjustment method. Figure 4 As shown, the determination method includes:

[0081] Step S321, defining a performance indicator function P(V);

[0082] Step S322: According to the performance index P(V t-1 ) and the performance index P(V t ) Adjust the threshold V for the next round of training t+1 ;

[0083] Step S323: When the threshold value changes by less than a preset range in multiple rounds of iterations, the adjustment of the threshold value is stopped, and the current threshold value is determined to be the preset threshold value.

[0084] This dynamic adjustment method can gradually optimize the model according to the actual performance of each round of training by dynamically adjusting the preset threshold during the training process. Compared with the fixed threshold method, it can adapt to changes in the data set more flexibly, improve the adaptability of the model on different convolution blocks, and thus improve the classification accuracy. In addition, this method continuously compares the performance indicators of the current and previous rounds of training to ensure that each adjustment of the threshold is based on the actual model performance. When the performance indicators change, the threshold will be adjusted in time, thereby helping the model identify the most important convolution blocks and preferentially introduce attention mechanisms or other optimization measures on these convolution blocks, ultimately improving the accuracy of the classification model. At the same time, it is set to stop adjusting when the threshold change is less than the preset value to prevent excessive or invalid adjustments. This not only reduces unnecessary computational burden, but also can determine the final threshold in time when the model performance tends to be stable, avoiding long-term invalid iterations.

[0085] In step S321, the performance indicator function P(V) is used to measure the performance of the model under different thresholds V. In this embodiment, the performance indicator function is defined based on the key performance indicators of the model. The formula of the performance indicator function P(V) is defined as P(V)=λ×A(V)-γ×C(V). A(V) represents the classification accuracy, which measures the classification accuracy of the model for semiconductor chip defects under the current threshold V. The higher the accuracy, the more favorable the current threshold is for the classification task of the model. C(V) represents the computational cost, which represents the computational resource consumption of the model for training and reasoning under the current threshold V, including computing time, memory usage, hardware resource usage, etc. By controlling the computational cost, the model can be avoided from being overly complicated. λ and γ both represent weight parameters. The weight parameter λ represents the degree of importance attached to the classification accuracy. The weight parameter γ represents the degree of importance attached to the computational cost, especially in the case of limited resources, where computing time, memory usage, etc. are important constraints of the model.

[0086] Figure 5 Shows Figure 4 The threshold value V in the next round of training in step S322 is shown as t+1 A schematic flow chart of the adjustment method. Figure 5 As shown, the adjustment method includes:

[0087] Step S3221, calculate the weight parameter λ for the current round of training according to the following formula: t and γ t , the weight parameter λ of the previous round of training t-1 and γ t-1 ,

[0088]

[0089] Step S3222: Calculate the performance index P(V) of the previous round of training according to the following formula:t-1 ) and the performance index P(V t ),

[0090] P(V i )=λ i ×A(V i )-γ i ×C(V i );

[0091] Step S3223, calculate the threshold V for the next round of training according to the following formula: t+1 , V t+1 =V t +η×sign(P(V t )-P(V t-1 )).

[0092] In step S3221, i represents the i-th round of training, α and β are learning rates, λ i is the first weight parameter of the i-th round of training, γ i is the second weight parameter of the i-th round of training, V i-1 is the threshold of the model in the i-1th round of training, A(V i-1 ) is the classification accuracy of the model in the i-1th round of training, ΔA(V i-1 ) is the difference in classification accuracy between the i-1th round of training and the i-2th round of training, C(V i-1 ) is the computational cost of the model in the i-1th round of training, ΔC(V i-1 ) is the difference in computational cost between the i-1th round of training and the i-2th round of training. The weight parameter λ of the current round of training t and γ t , the weight parameter λ of the previous round of training t-1 and γ t-1 When calculating, just replace i in the formula of this step with t and t-1.

[0093] Through this step, the model can dynamically adjust the weight parameter λ according to the changes in classification accuracy. If the classification accuracy is significantly improved, the formula will increase the weight parameter λ, which means that the model will pay more attention to improving the accuracy in the next round of training. The step size is controlled by the learning rate α to prevent the weight adjustment from being too fast and causing unstable training. The model can dynamically adjust the weight parameter γ according to the changes in computing resource consumption. If the computing cost increases significantly, the formula will increase the weight parameter γ, which means that in the next round of training, the model will pay more attention to resource optimization and consumption reduction. Similarly, by controlling the learning rate β, the adjustment process can be smooth and gradual.

[0094] When the accuracy difference improves significantly, while the computational cost increases slightly, the weight parameter λ in the formula increases faster than the weight parameter γ. This dynamic adjustment mechanism can achieve an adaptive balance between model performance and resource consumption. In other words, this adaptive balance mechanism allows the model to intelligently allocate resources and focus points based on the current training status. When the classification accuracy improves significantly, the system will give priority to performance improvement; when the computational cost increases too quickly, the system will tend to optimize resources. This method can dynamically adjust the focus of the model to ensure the best performance under resource constraints.

[0095] Learning rates α and β are important parameters that affect training stability and convergence speed, and usually need to be adjusted according to specific tasks and model structures. In one embodiment, the learning rates α and β are both preset values. The ranges of learning rates α and β are, for example, between 0.0001 and 0.01. The learning rate α can be, for example, 0.001, 0.002, 0.005, 0.008, etc. The learning rate β can be, for example, 0.0005, 0.0004, 0.0006, 0.0008, etc. When the learning rates α and β are at the lower limit of 0.0001, a smaller learning rate is suitable for complex models or data sets with large noise. In these cases, using a smaller learning rate can prevent the model from updating too much and causing fluctuations in the loss function, ensuring that the training process is more stable, especially in the later fine-tuning, a smaller learning rate can help the model to adjust the weights carefully to avoid skipping the local optimal solution. When the learning rates α and β are at the upper limit of 0.01, a larger learning rate is suitable for rapid convergence in the initial stage. When the model is in the early stage of training, it can speed up the convergence and reduce the training time. If the learning rate is too large, the training process will be unstable. However, the training process can be effectively controlled within the above range. For some simple data sets or models with fewer parameters, using a larger learning rate can help the model approach the optimal solution faster.

[0096] In step S3222, P(V i ) represents the comprehensive performance index of the i-th round of training, which reflects the overall performance of the model in the current training round, including the accuracy improvement of the model in this round and the balance between the training cost. The weight parameter λ i Indicates the importance of model accuracy in the i-th round. This parameter will be dynamically adjusted according to the performance of the model training to weigh the contribution of performance improvement. i ) represents the improvement of the model in a specific performance metric (such as accuracy, precision, recall, etc.) in the i-th round of training. This indicator reflects the effect of the positive improvement of the model in this round of training and is measured based on the performance on the validation set or test set. Weight parameter γ i It represents the trade-off of training cost in the i-th round, reflecting the importance of cost in the model optimization process. i) represents the price or cost of the i-th round of training. The cost may include the computational resource consumption of the model (such as time and memory), the amplitude of gradient fluctuation, or the degree of overfitting and other unfavorable factors.

[0097] λ i and γ i The value of is dynamically adjusted according to the current performance of the model. By adjusting these two weight parameters, the training system can more flexibly adapt to the training needs of different stages. i When it is larger, the model pays more attention to performance improvement, that is, positive factors dominate. i When it is larger, the impact of training cost is more significant, indicating that the current stage pays more attention to cost optimization. Through this step, the training of the model can comprehensively consider performance improvement and training cost, achieve a more balanced optimization strategy, and avoid blindly pursuing accuracy while ignoring model complexity and resource consumption issues.

[0098] In step S3223, the step parameter η represents the amplitude of adjusting the threshold. The step parameter η controls the step size of each adjustment and is a small positive value to ensure the stability of the threshold adjustment process and prevent drastic fluctuations in model training. The value of the step parameter η is between 0.0001 and 0.01. sign(P(V t )-P(V t-1 )) is a sign function used to determine the direction of performance change in the current round of training. t ) is greater than P(V t-1 ), indicating that the performance of the current round of training is improved compared with the previous round. The symbol function sign(P(V t )-P(V t-1 )) is positive (i.e. 1), which means that the threshold for the next round needs to be increased. t ) is less than P(V t-1 ), indicating that the performance of the current round of training is poor, and the sign function sign(P(V t )-P(V t-1 )) is negative (i.e. -1), indicating that the threshold for the next round needs to be reduced. t ) is equal to P(V t-1 ), that is, the performance of the current round is the same as that of the previous round, the result of the sign function is 0, and the threshold is not adjusted at this time.

[0099] This step calculates the performance changes of each round. The threshold adjustment process is adaptive and can be adjusted dynamically according to the training results to ensure that the model can balance the importance of the convolution block and the overall training effect at different stages. The step size η controls the amplitude of the adjustment to prevent the model from experiencing drastic fluctuations or over-adjustments, ensuring the stability of model training. This method optimizes the threshold round by round to ensure that the importance assessment of the convolution block is more reasonable, and the model can find the best balance between performance and cost, thereby improving training efficiency and the generalization ability of the final model.

[0100] In step S300, residual connections are introduced in the last N convolution blocks, where N is a positive integer greater than or equal to 3. The last N convolution blocks are located at the end of the network, and these convolution blocks are responsible for extracting higher-level features and performing final feature integration. As the network depth increases, the gradient may become more difficult to propagate in these deep convolution blocks, so the introduction of residual connections can help alleviate this problem.

[0101] In step S300, the step of introducing residual connections in the last N convolution blocks can be a conventional operation. For example, it can include the steps of defining residual blocks, integrating the residual blocks into the initial defect classification model, and training and optimization. Among them, defining the residual block can include the following steps: calculating the standard output of the convolution block; adding the input signal to the standard output to obtain the final convolution block output; applying an activation function (such as ReLU) to the final output. In the step of integrating the residual block into the initial defect classification model, the above-mentioned residual block structure is integrated into the last N convolution blocks of the model to ensure that the model can effectively use the residual connection for training and optimization in this part. In the training and optimization steps, the training effect and performance changes after the introduction of the residual connection are observed to optimize the model.

[0102] In one embodiment, the deep convolutional neural network is selected as a ResNet architecture. When the deep convolutional neural network is a small network such as ResNet-18, N is set to 3 to 5. When the deep convolutional neural network is a medium-sized network such as ResNet-50, N is set to 10 to 20. When the deep convolutional neural network is a large network such as ResNet-101 and ResNet-152, N is set to 20 to 50.

[0103] Figure 6 FIG. 1 is a schematic flow chart of a method for classifying semiconductor chip defects according to an embodiment of the present invention. Figure 6 As shown, the defect classification method includes:

[0104] Step S110, obtaining a target semiconductor chip image to be classified, the target semiconductor chip image containing defects;

[0105] Step S120 , inputting the target semiconductor chip image into the target defect classification model constructed by the aforementioned training method, and obtaining the defect type corresponding to the target semiconductor chip image.

[0106] Figure 7 Shows Figure 6 FIG. 1 is a schematic flow chart of a method for generating a defect type corresponding to a target semiconductor chip image in step S120. Figure 7 As shown, the generation method includes:

[0107] Step S121, obtaining an original semiconductor chip image;

[0108] Step S122, selecting windows of different scales for each pixel in the original semiconductor chip image;

[0109] Step S123, performing discrete Fourier transform on each window of the selected scale, converting the spatial domain data in the window into frequency domain data, and obtaining a corresponding frequency domain feature map;

[0110] Step S124, fusing the frequency domain feature maps obtained at all scales to generate a comprehensive frequency domain feature map;

[0111] Step S125 , combining the comprehensive frequency domain feature map with the spatial domain feature map of the original image to generate a multi-scale joint feature map as the target semiconductor chip image to be classified.

[0112] In step S121 , a high-resolution image acquisition device, such as a microscope, an industrial camera, or a scanner, is used to capture an image of the semiconductor chip.

[0113] In step S122, multiple scales of window sizes are defined to capture features at different levels in the image. The scale is selected according to actual needs and the characteristics of the image. For example, a small window is used for subtle defects, and a large window is used for defects of larger or complex structures. The internal features of each window are extracted, and these features will be used for subsequent frequency domain conversion and feature fusion. Each window and its features are stored in an appropriate data structure (such as a matrix or array) for subsequent frequency domain conversion and feature fusion processing. By selecting windows of different scales for each pixel in the original semiconductor chip image, features of different sizes and ranges in the image can be captured. This multi-scale processing method can effectively extract various information from local details to overall structure.

[0114] In step S123, the spatial domain data in the window is converted into frequency domain data using a discrete Fourier transform (DFT), and the fast Fourier transform (FFT) algorithm can be used to improve the computational efficiency. DFT converts the image from the spatial domain to the frequency domain, and the obtained frequency domain feature map contains information about the frequency components in the image, which can be used to identify defects of different scales and types. Useful information such as frequency distribution, amplitude spectrum, and phase spectrum are extracted from the frequency domain feature map, and these features are used to describe the frequency domain characteristics of the image. The frequency domain features are normalized to ensure that features of different scales have similar dimensions and ranges in subsequent fusion. The frequency domain feature map of each scale is stored as a matrix or array for subsequent fusion processing. In a further embodiment, after converting the spatial domain data in the window to frequency domain data, it also includes applying a frequency filter to the frequency domain data to extract high-frequency information and low-frequency information respectively. More abundant and detailed image features can be obtained by applying a frequency filter to the frequency domain data and extracting high-frequency information and low-frequency information respectively. Low-frequency information helps to capture the large-scale structure and texture features of the image, while high-frequency information can reveal details and edge features in the image. This method can significantly improve the detection capability of defect classification models for various types of defects and enhance the classification model's ability to identify complex and subtle defects, thereby improving the overall classification performance. In addition, by performing a discrete Fourier transform on each window of the selected scale, the spatial domain data is converted into frequency domain data to obtain a frequency domain feature map. This conversion helps to reveal periodic patterns and frequency components in the image.

[0115] In step S124, the frequency domain feature maps of each scale are fused together using a weighted average method, and the weights can be set based on the importance or information content of each scale. The frequency domain feature maps of all scales are spliced ​​to form a comprehensive feature map containing information of all scales. The fused features are reduced in dimension using dimensionality reduction techniques such as PCA to reduce the feature dimension and retain the main information. A comprehensive frequency domain feature map is generated, containing information of all scales. This feature map will be used to describe the overall frequency domain characteristics of the image and help identify defects of different scales. The most useful frequency domain features are selected according to actual needs and classification effects to optimize the representation capabilities of the comprehensive feature map.

[0116] In step S125, the comprehensive frequency domain feature map is combined with the spatial domain feature map of the original image, which can be achieved by splicing, weighted averaging or other fusion methods. A multiscale joint feature map containing spatial domain and frequency domain information is generated to more comprehensively describe the features and defects of the image. The generated multiscale joint feature map is normalized to ensure that the dimensions of different features are consistent and to improve the accuracy of subsequent classification. The dimension of the feature map is adjusted to adapt it to the input requirements of the classification model. The multiscale joint feature map is input as the target semiconductor chip image to be classified into the classification model for defect classification. The comprehensive frequency domain feature map is combined with the spatial domain feature map of the original image to generate a multiscale joint feature map. By fusing the spatial domain and frequency domain features, the representation ability of defects in the image can be effectively improved, and the sensitivity of the classification model to the defect type can be increased.

[0117] The multi-scale joint feature map finally obtained by the embodiment of the present invention is used as the target semiconductor chip image to be classified, which can provide a more comprehensive and detailed feature input, help the classification model to more accurately identify and classify defects, and improve the overall classification performance and accuracy.

[0118] In step S120, when the target semiconductor chip image contains the target defect, the defect classification result is the defect type corresponding to the target semiconductor chip image, and the target defect is the defect included in the training set. When the target semiconductor chip image contains an unknown defect, the defect classification result is an unclassified result, a low-confidence classification result, or an abnormal detection result, and the unknown defect is a defect not included in the training set. When the target semiconductor chip image does not contain a defect, the defect classification result is no defect. That is, in the target semiconductor chip image, if it contains a defect that matches the known defect type in the training set, the classification model will output the corresponding defect type as the classification result to ensure that the target defect is correctly identified and classified. If the target semiconductor chip image contains an unknown defect that is not involved in the training set, the model may not be able to make an accurate classification. At this time, the classification result may be an invalid classification, a low-confidence classification, or trigger an abnormal detection mechanism, indicating that the defect type has not been identified. If no defects are detected in the target image, the model will output a "no defect" result, indicating that no abnormalities were found on the chip surface in this round of detection. Through this design, the classification system can effectively handle a variety of situations including known defects, unknown defects, and no defects, ensuring the accuracy and reliability of the detection process.

[0119] Figure 8 FIG. 2 shows a schematic structural diagram of a semiconductor chip defect classification system according to an embodiment of the present invention. Figure 8 As shown, the defect classification system includes a communication interface, a memory 2, a communication bus 3 and a control module 4. The control module 4, the communication interface and the memory 2 communicate with each other through the communication bus 3.

[0120] The memory is used to store computer programs. The control module is used to implement the steps of the above-mentioned defect classification method when executing the program stored in the memory.

[0121] The control module may be a central processing unit (CPU). The control module may also be other general control modules, digital signal control modules (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0122] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the defect classification method in the embodiment of the present invention. The control module executes various functional applications and data processing of the control module, namely, the defect classification method, by running the non-transitory software programs, instructions and modules stored in the memory.

[0123] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the control module, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the control module, and these remote memories may be connected to the control module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0125] An embodiment of the present invention further provides a readable storage medium, which stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the steps corresponding to the aforementioned defect classification method.

[0126] The above are only some specific implementation methods of the present application, and any other improvements made based on the concept of the present application are deemed to be within the protection scope of the present application.

Claims

1. A method for training a defect classification model for a semiconductor chip, characterized in that: The steps include: Acquire training set data, wherein the training set data includes different types of defect images of semiconductor chips; Inputting the training set data into an initial defect classification model based on a deep convolutional neural network to train the initial defect classification model; During the training process, an attention mechanism module is introduced at the end of at least part of the convolution blocks of the initial defect classification model, and a residual connection is introduced in at least part of the convolution blocks; After multiple rounds of iterative training, the target defect classification model is obtained.

2. The training method according to claim 1, characterized in that: In the training process, an attention mechanism module is introduced at the end of at least part of the convolution blocks of the initial defect classification model, and a residual connection is introduced at least part of the convolution blocks, including the following steps: Obtaining importance index values ​​of each convolution block in the initial defect classification module; Compare the importance index value in each convolution block with the preset threshold in each convolution block; The convolution block whose importance index value is greater than the preset threshold is used as the target convolution block, and the attention mechanism module is introduced at the end of the target convolution block.

3. The training method according to claim 2, characterized in that: The method for determining the preset threshold comprises the following steps: Define the performance indicator function P(V); According to the performance index P(V t-1 ) and the performance index P(V t ) Adjust the threshold V for the next round of training t+1 ; When the threshold value changes by less than a preset range in multiple rounds of iterations, the adjustment of the threshold value is stopped, and the current threshold value is determined to be the preset threshold value.

4. The training method according to claim 3, characterized in that: The performance index P(V t-1 ) and the performance index P(V t ) Adjust the threshold V for the next round of training t+1 , including the following steps: The weight parameter λ of the current round of training is calculated according to the following formula: t and γ t , the weight parameter λ of the previous round of training t-1 and γ t-1 , Among them, i represents the i-th round of training, α and β are learning rates, and λ i is the first weight parameter of the i-th round of training, γ i is the second weight parameter of the i-th round of training, V i-1 is the threshold of the model in the i-1th round of training, A(V i-1 ) is the classification accuracy of the model in the i-1th round of training, ΔA(V i-1 ) is the difference in classification accuracy between the i-1th round of training and the i-2th round of training, C(V i-1 ) is the computational cost of the model in the i-1th round of training, ΔC(V i-1 ) is the difference in computational cost between the i-1th round of training and the i-2th round of training; The performance index P(V t-1 ) and the performance index P(V t ), P(V i )=λ i ×A(V i )-γ i ×C(V i ); The threshold V for the next round of training is calculated according to the following formula t+1 , V t+1 =V t +η×sign(P(V t )-P(V t-1 )), where η represents the step size parameter.

5. The training method according to any one of claims 1 to 4, characterized in that: During the training process, an attention mechanism module is introduced at the end of at least part of the convolution blocks of the initial defect classification model, and in the step of introducing residual connections in at least part of the convolution blocks, residual connections are introduced in the last N convolution blocks, where N is a positive integer greater than or equal to 3.

6. A method for classifying defects in semiconductor chips, characterized in that: The steps include: Acquire a target semiconductor chip image to be classified; The target semiconductor chip image is input into a target defect classification model constructed using the training method described in any one of claims 1 to 5 to obtain a defect classification result corresponding to the target semiconductor chip image.

7. The defect classification method according to claim 6, characterized in that: The step of obtaining the target semiconductor chip image to be classified comprises the following steps: Acquire an original semiconductor chip image; Selecting windows of different scales for each pixel in the original semiconductor chip image; Perform discrete Fourier transform on each window of selected scale, convert the spatial domain data in the window into frequency domain data, and obtain the corresponding frequency domain feature map; The frequency domain feature maps obtained at all scales are fused to generate a comprehensive frequency domain feature map; The comprehensive frequency domain feature map is combined with the spatial domain feature map of the original image to generate a multi-scale joint feature map as a target semiconductor chip image to be classified.

8. The defect classification method according to claim 7, characterized in that: The step of performing a discrete Fourier transform on each window of a selected scale, converting the spatial domain data in the window into frequency domain data, and obtaining a corresponding frequency domain feature map, further includes, after converting the spatial domain data in the window into frequency domain data, applying a frequency filter to the frequency domain data to extract high-frequency information and low-frequency information respectively.

9. The defect classification method according to any one of claims 6 to 8, characterized in that: Inputting the target semiconductor chip image into a target defect classification model constructed by the training method according to any one of claims 1 to 5 to obtain a defect classification result corresponding to the target semiconductor chip image comprises the following steps: When the target semiconductor chip image includes a target defect, the defect classification result is a defect type corresponding to the target semiconductor chip image, and the target defect is a defect included in a training set; When the target semiconductor chip image contains an unknown defect, the defect classification result is an unclassified result, a low confidence classification result or an abnormal detection result, and the unknown defect is a defect not included in the training set; When the target semiconductor chip image contains no defects, the defect classification result is no defect.

10. A semiconductor chip defect classification system, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute steps corresponding to the defect classification method according to any one of claims 6 to 9.

11. A readable storage medium, characterized in that: The readable storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement steps corresponding to the defect classification method according to any one of claims 6 to 9.

Images