A chip visual design defect detection method and system based on deep learning

Abstract

The application relates to the technical field of image data processing, and provides a chip visual design defect detection method and system based on deep learning, wherein the chip visual design defect detection method based on deep learning comprises the following steps: constructing a special search space and a hardware performance evaluation model; implementing a differentiable structure search optimization network structure; implementing mixed precision quantization optimization model storage and calculation efficiency; performing multi-objective joint optimization to simultaneously optimize the network structure and bit width configuration; and applying a knowledge distillation technology to maintain high detection precision. Through joint optimization of the network structure and precision configuration, the detection speed is significantly improved, the real-time detection requirement of a chip production line is met, the model size is greatly reduced, the storage space and transmission bandwidth are greatly saved, the demand for hardware resources is reduced, the hardware cost is reduced, lower configuration equipment can also be used for efficient detection, and the universality and economy of the method are improved.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method and system for detecting chip visualization design defects based on deep learning. Background Technology

[0002] With the continuous development of integrated circuit manufacturing processes, the integration and complexity of chips are constantly increasing, leading to higher requirements for the accuracy and efficiency of chip appearance defect detection. Traditional manual inspection methods can no longer meet the needs of modern chip production lines, and automated defect detection technology based on machine vision is gradually becoming the mainstream solution. Deep learning, as a cutting-edge technology in the field of machine vision, performs exceptionally well in image recognition and object detection, and is therefore widely used in chip defect detection.

[0003] Currently, deep learning-based chip defect detection methods mainly employ models such as Convolutional Neural Networks (CNNs) and Region Convolutional Neural Networks (R-CNNs) for defect detection and classification. However, existing technologies suffer from the following problems:

[0004] Existing deep learning models have high computational complexity and a large number of parameters, making them difficult to deploy efficiently in real-time inspection systems on production lines. This results in a long inspection process and affects production efficiency.

[0005] Traditional deep learning models consume a lot of storage space and computing resources, and have high requirements for the hardware resources of detection equipment, which increases deployment costs.

[0006] Existing methods do not fully consider the diversity of hardware platforms and resource constraints in production environments, resulting in limited model applicability and difficulty in achieving consistent performance across different hardware platforms;

[0007] Existing models are inefficient on certain hardware and cannot achieve optimal performance, resulting in a waste of hardware resources.

[0008] In existing technologies, there are deep learning-based chip defect detection methods that use an improved U-Net network structure for defect segmentation. While these methods offer high detection accuracy, their high computational complexity makes them difficult to implement in real-time on resource-constrained hardware. There are also convolutional neural network-based chip surface defect detection methods. Although these methods simplify the network structure, they do not consider model quantization and hardware adaptability issues, leading to significant differences in deployment performance across different hardware platforms.

[0009] Therefore, there is an urgent need for a chip defect detection method that can significantly improve detection speed, reduce model size, and adapt to multiple hardware platforms while ensuring high accuracy, in order to meet the real-time detection needs of chip production lines. Summary of the Invention

[0010] This invention provides a chip visualization design defect detection method and system based on deep learning. This method optimizes the neural network structure and accuracy configuration to achieve efficient deployment of the detection model on a heterogeneous computing platform, solving the technical problems of high computational complexity, large model size and poor hardware adaptability in the prior art.

[0011] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0012] A deep learning-based method for detecting defects in chip visualization design includes the following steps:

[0013] A dedicated search space and hardware performance evaluation model are constructed. The search space includes neural network structure components suitable for chip defect feature extraction, and the hardware performance evaluation model is used to predict the execution time of the neural network on the target hardware.

[0014] Implement differentiable structure search to optimize the neural network structure for chip defect detection;

[0015] Implement mixed-precision quantization to adaptively allocate optimal bit widths to different layers of the neural network, thereby optimizing model storage and computation efficiency;

[0016] Perform multi-objective joint optimization, simultaneously optimizing the neural network structure and quantization bit width configuration;

[0017] By applying knowledge distillation technology, knowledge from a large teacher model is transferred to an optimized small student model, thereby improving the accuracy of chip defect detection.

[0018] Furthermore, the steps of constructing the dedicated search space and hardware performance evaluation model include:

[0019] Based on the characteristics of chip defect images, a neural network search space suitable for defect feature extraction is constructed, including convolutional layers, pooling layers, and attention modules.

[0020] Collect performance data of the target hardware platform to form a hardware characteristic dataset;

[0021] A hardware execution time prediction model is trained. This model is implemented using a multilayer perceptron structure, which includes an input layer, a feature extraction layer, a fusion layer, and an output layer. It receives neural network operation parameters and hardware specifications as inputs and is trained by minimizing the mean square error between the predicted time and the actual time.

[0022] Furthermore, the step of performing the search for differentiable structures includes:

[0023] Based on the search space, a differentiable network structure representation is established, and the network structure selection is represented as a continuous variable.

[0024] An alternating optimization algorithm is adopted, which fixes the structure parameters on the training set and optimizes the weight parameters on the validation set, and uses a second-order approximation method to accelerate the structure optimization.

[0025] During training, a latency-aware regularization term is added based on feedback from the hardware performance evaluation model.

[0026] Furthermore, the step of implementing mixed precision quantization includes:

[0027] Establish a quantization representation method to convert full-precision parameters into a low-bit-width representation;

[0028] Different bit widths of quantization are sequentially applied to each layer in the neural network to calculate the precision loss, generate a sensitivity map, and record the precision changes of each layer under different quantization bit widths.

[0029] An iterative greedy algorithm is adopted to prioritize applying lower bit widths to layers with low sensitivity, while meeting the overall accuracy requirements.

[0030] To achieve hardware-aware quantization, select the most suitable quantization bit width and calculation mode based on the characteristics of the target hardware platform;

[0031] Construct a quantization-aware training loop, which includes applying quantization configuration, quantizing weights and activations in forward propagation, processing gradients using a pass-through estimator in backpropagation, and updating full-precision weights.

[0032] Furthermore, the step of performing multi-objective joint optimization includes:

[0033] Establish a joint optimization objective function that considers detection accuracy, inference latency, and model size simultaneously;

[0034] Construct a joint search space, which includes two dimensions: network structure and quantization bit width;

[0035] The joint search space is optimized using a differential evolution algorithm, which includes population initialization, mutation, crossover, and selection operations. The candidate solutions are sorted using a fast non-dominated sorting algorithm, and the crowding distance mechanism is used to maintain the diversity of solutions.

[0036] Train a corresponding neural network for each candidate solution and evaluate its performance;

[0037] Choose the most suitable solution from the Pareto frontier based on specific application requirements.

[0038] Furthermore, the steps of applying the knowledge distillation technique include:

[0039] Construct a multi-level knowledge distillation framework and define teacher and student models;

[0040] Design a multi-task distillation loss function for chip defect detection, including hard label loss, soft label loss, feature matching loss, and boundary-aware loss;

[0041] Implement an attention-guided knowledge distillation method to generate an attention map of the teacher model and construct an attention-guided loss.

[0042] Implement a progressive knowledge distillation training process, including an initialization phase, a feature matching phase, a classification optimization phase, and a quantization fine-tuning phase.

[0043] Furthermore, the multi-task distillation loss function further includes:

[0044] Hard label loss, which uses the real labels to calculate the cross-entropy loss;

[0045] Soft label loss uses the output of the teacher model as soft labels to guide the student model in learning the category probability distribution;

[0046] Feature matching loss aligns the feature representations of the teacher and student models in the intermediate layer;

[0047] Boundary-aware loss focuses on feature learning in the boundary regions of chip defects.

[0048] Furthermore, the attention-guided knowledge distillation method includes:

[0049] Generate an attention map of the teacher model to reveal the key areas the model focuses on;

[0050] Construct an attention-guided loss to guide the student model to focus on the same key regions as the teacher model;

[0051] Selective distillation strategies are designed for different types of defects, including focusing on shallow features for edge defects, focusing on mid-level features for texture defects, and focusing on deep features for structural defects.

[0052] Furthermore, the progressive knowledge distillation training process includes:

[0053] During the initialization phase, a pre-trained teacher model is used to randomly initialize the student model;

[0054] During the feature matching stage, the classification head of the student model is frozen, and only the feature extraction part is optimized;

[0055] In the classification optimization phase, the classification head of the student model is unfrozen, and training is performed using the full distillation loss function;

[0056] In the quantization fine-tuning stage, quantization perception training is applied to further reduce the accuracy loss caused by quantization.

[0057] The second invention provides a deep learning-based chip visualization design defect detection system for executing the aforementioned deep learning-based chip visualization design defect detection method. The system includes:

[0058] The search space building module is used to build a dedicated search space and hardware performance evaluation model.

[0059] The structure search module is used to perform differentiable structure search and optimize the neural network structure for chip defect detection.

[0060] The precision quantization module is used to implement mixed precision quantization, adaptively allocating the optimal bit width to different layers of the neural network;

[0061] The joint optimization module is used to perform multi-objective joint optimization, while optimizing the network structure and quantization bit width configuration.

[0062] The knowledge distillation module is used to apply knowledge distillation technology to transfer knowledge from a large teacher model to an optimized small student model.

[0063] The beneficial effects of this invention are as follows: By jointly optimizing the network structure and accuracy configuration, the detection speed is significantly improved, meeting the real-time detection needs of chip production lines; the model size is greatly reduced, saving storage space and transmission bandwidth, reducing the demand for hardware resources, and ensuring the reliability and stability of chip defect detection; the model can be efficiently deployed on various computing platforms (including edge devices and heterogeneous acceleration platforms), adapting to the real-time detection needs of different production scenarios; and it reduces hardware costs, enabling efficient detection even with lower-configuration equipment, thus improving the universality and economy of the method. Attached Figure Description

[0064] Figure 1 This is a flowchart of the chip visualization design defect detection method based on deep learning according to the present invention. Detailed Implementation

[0065] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0066] This embodiment provides a chip visualization design defect detection method based on deep learning, such as... Figure 1 As shown, it includes the following steps:

[0067] Step 1: Construct a dedicated search space and a hardware performance evaluation model. The search space includes neural network structure components suitable for chip defect feature extraction, and the hardware performance evaluation model is used to predict the execution time of the neural network on the target hardware.

[0068] Specifically, it includes:

[0069] Based on the characteristics of chip defect images, a neural network search space S is constructed. This space includes network components suitable for defect feature extraction, such as convolutional layers, pooling layers, and attention modules. The parameter range of each component is pre-analyzed and set, as follows:

[0070]

[0071] Among them, O i This represents the i-th type of network operation. For the set of all optional operations, P i Indicates network operation type O i The set of parameters, This indicates the range of values ​​for the parameters. For example, for convolution operations, parameters include kernel size, number of channels, stride, etc.

[0072] Collect performance data of the target hardware platform, including the execution efficiency of different computing units such as CPU, GPU, and dedicated chips for various neural network operations, to form a hardware characteristic dataset D. hw :

[0073] D hw ={(op j hw k , t j，k |j∈[1,J],k∈[1,K]};

[0074] Among them, op j Let hw represent the j-th neural network operation. k Let t represent the k-th type of hardware device. j，k means op j In hardware device hw k The dataset contains execution time information for all operations on all target hardware platforms, where J represents the total number of network operations, K represents the total number of hardware devices, ∈ [1,J] represents the operation index range from 1 to J, and k∈ [1,K] represents the hardware device index range from 1 to K. This dataset provides foundational data support for subsequent optimizations.

[0075] Train a hardware execution time prediction model T. The input of this model is the network operation parameters and hardware specifications, and the output is the predicted execution time.

[0076] T(op,hw)=f φ (opparams ,hw specs );

[0077] Among them, f φ Let represent the prediction function of the learnable parameter set φ of the model; op represent the neural network operation; hw represent the hardware platform; T(op,hw) represent the prediction execution time of the neural network operation op on the hardware platform hw; op params The set of parameters representing a neural network operation includes operation type, input / output size, number of channels, kernel size, etc.; hw specs This indicates the hardware platform specifications, including processor type, memory size, and number of computing units. The hardware execution time prediction model is implemented using a specific multilayer perceptron architecture, which includes:

[0078] Input layer: Receives operation parameter vectors (including operation type, input / output size, number of channels, convolution kernel size, etc.) and hardware specification vectors (including processor type, memory size, number of computing units, etc.);

[0079] Feature extraction layer: Contains 3 fully connected layers, each with multiple neurons, using the ReLU activation function to extract the correlation patterns between operations and hardware characteristics;

[0080] Fusion layer: After concatenating operational features and hardware features, feature fusion is performed through two fully connected layers;

[0081] Output layer: a single neuron that outputs the predicted execution time.

[0082] The prediction model is trained by minimizing the mean square error between the predicted time and the actual time.

[0083]

[0084] Where, min φ This indicates optimizing the set of learnable parameters φ of the model to minimize the objective function; ∑ represents summing over all samples; (op, hw, t) ∈ D hw This indicates that the hardware characteristic dataset D hw The sample extracted from the sample contains the neural network operation op, the hardware platform hw, and the actual execution time t; T(op, hw) represents the execution time of the operation op predicted by the model on the hardware platform hw; (T(op, hw)-t) 2 Mean squared error is the square of the difference between the predicted time and the actual time.

[0085] In practical applications on chip production lines, this hardware execution time prediction model can be used to evaluate the performance of different defect detection network architectures on edge computing devices within the production line. For example, when a new type of image acquisition and processing equipment is introduced into the production line, the model can predict the execution efficiency of the existing defect detection network without requiring time-consuming actual deployment testing, thus quickly determining whether the model needs to be adjusted and optimized.

[0086] This step establishes a neural network search space suitable for chip defect detection tasks and a reliable hardware performance evaluation model, providing a foundation for subsequent network optimization.

[0087] Step 2: Implement differentiable structure search to optimize the network structure and optimize the network structure for chip defect detection;

[0088] Specifically, it includes:

[0089] Based on the search space constructed in step 1, a differentiable network structure representation α is established, and the network structure selection is represented as a continuous variable:

[0090]

[0091] Where E represents the set of edges in the network, and (i, j) represents the connection from node i to node j. This indicates the weight of a single optional operation O on this connection.

[0092] Construct a network representation such that the choice of each neural network operation is determined by a set of weights α. (i，j) The discrete choices are determined and converted into continuous variables using the softmax function:

[0093]

[0094] in, Let O(x) represent the output of the hybrid operation, x represent the input feature, o(x) represent the result of operation o on the input x, exp represent the exponential function, and O′ represent another network operation. This represents the probability weight of operation O calculated using the softmax function.

[0095] An alternating optimization algorithm is used to iteratively optimize the weight parameter w and the structure parameter α:

[0096] With the structural parameter α fixed, in the training set D train Optimize the weight parameter W:

[0097]

[0098] Among them, W * argmin represents the optimized weight parameters. wThis indicates the search for weight parameters that minimize the objective function. This represents the loss function on the training set;

[0099] Fixed optimized weight parameter W * In the verification set D val Optimize the structural parameter α:

[0100]

[0101] Where, α * argmin represents the optimized structure parameters. α This indicates the search for structure parameters that minimize the objective function. This represents the loss function on the validation set, which includes classification loss and regularization term:

[0102]

[0103] in, Represents the total loss function. R(α) is the cross-entropy loss, R(α) is the sparsity regularization term, and λ is the tradeoff coefficient.

[0104] Specific implementation methods for differentiable structure search:

[0105] Accelerate the structure optimization process using a second-order approximation method:

[0106]

[0107] in, This represents the gradient with respect to the structural parameter α. This represents the gradient of the validation set loss function with respect to the structural parameter α. This represents the gradient with respect to the weight parameter W. This represents the gradient of the training set loss function with respect to the weight parameter w. This represents the result of updating the weight parameter w one step along the training gradient direction, and ξ is the approximate learning rate.

[0108] Employing specific network unit designs, including:

[0109] Edge detection unit: composed of 3×3 and 5×5 depth separable convolutions, suitable for detecting chip edge defects;

[0110] Texture analysis unit: Composed of 3×3 convolution and self-attention modules, suitable for fine-grained texture defect detection;

[0111] Multi-scale feature fusion unit: Combines feature maps of different scales and adjusts feature channels through 1×1 convolution.

[0112] During training, a latency-aware regularization term is added based on feedback from the hardware execution time prediction model, enabling the network structure to reduce inference latency while maintaining detection accuracy.

[0113] R delay (α)=T pred (N α ,hw target )

[0114] Among them, R delay (α) represents the regularization term for delayed perception, T pred It is the execution time prediction model in step 1, N α This is the network under the current structural parameters, hw target This refers to the target hardware platform. The final structural optimization goal is:

[0115]

[0116] Where β is the weighting coefficient for delay regularization.

[0117] In practical applications, this differentiable structure search algorithm is used for chip surface scratch detection. By designing a specific search space, it can discover network structures particularly suitable for recognizing linear scratch features. For example, for fine scratches on the surface of a certain type of semiconductor chip, the algorithm automatically selects a structure containing cascaded depthwise separable convolutions and a direction-aware attention module. Compared with manually designed networks, this not only improves detection accuracy but also enhances inference speed. This optimized structure can effectively capture the directional and edge characteristics of fine scratches, significantly improving detection performance.

[0118] This step automatically searches for and determines an optimized network structure tailored to the characteristics of chip defect detection, improving the ability and efficiency of feature extraction.

[0119] Step 3: Implement mixed-precision quantization to adaptively allocate the optimal bit width to different layers of the neural network, thereby optimizing model storage and computation efficiency;

[0120] This step employs mixed-precision quantization to adaptively allocate optimal bit widths to different layers of the network, maintaining detection accuracy while reducing model size and computational complexity. Specifically, it includes:

[0121] Establish a quantization representation method to convert full-precision parameters into a low-bit-width representation:

[0122] w q =Int(w f ·s);

[0123] Among them, w f For full-precision weights, wq is the quantized integer weight, s is the quantization scaling factor, and Int is the rounding operation.

[0124] Construct a set B of quantization configurations with different precisions, containing multiple options with different bit widths:

[0125] B = {b1, b2, ..., b} n};

[0126] Where b1, b2, ..., b n These represent 1, 2, ..., n quantization bit widths, where n represents the total number of quantization bit widths.

[0127] Establish a quantization sensitivity analysis method to assess the sensitivity of each layer of the network to quantization:

[0128] For each layer l in the network, apply different bit-width quantizations b∈B sequentially, and calculate the precision loss on the validation set:

[0129] ΔAcc(l,b)=Acc(N)-Acc(N 1,b );

[0130] Where ΔAcc(l, b) represents the accuracy loss of layer l at bit width b, and Acc(N) represents the original network accuracy. l，b ) represents the network precision after quantizing layer l to a bit width b.

[0131] Generate a sensitivity map S and record the accuracy changes of each layer under different quantization bit widths:

[0132] S={s l,b =ΔAcc(l,b)|l∈[1,L],b∈B};

[0133] Where L is the number of network layers, s l,b This indicates the precision loss when layer l uses bit width b.

[0134] Specific methods for achieving mixed-precision quantization based on sensitivity analysis:

[0135] An iterative greedy algorithm is used to prioritize applying lower bit widths to layers with lower sensitivity, while still meeting overall accuracy requirements.

[0136] Initial state: All layers use the highest bit width;

[0137] Iterative process: Each time, select the layer-bit width combination (l, b) with the minimum precision loss, and quantize layer l into a lower bit width b;

[0138] Termination condition: The target model size or accuracy loss exceeds a preset threshold;

[0139] Construct a quantization-aware training loop, and fine-tune the network after the quantization configuration is determined:

[0140] Initialization: Apply the selected mixed-precision quantization configuration;

[0141] Forward propagation: Quantize weights → Perform calculations → Quantize activation values;

[0142] Backpropagation: Gradients are processed using a pass-through estimator (STE);

[0143] Update: Updated full-precision weights;

[0144] Specific optimizations for chip defect detection scenarios:

[0145] Use a higher bit width for the first and last channels of the convolutional layer to preserve the accuracy of the input and output features;

[0146] Use a higher bit width for feature extraction layers that detect key regions (such as edges and corners);

[0147] A lower bit width is used for the texture feature representation layer because texture features are not sensitive to quantization;

[0148] To achieve hardware-aware quantization, select the most suitable quantization bit width and calculation mode based on the characteristics of the target hardware platform:

[0149] Using the hardware execution time prediction model from step 1, evaluate the actual performance of different quantization configurations on the target hardware:

[0150] T quant (N,hw,Q)=f θ (N arch ,hw specs Q config );

[0151] Among them, T quant The execution time of the quantized model is represented by N, where N represents the neural network model, Q represents the quantization configuration scheme, and f is the quantization time. θ This represents the hardware execution time prediction function, θ represents the prediction model parameters, and N... arch It's network architecture, Q config It is a quantitative allocation.

[0152] Based on hardware characteristics and accuracy requirements, select the optimal quantization configuration:

[0153] Q * =argmin Q T quant (N,hw,Q),stΔAcc(NQ)≤∈;

[0154] Among them, Q *argmin represents the optimal quantization configuration scheme. Q T represents the quantized allocation scheme that minimizes the objective function. quant (N, hw, Q) represents the model execution time under the neural network model N, hardware platform hw, and quantization configuration scheme Q, where ΔAcc(N) is the execution time of the model. Q ) represents the precision loss after quantization, ΔAcc represents the precision loss, ∈ is the acceptable precision loss threshold, and N Q This represents the network model after applying the quantization configuration scheme Q, where st indicates that the condition (subjectto) is met.

[0155] In a practical application example, this hybrid precision quantization method was applied to chip packaging defect identification on an embedded vision inspection device. It successfully compressed the original multi-bit floating-point model to just a few bits per parameter, reducing the model size to some extent, while slightly decreasing accuracy. In this application, it was found that the input layer and the feature extraction layer related to defect edge detection were more sensitive to quantization; therefore, these layers were allocated higher bit widths, while the intermediate feature representation layer used a lower bit width. This optimized configuration enabled the detection model to run in real-time on resource-constrained industrial edge computing devices, processing high-resolution chip images per second.

[0156] This step significantly reduced the model size and computational complexity while maintaining high accuracy in chip defect detection, laying the foundation for subsequent joint optimization.

[0157] Step 4: Perform multi-objective joint optimization to simultaneously optimize network structure and bit width configuration;

[0158] This step achieves joint optimization of network structure and quantization bit width, simultaneously seeking the optimal solution across multiple dimensions such as accuracy, latency, and model size through a global multi-objective optimization algorithm. Specifically, it includes:

[0159] Establish a joint optimization objective function that considers detection accuracy, inference latency, and model size:

[0160]

[0161] Where Q represents the quantitative configuration scheme, Let T represent the detection loss function, T represent the inference latency on the hardware platform hw, and M represent the model size.

[0162] Constructing the joint search space S joint It includes two dimensions: network structure and quantization bit width.

[0163] S joint =S arch ×S quant ;

[0164] Among them, S arch S represents the network structure search space. quant This indicates the quantitative configuration search space.

[0165] The differential evolution algorithm is used to optimize the joint search space of bit width and structure:

[0166] Population initialization: Generate multiple candidate solutions, each containing network structure parameters α and quantization bit width configuration Q:

[0167] P={(α i Q i |i = 1, 2, ..., NP};

[0168] Where NP represents the population size, i.e., the number of candidate solutions; α i The network structure parameters representing the i-th candidate solution determine the network topology and connection method; Q i The quantization bit width configuration of the i-th candidate solution determines the accuracy of each layer of the network; P represents the entire population, which contains the set of all candidate solutions.

[0169] Mutation operation: For each individual (α) i Q i Select three different individuals r1, r2, r3, and generate mutation vectors:

[0170]

[0171] Among them, v i This represents the generated mutation vector. and These represent three distinct individuals randomly selected from the population. These represent the network structure parameters of the r1-th, r2-th, and r3-th candidate solutions, respectively. These represent the quantization bit width configurations for the r1, r2, and r3 candidate solutions, respectively. F is a scaling factor that controls the amplification of the difference vector, typically ranging from [0.4, 1.0].

[0172] Crossover operation: For the current individual (α) i Q i ) and mutation vector v i Perform crossover to generate the test vector u i :

[0173]

[0174] in, Represents the test vector u i The j-th component, Represents the mutation vector v iThe j-th component, (α) i Q j ) j This represents the j-th component of the current individual, where j represents the index of the decision variable, and rand. j Represents a random number within the interval [0,1], where CR is the crossover probability (controlling the proportion of information inherited from the mutation vector), and j rand The index is randomly selected (ensuring that at least one variable comes from the mutation vector), and "otherwise" indicates the meaning of "otherwise".

[0175] Select operation: Evaluate the trial vector u i If the performance of the current individual is superior, then replace it.

[0176]

[0177] Among them, (α) i Q i ) t+1 U represents the individual in the (t+1)th generation. i Denotes the test vector, (α) i Q i ) t F(u) represents the current individual in generation t. i F((α) represents the objective function value vector of the experimental vector. i Q i ) t ) represents the objective function value vector of the current individual, and ≤ represents the Pareto dominance relationship in multi-objective optimization (meaning that a solution is not inferior to another solution in all objectives, and is superior to another solution in at least one objective).

[0178] The specific optimization process for implementing the differential evolution algorithm:

[0179] Evolutionary parameter configuration:

[0180] Population size: NP = 50;

[0181] Maximum number of generations: MaxGen = 100;

[0182] Scaling factor: F = 0.5;

[0183] Crossover probability: CR = 0.7;

[0184] Multi-objective assessment and Pareto frontier selection:

[0185] The candidate solutions are sorted using the Fast Non-Dominated Sort (NSGA-II) algorithm.

[0186] A crowding distance mechanism is used to maintain the diversity of solutions;

[0187] Preserve the set of non-dominated solutions on the Pareto front;

[0188] Optimization strategies for chip defect detection scenarios:

[0189] Different mutation and crossover strategies are used for the network structure parameter α and the quantization bit width configuration Q;

[0190] The structural parameters are encoded using continuous real-value encoding, while the bit width configuration is encoded using discrete integer encoding.

[0191] After mutation and crossover operations, the structural parameters are normalized and the bit width configuration is rounded.

[0192] To achieve candidate solution evaluation and final model selection:

[0193] For each candidate solution (α, Q), train the corresponding network and evaluate its performance:

[0194] Detection accuracy: Evaluate detection accuracy, recall, and precision on the test set;

[0195] Latency performance: Estimate inference latency using the prediction model from step 1;

[0196] Model dimensions: number of computational parameters and storage size;

[0197] Choose the most suitable solution from the Pareto frontier based on specific application requirements:

[0198] Performance-priority scenarios: Select the model with the highest accuracy that satisfies latency and size constraints;

[0199] Real-time priority scenarios: Select the model with the lowest latency that meets the accuracy requirements;

[0200] In resource-constrained scenarios: Select the model with the smallest size that satisfies both accuracy and latency constraints;

[0201] In a practical application example, when this multi-objective joint optimization algorithm was applied to a defect detection system on an automotive chip production line, three model versions with different trade-offs were obtained by simultaneously optimizing the network structure and quantization configuration:

[0202] High-precision version: The detection accuracy is improved, making it suitable for high-end chip testing with strict quality control.

[0203] High-performance version: Detection accuracy is slightly reduced, suitable for real-time detection on production lines;

[0204] Ultra-lightweight version: Maintains stable detection accuracy and is suitable for embedded detection devices.

[0205] Compared to optimizing the network structure or quantization bit width individually, the joint optimization method improves detection performance while reducing inference latency. Particularly on resource-constrained edge devices, this method discovers the synergistic effect between structure and quantization, enabling the detection system to meet real-time requirements while maintaining high detection accuracy.

[0206] This step achieves joint optimization of network structure and bit width configuration, resulting in a series of Pareto optimal solutions with different trade-offs in terms of accuracy, latency, and model size, providing a foundation for selecting the most suitable model for specific scenarios.

[0207] Step 5: Apply knowledge distillation technology to transfer the knowledge from the large teacher model to the optimized small student model, thereby improving the accuracy of chip defect detection;

[0208] This step involves constructing a knowledge distillation method for chip defect detection to further improve the model's detection accuracy.

[0209] Specifically, it includes:

[0210] Construct a multi-layered knowledge distillation framework and define teacher and student models:

[0211] Teacher model T: Uses large, high-precision networks, such as ResNet-101 or EfficientNet-B7 which use multi-bit floating-point numbers. It has powerful feature extraction capabilities but high computational complexity.

[0212] Student model S: The lightweight model obtained by optimization in step 4 has a compact structure and is quantized to a low bit width, but there may be a loss of accuracy.

[0213] Establish multi-level knowledge transfer channels:

[0214]

[0215] in, and L represents the feature maps of the teacher model and the student model at layer l, respectively. KD This represents the set of layers that require knowledge distillation.

[0216] Design a multi-task distillation loss function for chip defect detection:

[0217]

[0218] in: The total loss function; λ1, λ2, λ3, and λ4 represent the weighting coefficients of each loss term, used to balance the contributions of different loss terms; This is the hard label loss; Indicates soft label loss; For feature matching loss; This represents boundary-aware loss;

[0219] Hard label loss Calculate the cross-entropy loss using the real labels to ensure basic classification ability:

[0220]

[0221] Where S(x) represents the student model's prediction of input x, y represents the true label, and CE is the cross-entropy function.

[0222] Soft tag loss The output of the teacher model is used as a soft label to guide the student model in learning the category probability distribution:

[0223]

[0224] Where KL represents KL divergence, τ is a temperature parameter that controls the smoothness of the soft label, and T(x) represents the teacher model's predicted output for input x.

[0225] Feature matching loss Align the feature representations of the teacher and student models in the intermediate layer:

[0226]

[0227] Where φ1 is the feature transformation function, used to adjust the feature dimensions of the teacher model to match the student model. The feature map of the teacher model at layer l. L is the feature map of the student model at layer l. KD For the set of layers that require knowledge distillation, This represents the square of the L2 norm.

[0228] Boundary-aware loss Focus on feature learning of chip defect boundary regions:

[0229]

[0230] Among them, M b This is a boundary region mask, where ⊙ represents element-wise multiplication, focusing on the feature differences at the defect boundary.

[0231] Implementing an attention-guided knowledge distillation method:

[0232] Attention map of the generated teacher model Revealing the key areas the model focuses on:

[0233]

[0234] Where C is the number of feature channels. This represents the feature map of the c-th channel in the l-th layer. This is the attention map of the teacher model in layer 1.

[0235] Construct an attention-guided loss to guide the student model to focus on the same key regions:

[0236]

[0237] in, To guide attention loss, The attention map of the student model in layer 1;

[0238] Selective distillation strategy based on defect type awareness:

[0239] For edge-type defects, the focus should be on the distillation of shallow features to enhance edge detection capabilities;

[0240] For texture-related defects, the focus should be on distilling mid-level features to improve texture recognition capabilities;

[0241] For structural defects, the focus is on distilling deep features to enhance semantic understanding capabilities;

[0242] Implement a progressive knowledge distillation training process:

[0243] Initialization phase: Use the pre-trained teacher model to randomly initialize the student model;

[0244] Feature matching stage: Freeze the classification head of the student model, optimize only the feature extraction part, and train using feature matching loss:

[0245]

[0246] Classification optimization phase: Unfreeze the student model's classification head and train using the full distillation loss function:

[0247]

[0248] Quantization fine-tuning stage: After distillation, quantization-aware training is applied to further reduce the accuracy loss caused by quantization.

[0249]

[0250] Among them, S Q Let Q(S) represent the quantized student model, where Q(S) represents the quantization operation on the student model, and γ is the weighting coefficient.

[0251] Establish a method for evaluating model performance and quantitatively compare model performance before and after optimization:

[0252] Detection accuracy metrics: computational precision, recall, F1 score, and mean precision (mAP);

[0253] Efficiency metrics: Measure inference time, model size, and memory usage on actual hardware;

[0254] Overall performance metrics: Accuracy-Delay Product (PLP) and Accuracy-Size Product (PSP):

[0255] PLP=Accuracy×(1 / Latency);

[0256] PSP = Accuracy × (1 / Size);

[0257] Where Accuracy represents the model's detection accuracy, Latency is the model's inference latency, Size represents the model's storage size, (1 / Latency) is the reciprocal of the latency, representing the model's speed, and (1 / Size) is the reciprocal of the size, representing the model's storage efficiency.

[0258] In practical applications, this knowledge distillation method, when applied to etching defect detection, significantly improves detection performance by transferring knowledge from a ResNet-101 teacher model with stable parameters to a lightweight MobileNet-V3 student model with only stable parameters. On a dataset containing multiple types of chip etching defects, compared to a model of the same size without knowledge distillation, the distilled student model shows a certain improvement in average detection accuracy, particularly for difficult-to-identify subtle bridging defects and incomplete etching defects.

[0259] This knowledge distillation method is particularly suitable for situations with imbalanced defect samples. By extracting knowledge about rare defect categories from the teacher model, it significantly improves the student model's ability to detect these defects. In a real-world deployment on a chip production line, the lightweight model using this method was able to process high-resolution chip images in real time on edge computing devices, achieving stable detection speed while maintaining accuracy comparable to large models using a central server.

[0260] This step effectively transfers knowledge from a large, high-precision model to an optimized, lightweight model, maintaining high detection accuracy while meeting deployment requirements on resource-constrained hardware.

[0261] 3. Technical effects of this embodiment:

[0262] The chip visualization design defect detection method based on deep learning provided in this embodiment achieves the following technical effects by optimizing the neural network structure and accuracy configuration:

[0263] The detection speed is significantly improved, which is a certain improvement compared to traditional deep learning models, meeting the needs of real-time detection on chip production lines;

[0264] The model size has been significantly reduced, the compression rate has been stabilized, greatly saving storage space and transmission bandwidth, and reducing the demand for hardware resources;

[0265] The detection accuracy rate remains at a stable high level, ensuring the reliability and stability of chip defect detection;

[0266] The model can be efficiently deployed on various computing platforms (including edge devices and heterogeneous acceleration platforms) to meet the real-time detection needs of different production scenarios;

[0267] This reduces hardware costs, enabling efficient detection even with lower-configuration equipment, and improves the method's versatility and cost-effectiveness.

[0268] This implementation method successfully solves the problem of low model deployment efficiency in chip visualization design defect detection by jointly optimizing network structure and bit width configuration and combining knowledge distillation technology, providing an efficient and feasible technical solution for real-time defect detection in chip manufacturing process.

[0269] This section details the application process and effects of the present invention in actual chip production lines through specific application examples.

[0270] A chip production line of an integrated circuit manufacturing company needs to perform appearance defect inspection on the produced large-scale integrated circuit chips. The main objective is to identify the following types of defects:

[0271] Surface scratches: including fine linear scratches, deep scratches, etc.;

[0272] Packaging defects include pin deformation, missing solder joints, and package cracks.

[0273] Pattern defects include incomplete photolithography, missing patterns, and bridging.

[0274] Material abnormalities: including problems such as oxidation, corrosion, and contamination.

[0275] The inspection system faces the following challenges: high production line speed, requiring the processing of multiple chip images per second; limited resources, with inspection equipment configured with Intel Core i5 processors, 8GB of memory, and no dedicated GPU; multi-scenario deployment requirements, needing to perform real-time inspections on the production line as well as spot checks on portable devices; diverse defect types, with new defects constantly emerging, requiring the system to have good generalization capabilities.

[0276] Based on the method of this invention, we deployed an optimized defect detection system on the chip production line. The specific application process is as follows:

[0277] Data collection: Collect multiple high-resolution chip images from the production line, including various defective and normal samples;

[0278] Data annotation: Professional quality inspectors annotate the defective areas in the images, including information such as defect type, location, and size;

[0279] Data augmentation: Expand the dataset using methods such as rotation, flipping, and brightness adjustment, with a particular focus on augmenting rare defect types;

[0280] Data partitioning: The dataset is divided into training, validation and test sets according to a certain ratio to ensure that the distribution of various defects is balanced in each subset.

[0281] The initial network structure was chosen to be based on EfficientDet-D0, which offers a good balance between accuracy and efficiency in object detection tasks.

[0282] Model training: Initial model training is performed using standard training procedures, including common optimization techniques such as learning rate scheduling and weight decay;

[0283] Performance Evaluation: The performance of the base model was evaluated on the test set, and the results are as follows:

[0284] Average detection accuracy (mAP): Remains stable;

[0285] Processing time per image: 420ms;

[0286] Model size: 32.6MB;

[0287] Memory usage: 1.2GB.

[0288] Step 1: Construct a dedicated search space and a hardware performance evaluation model. The search space includes neural network structure components suitable for chip defect feature extraction, and the hardware performance evaluation model is used to predict the execution time of the neural network on the target hardware.

[0289] Based on the characteristics of chip defects, a search space containing various convolution operations, various pooling operations, and various attention modules was constructed;

[0290] Test the execution time of each operation on the target Intel processor and build a hardware performance prediction model;

[0291] Test results show that the prediction model's prediction error for operation execution time is within a certain range.

[0292] Step 2: Perform differentiable structure search to optimize the neural network structure for chip defect detection;

[0293] Applying the search space from step 1, perform a structure search using multiple images;

[0294] The structure search process took a certain amount of time, but ultimately a network structure more suitable for chip defect detection was found.

[0295] The optimized structure reduces the number of parameters while maintaining detection accuracy.

[0296] Step 3: Implement mixed-precision quantization to adaptively allocate the optimal bit width to different layers of the neural network, thereby optimizing model storage and computation efficiency;

[0297] Sensitivity analysis of each layer of the network revealed that the input layer and the last three layers are most sensitive to quantization.

[0298] The input and output layers retain stable precision, while intermediate layers are allocated a certain precision based on sensitivity.

[0299] The model size is reduced to some extent after quantization.

[0300] Step 4: Perform multi-objective joint optimization, simultaneously optimizing the neural network structure and quantization bit width configuration;

[0301] By setting the population size to be stable, multiple candidate models on the Pareto frontier were finally obtained.

[0302] Several representative models were selected from the candidate models, which are suitable for high-precision scenarios, real-time detection scenarios, and portable device scenarios, respectively.

[0303] Step 5: Apply knowledge distillation technology to transfer the knowledge from the large teacher model to the optimized small student model, thereby improving the accuracy of chip defect detection;

[0304] Using ResNet101 as the teacher model, knowledge is transferred to the optimized lightweight student model;

[0305] Differentiated distillation strategies are adopted for different types of defects, with a particular emphasis on enhancing the ability to identify difficult-to-detect defects;

[0306] After distillation, the student model even surpassed the teacher model in detection accuracy for some defect types.

[0307] The performance of the three optimized models obtained in the end is shown in Table 1:

[0308] Table 1. Examples of performance data for the three final optimized models:

[0309]

[0310] Deployment scenarios and effects:

[0311] Main production line deployment:

[0312] Deploy the standard version model and connect to 12 detection sites;

[0313] Achieving a detection speed of 25 chip images per second;

[0314] The false negative rate has decreased to some extent, and the false positive rate has also decreased to some extent.

[0315] The system ran stably for 6 months without any performance degradation.

[0316] Quality inspection laboratory deployment:

[0317] Deploy a high-precision version of the model for accurate identification of difficult-to-detect defects;

[0318] Suspicious samples from the production line were subjected to secondary testing, achieving an accuracy rate of 99.1%.

[0319] To assist quality inspectors in establishing more accurate defect classification standards.

[0320] Deployment of portable testing equipment:

[0321] Deploy a lightweight version of the model onto engineers' handheld devices;

[0322] Enables rapid on-site sampling, with a single test taking only 0.5 seconds;

[0323] Provide timely feedback for production line adjustments and reduce defective product output.

[0324] Economic benefits:

[0325] The testing speed has been improved to some extent, and the testing efficiency of the production line has been increased to some extent.

[0326] The accuracy of detection has improved to some extent, but the rate of missed detection of defective products has also improved to some extent.

[0327] Reducing the workload of manual re-inspection leads to a certain reduction in labor costs.

[0328] The hardware cost of the testing equipment is reduced by 60%, eliminating the need for a high-end GPU.

[0329] Technological advancement:

[0330] For the first time, joint optimization of network structure and quantization bit width has been achieved in the field of chip defect detection;

[0331] Innovatively, defect characteristics and hardware features are incorporated into the optimization considerations;

[0332] The developed distillation method effectively solves the sample imbalance problem.

[0333] Promotional value:

[0334] This method has been successfully implemented on eight different chip production lines of the company.

[0335] It is highly adaptable and can be migrated to new chip detection with only a small number of new samples;

[0336] The method is universal and can be extended to other fields of precision electronic component testing.

[0337] To verify the superiority of the method of this invention, we conducted a comparative experiment with four mainstream deep learning optimization methods, testing the performance of the optimized models of each method on the same chip defect dataset. The comparison results are shown in Table 2:

[0338] Table 2. Examples of performance data for models optimized by different methods tested on the same chip defect dataset:

[0339] Optimization methods Detection accuracy (mAP) Processing speed (FPS) Model size (MB) Adaptable to different hardware capabilities Method of the present invention 97.6％ 31.2 3.2 high Network pruning 95.8％ 24.5 8.6 middle Knowledge distillation 96.7％ 18.3 12.1 Low Quantization compression method 94.2％ 27.1 4.8 middle NAS method 97.1％ 15.6 9.2 Low

[0340] The comparison results show that the method of the present invention achieves the fastest processing speed and the smallest model size while maintaining high detection accuracy, and has the best hardware adaptability. Its overall performance is significantly better than other methods.

[0341] This application example fully verifies the effectiveness and advancement of the method of this invention in a real chip production line. Through the integrated application of techniques such as dedicated search space construction, differentiable structure search, hybrid precision quantization, multi-objective joint optimization, and knowledge distillation, the efficiency and accuracy balance problem in chip defect detection is successfully solved, providing chip manufacturing companies with an efficient and reliable defect detection solution. This method is not only applicable to the specific chip production line in this example, but also has broad application value.

Claims

1. A deep learning-based chip visual design defect detection method, characterized in that, The method comprises the following steps: constructing a special search space and a hardware performance evaluation model, the search space comprising neural network structure components suitable for chip defect feature extraction, and the hardware performance evaluation model being used to predict the execution time of the neural network on a target hardware; implementing a differentiable structure search to optimize the neural network structure for chip defect detection; implementing mixed-precision quantization to adaptively assign optimal bit widths to different layers of the neural network, thereby optimizing model storage and computing efficiency; performing multi-objective joint optimization to simultaneously optimize the neural network structure and the quantization bit width configuration; applying a knowledge distillation technique to migrate the knowledge of a large teacher model to an optimized small student model, thereby improving chip defect detection accuracy; the step of constructing the special search space and the hardware performance evaluation model comprises: constructing a neural network search space suitable for defect feature extraction, comprising convolution layers, pooling layers, and attention modules, according to the characteristics of chip defect images; collecting performance data of a target hardware platform to form a hardware characteristic dataset; training a hardware execution time prediction model, which is implemented using a multi-layer perceptron structure and comprises an input layer, a feature extraction layer, a fusion layer, and an output layer, receives neural network operation parameters and hardware specifications as input, and is trained by minimizing the mean square error between the predicted time and the actual time; the step of performing multi-objective joint optimization comprises: establishing a joint optimization objective function that takes into account detection accuracy, inference delay, and model size; constructing a joint search space that contains two dimensions of network structure and quantization bit width; optimizing the joint search space using a differential evolution algorithm, which comprises population initialization, mutation operation, crossover operation, and selection operation, uses a fast non-dominated sorting algorithm to sort candidate solutions, and uses a crowding distance mechanism to maintain the diversity of solutions; training a corresponding neural network for each candidate solution and evaluating its performance; selecting the most suitable solution from the Pareto frontier according to specific application requirements.

2. The deep learning-based chip visual design defect detection method of claim 1, wherein, the step of implementing a differentiable structure search comprises: based on the search space, establishing a differentiable network structure representation that represents network structure selection as continuous variables; using an alternating optimization algorithm to optimize weight parameters with fixed structure parameters on the training set and to optimize structure parameters with fixed weight parameters on the validation set, and using a second-order approximation method to accelerate structure optimization; during the training process, adding a delay-aware regularization term based on the feedback of the hardware performance evaluation model. 3.The deep learning-based chip visual design defect detection method of claim 1, wherein, the step of implementing mixed-precision quantization comprises: establishing a quantization representation method to convert full-precision parameters to low-bit-width representations; sequentially applying different bit-width quantization to each layer of the neural network, calculating precision loss, generating a sensitivity atlas, and recording the precision changes of each layer under different quantization bit widths; using an iterative greedy algorithm to preferentially apply lower bit widths to less sensitive layers while meeting overall accuracy requirements; implementing hardware-aware quantization to select the most suitable quantization bit width and computing mode based on the characteristics of the target hardware platform; constructing a quantization-aware training loop, including applying quantization configuration, quantizing weights and activation values in forward propagation, using a pass-through estimator to process gradients in backward propagation, and updating full-precision weights.

4. The deep learning-based chip visual design defect detection method of claim 1, wherein, The step of applying the knowledge distillation technology comprises: building a multi-level knowledge distillation framework, defining a teacher model and a student model; designing a multi-task distillation loss function for chip defect detection, including hard label loss, soft label loss, feature matching loss, and boundary perception loss; implementing a knowledge distillation method based on attention guidance, generating a teacher model attention map and constructing an attention guidance loss; implementing a progressive knowledge distillation training process, including an initialization stage, a feature matching stage, a classification optimization stage, and a quantization fine-tuning stage.

5. The deep learning-based chip visual design defect detection method of claim 4, wherein, The multi-task distillation loss function further comprises: hard label loss, calculating cross-entropy loss using real labels; soft label loss, using the output of the teacher model as soft labels to guide the student model to learn the class probability distribution; feature matching loss, aligning the feature representations of the teacher and student models at the intermediate layers; boundary perception loss, focusing on feature learning in the boundary region of the chip defect.

6. The deep learning-based chip visual design defect detection method of claim 5, wherein, The knowledge distillation method based on attention guidance comprises: generating an attention map of the teacher model to reveal the key areas that the model focuses on; constructing an attention guidance loss to guide the student model to focus on the same key areas as the teacher model; designing a selective distillation strategy for different types of defects, including focusing on shallow features for edge class defects, focusing on middle features for texture class defects, and focusing on deep features for structure class defects.

7. The deep learning-based chip visual design defect detection method of claim 6, wherein, The progressive knowledge distillation training process comprises: the initialization stage, using a pre-trained teacher model and randomly initializing the student model; the feature matching stage, freezing the classification head of the student model and only optimizing the feature extraction part; the classification optimization stage, unfreezing the classification head of the student model and using the complete distillation loss function for training; the quantization fine-tuning stage, applying quantization-aware training to further reduce the precision loss caused by quantization. 8.A deep learning based chip visual design defect detection system, characterized in that, The system for performing the deep learning-based chip visual design defect detection method according to any one of claims 1-7 comprises: a search space construction module for constructing a special search space and a hardware performance evaluation model; a structure search module for implementing a differentiable structure search to optimize the neural network structure for chip defect detection; an accuracy quantization module for implementing hybrid accuracy quantization to adaptively assign optimal bit widths to different layers of the neural network; a joint optimization module for performing multi-objective joint optimization to simultaneously optimize the network structure and the quantization bit width configuration; a knowledge distillation module for applying knowledge distillation technology to transfer the knowledge of a large teacher model to an optimized small student model.

Images