Chip defect detection method based on heterogeneous computing architecture and deep learning model

Abstract

The application discloses a chip defect detection method based on a heterogeneous computing architecture and a deep learning model, relates to the technical field of semiconductor detection and artificial intelligence, and comprises the following steps: an improved double-head defect detection model comprising a backbone network, a feature pyramid network and a double-head is constructed; a teacher model is obtained by training the model with the detection total loss of the double-head as an optimization target; a student model with a smaller parameter quantity than the teacher model is constructed; a distillation loss is defined based on the CBAM attention map difference of the teacher model and the student model at the intermediate feature layer; after joint optimization training of the detection total loss of the student model, gradient sensitivity-based structured pruning is performed; and then, quantization perception training and INT8 quantization are performed to obtain a lightweight defect detection model, which is deployed to a heterogeneous computing platform, and chip defect detection with high precision and low computational complexity is realized.

Description

Technical Field

[0001] This invention belongs to the field of semiconductor testing and artificial intelligence technology, specifically relating to a chip defect detection method based on heterogeneous computing architecture and deep learning model. Background Technology

[0002] With the rapid development of the semiconductor industry, chip manufacturing processes are constantly shrinking, gradually pushing the minimum linewidth to its physical limits. Simultaneously, the explosive growth of technologies such as 5G, artificial intelligence, and the Internet of Things is driving a structural upgrade in the demand for high-end chips. The computing power demand for cloud training chips in the AI ​​field continues to grow rapidly, propelling data center chips towards processes below 7nm. To improve chip quality and avoid the cost of ineffective packaging, numerous steps for detecting chip defects and performance have been added to the manufacturing process. These rigorous quality control measures aim to promptly identify and eliminate chips with manufacturing defects or substandard performance, thereby ensuring the quality and reliability of the final product, improving production yield, reducing manufacturing costs, and meeting the market's demand for high-performance, high-quality chips.

[0003] In the early stages of the semiconductor industry, chip defect detection technology primarily relied on manual visual inspection, a method characterized by low efficiency, low accuracy, high labor costs, and susceptibility to subjective human factors. With the rapid development of machine vision technology, chip defect detection technology has gradually evolved towards intelligentization. Simultaneously, with advancements in chip design and manufacturing technologies, chip production speeds have continuously increased, while traditional inspection methods lag far behind manufacturing speeds, limiting production efficiency. Furthermore, as chip manufacturing processes continue to shrink, the demands on inspection technology are increasing, further driving the development of the chip inspection industry through the integration and application of artificial intelligence with inspection technology.

[0004] Currently, the mainstream testing method in the market is still the traditional manual sampling inspection method. This method suffers from problems such as probability, incompleteness, and a single evaluation standard, making it unable to meet the testing needs of the large-scale chip market and resulting in low defect detection accuracy. Sorting systems based on machine vision technology use cameras or spectral imaging equipment to replace human vision in acquiring images of chips, and then process these images using computers to achieve chip image recognition and inspection. However, there are still many key issues to be resolved in using machine vision for chip defect detection.

[0005] Existing technologies mainly include manual visual inspection and traditional machine vision inspection methods. Their shortcomings are: manual inspection is inefficient and highly subjective, making it difficult to meet the needs of large-scale production; traditional algorithms have limited ability to identify complex backgrounds and minor defects; and deep learning-based inspection models have high computational complexity and are highly dependent on hardware resources.

[0006] Therefore, there is an urgent need for a chip defect detection method that balances high accuracy and low computational complexity. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a chip defect detection method based on heterogeneous computing architecture and deep learning model.

[0008] The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a chip defect detection method based on a heterogeneous computing architecture and a deep learning model, the chip defect detection method comprising: An improved dual-detection-head defect detection model is constructed; the improved dual-detection-head defect detection model includes a backbone network, a feature pyramid network, and dual detection heads; the feature pyramid network includes a global path and a local path; the dual detection heads include a local detection head and a global detection head; The improved dual-detection head defect detection model is trained with the total detection loss of the dual detectors as the optimization objective, resulting in a trained improved dual-detection head defect detection model. The improved dual-detector defect detection model that has been trained is used as the teacher model, and a student model with fewer parameters is constructed. The distillation loss is defined based on the difference between the teacher model and the student model in the CBAM attention map of the intermediate feature layer. The student model is then jointly optimized and trained in combination with the total detection loss of the student model to obtain the student model after knowledge distillation. The student model after knowledge distillation is subjected to gradient-sensitive structured pruning, and then a lightweight defect detection model is obtained through quantization-aware training and INT8 quantization. The lightweight defect detection model is deployed to a heterogeneous computing platform to perform defect detection on the grayscale image of the input chip surface to be tested.

[0009] Secondly, the present invention provides a chip defect detection device based on a heterogeneous computing architecture and a deep learning model, the chip defect detection device comprising: A construction module is used to build an improved dual-detection-head defect detection model; the improved dual-detection-head defect detection model includes a backbone network, a feature pyramid network, and dual detectors; the feature pyramid network includes a global path and a local path; the dual detectors include a local detector and a global detector. The training module is used to train the improved dual-detection head defect detection model with the total detection loss of the dual detection heads as the optimization objective, so as to obtain the trained improved dual-detection head defect detection model. The joint optimization module is used to take the trained improved dual-detector defect detection model as the teacher model and construct a student model with fewer parameters than the teacher model; the distillation loss is defined based on the difference between the teacher model and the student model in the CBAM attention map of the intermediate feature layer, and the student model is subjected to joint optimization training in combination with the total detection loss of the student model to obtain the knowledge-distilled student model. The pruning and quantization module is used to perform gradient-sensitive structured pruning on the student model after knowledge distillation, and then obtain a lightweight defect detection model through quantization-aware training and INT8 quantization. The deployment module is used to deploy the lightweight defect detection model to a heterogeneous computing platform to perform defect detection on the grayscale image of the input chip surface to be detected.

[0010] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a computer program stored in memory, it implements the steps described in any of the above-mentioned chip defect detection methods based on heterogeneous computing architecture and deep learning models.

[0011] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in any of the above-mentioned chip defect detection methods based on heterogeneous computing architecture and deep learning models.

[0012] The present invention provides an improved dual-detection-head defect detection model in a chip defect detection method based on heterogeneous computing architecture and deep learning model. By performing dual-branch feature fusion through the global and local paths of the feature pyramid network, and combining global and local detection heads for collaborative detection, it can simultaneously and accurately identify macroscopic and micro defects, significantly improving the detection accuracy of multi-scale defects.

[0013] By employing a joint optimization strategy that includes attention distillation, structured pruning, and INT8 quantization, using both teacher and student models, the number of model parameters and computational load are significantly reduced while maintaining high accuracy, achieving millisecond-level single-frame inference speed and meeting the high-throughput real-time full inspection requirements of semiconductor production lines.

[0014] By deploying a lightweight defect detection model to a heterogeneous computing platform, inference latency and overall power consumption are significantly reduced, enabling independent control of the entire chip defect detection process and ensuring data security without leaving the factory.

[0015] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a chip defect detection method based on a heterogeneous computing architecture and a deep learning model provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the improved dual-detection-head defect detection model provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the combined lightweighting technology of knowledge distillation and structured pruning provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a chip defect detection device based on a heterogeneous computing architecture and a deep learning model provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0018] To address the challenges of balancing high accuracy and low computational complexity in existing chip defect detection methods, as well as the risks associated with data security and autonomous controllability, this invention provides a chip defect detection method based on a heterogeneous computing architecture and a deep learning model. (See also...) Figure 1 , Figure 1 This is a flowchart illustrating a chip defect detection method based on a heterogeneous computing architecture and a deep learning model, provided by an embodiment of the present invention. The method specifically includes the following steps: Step S101: Construct an improved dual-detection-head defect detection model; the improved dual-detection-head defect detection model includes a backbone network, a feature pyramid network, and dual detection heads; the feature pyramid network includes a global path and a local path; the dual detection heads include a local detection head and a global detection head.

[0019] In this embodiment of the invention, a deep neural network comprising a Backbone, a Neck, and dual detection heads is constructed based on the YOLOv8 architecture to serve as an improved dual-detection-head defect detection model.

[0020] The backbone network is used to extract feature maps of four different scales based on the preprocessed grayscale image of the surface of the chip to be inspected.

[0021] In this embodiment of the invention, the preprocessing method for the grayscale image of the chip surface to be detected includes: Acquire the initial grayscale image of the surface of the chip to be inspected; The initial grayscale image is scaled to a fixed resolution using bilinear interpolation, and then the pixel values ​​are normalized to obtain a normalized grayscale image. The Gaussian filtering algorithm is used to filter noise from the normalized grayscale image to obtain the grayscale image.

[0022] Specifically, the initial grayscale image is first scaled to a fixed resolution (e.g., 1024×1024 pixels) using bilinear interpolation, and then the pixel values ​​are normalized to the [0,1] range to obtain a normalized grayscale image.

[0023] The Gaussian filtering algorithm is used to remove noise from the normalized grayscale image. The kernel size is set to 3×3 and the standard deviation sigma=1.0. After noise filtering, high-frequency random noise is suppressed while edge features are preserved to obtain the grayscale image.

[0024] Furthermore, during the deployment phase, the DVPP module built into chips like the Ascend chip can be used to directly decode, scale, and convert color spaces (e.g., RGB to BGR) of the acquired chip images at the hardware level, avoiding the data transfer overhead between the CPU (Central Processing Unit) and the NPU (Neural Processing Unit). The DVPP module is a hardware acceleration module responsible for digital vision preprocessing.

[0025] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the improved dual-detector defect detection model provided in this embodiment of the invention. The backbone network adopts the CSPDarknet53 (a deep convolutional neural network model for computer vision tasks) structure, including a Focus module (a downsampling module), four CBS modules, and four CSP (Cross Stage Partial Connections) modules. The Focus module performs downsampling, halving the width and height of the input grayscale image while expanding the number of channels. The CBS module includes convolutional layers, batch normalization layers, and activation functions, used to downsampling at different stages of the network to gradually expand the receptive field. The CSP module divides the input feature map into two parts: one part is directly connected via skip connections, and the other part undergoes deep feature extraction through a residual structure. Finally, the two feature parts are concatenated along the channel dimension.

[0026] Specifically, the backbone network includes the Focus module, the first CBS module, the first CSP module CSP2_1, the second CBS module, the second CSP module CSP8_1, the third CBS module, the third CSP module CSP8_2, the fourth CBS module, and the fourth CSP module CSP4_1, which are connected in sequence.

[0027] Specifically, the Focus module is a 3×3 convolutional module (Conv); the first CBS module has a kernel size of 3×3, a stride of 2, and 128 channels; the first CSP module has 128 channels; the second CBS module has a kernel size of 3×3, a stride of 2, and 256 channels; the second CSP module has 256 channels; the third CBS module has a kernel size of 3×3, a stride of 2, and 512 channels; the third CSP module has 512 channels; the fourth CBS module has a kernel size of 3×3, a stride of 2, and 1024 channels; the fourth CSP module has 1024 channels.

[0028] The backbone network extracts feature maps at four scales through layer-by-layer downsampling, including feature map P2 at the first scale, feature map P3 at the second scale, feature map P4 at the third scale, and feature map P5 at the fourth scale.

[0029] For example, when the input grayscale image size is 1024×1024, the first-scale feature map P2 is 256×256 in size, rich in shallow edge and texture information. The second-scale feature map P3 is 128×128 in size, retaining more edge and texture details. The third-scale feature map P4 is 64×64 in size, rich in semantic information and with a large receptive field. The fourth-scale feature map P5 is 32×32 in size, with the largest receptive field and the strongest semantic information.

[0030] Among them, the third-scale feature map P4 and the fourth-scale feature map P5 are deep, low-resolution feature maps that can be used for overall defect detection; the first-scale feature map P2 and the second-scale feature map P3 are shallow, high-resolution feature maps that can be used for micro-defect detection.

[0031] In this embodiment of the invention, a feature pyramid network is used to fuse feature maps of four different scales using global and local paths to obtain a low-resolution fused feature map and a high-resolution fused feature map.

[0032] The feature pyramid network portion employs an improved PAN-FPN (Feature Pyramid Network-Path Aggregation Network) structure, including a global path and a local path, to fuse feature maps at different scales using dual paths. The improved PAN-FPN structure provided in this embodiment enhances the perception capability of the improved dual-detector defect detection model for minute defects through attention enhancement and dilated convolution operations in the local path. The attention enhancement operation can be implemented using the CBAM (Convolutional Block Attention Module, a lightweight attention mechanism to enhance the performance of convolutional neural networks); the dilated convolution operation is implemented using a dilated convolution module, with a dilation coefficient of 2 for the dilated convolution layer.

[0033] The global path is used to fuse deep low-resolution feature maps from four feature maps at different scales, outputting a low-resolution fused feature map to enhance the detection capability of macroscopic defects such as scratches and contamination. The low-resolution fused feature map can include a first low-resolution fused feature sub-map and a second low-resolution fused feature sub-map.

[0034] Specifically, in the global path, the fourth-scale feature map P5 is upsampled by 2 times, and then fused with the third-scale feature map P4 through the first concatenation and convolution fusion module (Concat+Conv) to obtain the first low-resolution fused feature sub-map P5_out; then the first low-resolution fused feature sub-map P5_out is downsampled by 2 times, and fused with the third-scale feature map P4 through the second concatenation and convolution fusion module to obtain the second low-resolution fused feature sub-map P4_out.

[0035] For example, when the size of the input grayscale image is 1024×1024, the size of the first low-resolution fusion feature sub-image P5_out is 32×32, and the size of the second low-resolution fusion feature sub-image P4_out is 64×64.

[0036] The local path is used to fuse shallow high-resolution feature maps from four feature maps at different scales, outputting a high-resolution fused feature map to enhance the perception of minute defects such as TSV (Through Silicon Vias) holes and microcracks. The high-resolution fused feature map includes a first high-resolution fused feature sub-map and a second high-resolution fused feature sub-map.

[0037] In this embodiment of the invention, a CBAM module and a dilated convolution module are introduced before the local path enters the local head to enhance the ability to express local detailed features and expand the receptive field.

[0038] Specifically, in the local path, the second-scale feature map P3 undergoes attention enhancement through the first CBAM module, and then its receptive field is expanded through the first dilated convolution module to obtain the first intermediate feature. Similarly, the first-scale feature map P2 undergoes attention enhancement through the second CBAM module, and then its receptive field is expanded through the second dilated convolution module to obtain the second intermediate feature. The first intermediate feature is upsampled by a factor of 2 and input together with the second intermediate feature into the fourth concatenation and convolution fusion module for concatenation and fusion, resulting in the first high-resolution fused feature sub-map P2_loc. The second intermediate feature is downsampled by a factor of 2 and input together with the first intermediate feature into the third concatenation and convolution fusion module for concatenation and fusion, resulting in the second high-resolution fused feature sub-map P3_loc. The dilation coefficient of both the first and second dilated convolution modules is 2.

[0039] For example, when the size of the input grayscale image is 1024×1024, the size of the first high-resolution fusion feature sub-image P2_loc is 256×256, and the size of the second high-resolution fusion feature sub-image P3_loc is 128×128.

[0040] In this embodiment of the invention, a dual detection head is used for defect detection based on a low-resolution fused feature map and a high-resolution fused feature map.

[0041] In one implementation, the dual detection heads include a global detection head and a local detection head; The global detection head is used to detect macroscopic defects based on low-resolution fused feature maps, and outputs the first predicted category, the first bounding box, and the first confidence score. The local detection head is used to detect minute defects based on high-resolution fused feature maps, and outputs a second predicted category, a second bounding box, and a second confidence score.

[0042] Two detection heads work together to achieve multi-scale detection of chip defects.

[0043] In this embodiment of the invention, the global detection head is configured to receive low-resolution fused feature maps from the global path output of the feature pyramid network, including a first low-resolution fused feature sub-map and a second low-resolution fused feature sub-map. The low-resolution fused feature maps have a large receptive field and rich deep semantic information, making them suitable for characterizing the contextual features of macroscopic defects such as scratches, contaminated areas, or overall structural shifts. Based on this, the global detection head performs multi-scale defect prediction, outputting a first predicted category (e.g., macroscopic defect type), a first bounding box (e.g., the position coordinates of the macroscopic defect in the grayscale image), and a first confidence level (e.g., the degree of certainty that the macroscopic defect exists) for the macroscopic defect.

[0044] The local detection head is configured to receive high-resolution fused feature maps from local paths in the feature pyramid network, including a first high-resolution fused feature sub-map and a second high-resolution fused feature sub-map. The high-resolution fused feature maps retain more shallow texture details and edge information, which is beneficial for capturing fine features of minute defects such as TSV holes, grain edge chipping, or microcracks. Based on this, the local detection head performs high-density prediction, outputting a second predicted category (e.g., defect type), a second bounding box (e.g., the location coordinates of the defect in the grayscale image), and a second confidence level (e.g., the degree of certainty of the defect's existence).

[0045] In this embodiment of the invention, CSPDarknet53 is used as the backbone network, and multiple CSP modules are used to extract multi-scale deep semantic features of the chip image. A dual-path feature enhancement mechanism is introduced into the feature pyramid network to solve the problem of huge differences between macroscopic and microscopic defect scales; two independent detection heads are connected in parallel at the end of the feature pyramid network, each responsible for detecting defects at different scales.

[0046] Step S102: The improved dual-detection head defect detection model is trained with the total detection loss of the dual detectors as the optimization objective, and the trained improved dual-detection head defect detection model is obtained.

[0047] To further improve the defect detection accuracy of the improved dual-detector defect detection model, the following training was performed: In this embodiment of the invention, the improved dual-detection head defect detection model is trained with the total detection loss of the dual detectors as the optimization objective, resulting in a trained improved dual-detection head defect detection model, including: Acquire multiple grayscale image samples of defects on the chip surface and their corresponding ground truth labels; the ground truth labels corresponding to the defect grayscale image samples include the chip's actual defect category and actual bounding box; After preprocessing the grayscale image samples of each defect, they are input into the improved dual-detection-head defect detection model for defect detection, and the predicted category, predicted bounding box, and predicted confidence of the defect are output. Using the real labels corresponding to each defect grayscale image sample, as well as the predicted category, predicted bounding box, and predicted confidence of the defect, the total detection loss of the dual detection heads is calculated. The improved dual-head defect detection model is optimized in the direction of reducing the total detection loss until the preset training conditions are met, resulting in a trained improved dual-head defect detection model.

[0048] In this embodiment of the invention, the real label corresponding to the defect grayscale image sample includes the chip's real defect category and real bounding box, specifically including a first real defect category (real type of macroscopic defect) and a first real bounding box (real position coordinates of macroscopic defects in the defect grayscale image sample), as well as a second real defect category (real type of micro-defect) and a second real bounding box (real position coordinates of micro-defects in the defect grayscale image sample).

[0049] After preprocessing each defect grayscale image sample, it is input into the improved dual-detector defect detection model for defect detection. The model outputs the predicted category, predicted bounding box, and predicted confidence of the defect. Specifically, it includes the first predicted category (predicted type of macroscopic defect), the first predicted bounding box (predicted position coordinates of macroscopic defect in defect grayscale image sample), and the first predicted confidence (degree of certainty of the existence of macroscopic defect), as well as the second predicted category (predicted type of microscopic defect), the second predicted bounding box (predicted position coordinates of microscopic defect in defect grayscale image sample), and the second predicted confidence (degree of certainty of the existence of microscopic defect).

[0050] In this embodiment of the invention, calculating the total detection loss of the dual detection heads includes: Calculate the global loss corresponding to the global detection head. The global loss includes the global classification loss, the global bounding box regression loss, and the global object confidence loss. Calculate the local loss corresponding to the local detection head. The local loss includes the local classification loss, the local bounding box regression loss, and the local target confidence loss. The total detection loss is obtained by weighting and summing the global and local losses based on preset weight coefficients.

[0051] Specifically, the global classification loss is calculated based on the difference between the first true defect category and the first predicted category, and is used to measure the accuracy of the model's judgment on macroscopic defect types; the global bounding box regression loss is calculated based on the geometric deviation between the first true bounding box and the first predicted bounding box, and is used to measure the degree of overlap between the model's predicted macroscopic defect location and the true location; the local classification loss is calculated based on the difference between the second true defect category and the second predicted category, and is used to measure the accuracy of the model's judgment on minor defect types; the local bounding box regression loss is calculated based on the geometric deviation between the second true bounding box and the second predicted bounding box, and is used to measure the degree of overlap between the model's predicted minor defect location and the true location; the global target confidence loss and the local target confidence loss are calculated based on the matching degree between the first predicted confidence and the second predicted confidence and the corresponding true label, respectively, and are used to measure the reliability of the model's judgment on the existence of defects.

[0052] In this embodiment of the invention, the total loss of the dual detection heads... include: ; in, Indicates global loss; Indicates localized loss; and For preset weighting coefficients, , The optimal sensitivity can be determined through experimental tuning to balance the detection sensitivity of macroscopic and microscopic defects. and All use the original YOLOv8 loss function.

[0053] In this embodiment of the invention, preset weighting coefficients for defects of different scales are dynamically balanced. The improved dual-detection-head defect detection model exhibits high throughput characteristics with an accuracy loss of <1%, and can be adapted to different process nodes and wafer fab inspection environments through fine-tuning without retraining.

[0054] Testing revealed that the dual detection head provided in this embodiment of the invention achieved a 12.7% improvement in detection accuracy (mAP@0.5) on a chip die dataset compared to a single detection head. Specifically, the detection rate for 0.5μm-level TSV void defects increased from 83.2% to 96.8%, while maintaining a real-time inference speed of 45 FPS. Feature visualization analysis revealed that the global detection head's activation area primarily covers the overall defect distribution on the wafer surface, while the local detection head precisely locates the local features of grain-level defects. The superposition of these two feature maps forms a "global localization-local magnification" detection mode, effectively overcoming the technical bottleneck of single-scale features failing to simultaneously detect both macroscopic and microscopic defects.

[0055] Step S103: The improved dual-detector defect detection model that has been trained is used as the teacher model, and a student model with fewer parameters is constructed. The distillation loss is defined based on the difference between the teacher model and the student model in the CBAM attention map of the intermediate feature layer. The student model is then jointly optimized and trained by combining the total detection loss of the student model to obtain the student model after knowledge distillation.

[0056] In this embodiment of the invention, the improved dual-detector defect detection model is used as the teacher model, and a student model with fewer parameters is constructed.

[0057] By using an improved dual-detector defect detection model as the teacher model, high-precision defect feature representation can be provided. During the lightweighting phase, a student model corresponding to the teacher model is constructed. The student model maintains the same overall topology as the teacher model, but its complexity is reduced by decreasing the number of network channels and convolutional parameters. The student model learns the attention distribution and semantic representation capabilities of the teacher model at different scale feature layers through knowledge distillation, thus maintaining high detection accuracy even after lightweighting.

[0058] For example, the complex teacher network model was compressed from 43.1MB of parameters and 84.2 GFLOPs of floating-point operations to a highly efficient student model of only 14.7MB. The number of backbone network channels was pruned from [64,128,256,512,1024] to [48,96,192,384,768], achieved by a 25% reduction. All 3×3 convolutions in the feature pyramid network of the teacher network were replaced with 1×1+3×3 grouped convolutions, where group (number of groups) = 4, reducing redundant computation. While maintaining the original dual-detector topology, i.e., 3 anchor-free heads per scale, the number of output channels of each detector head was compressed from 64 to 48, thereby effectively reducing the computational cost in the final prediction stage.

[0059] In this embodiment of the invention, the distillation loss is defined based on the difference between the CBAM attention maps of the teacher model and the student model in the intermediate feature layer, including: The feature maps of the teacher model and the student model at three scales output by the feature pyramid network are obtained respectively; the feature maps at three scales here include the second scale feature map, the third scale feature map, and the fourth scale feature map; A lightweight attention mechanism to enhance the performance of the convolutional neural network is applied to the feature map at each scale to generate teacher CBAM attention map and student CBAM attention map; Calculate the L2 distance between the teacher's CBAM attention map and the student's CBAM attention map, and use the sum of the L2 distances as the distillation loss.

[0060] In an embodiment of the invention, distillation loss include: ; in, This represents a lightweight attention mechanism that enhances the performance of convolutional neural networks. In the teacher model, the first Feature maps at various scales; In the student model, the first Feature maps at various scales; among them For scale indexing, The corresponding feature map P3 is the second-scale feature map in the feature pyramid network. The corresponding feature map P4 is the third-scale feature map in the feature pyramid network. P5 corresponds to the fourth-scale feature map in the feature pyramid network; Represents the teacher's CBAM attention map; This represents the student's CBAM attention map.

[0061] In this embodiment of the invention, the joint optimization loss function that combines the distillation loss with the total detection loss of the student model for joint optimization training includes: ; in, Indicates the joint optimization loss; This represents the total detection loss of the student model; This indicates the preset distillation weighting coefficient.

[0062] By minimizing the joint optimization loss function, the student model is forced to mimic the teacher model's attention distribution to minor defects in the intermediate feature layers, thus maintaining high sensitivity even after model compression.

[0063] See Figure 3 , Figure 3 This is a schematic diagram of the knowledge distillation and structured pruning joint lightweight technology provided in this embodiment of the invention. The trained teacher model outputs a teacher CBAM attention map to form soft supervision information, and simultaneously outputs teacher prediction results; the student model to be trained obtains the corresponding student CBAM attention map and student prediction results. The L2 distance between the teacher CBAM attention map and the student CBAM attention map is calculated, and the sum of the L2 distances is used as the distillation loss; the ground truth labels and student prediction results are used to calculate the total detection loss of the student model. The distillation loss and the total detection loss of the student are weighted and summed to form a joint optimization objective, and the knowledge-distilled student model is obtained through joint optimization.

[0064] Step S104: Perform gradient-sensitive structured pruning on the student model after knowledge distillation, and then obtain a lightweight defect detection model through quantization-aware training and INT8 quantization.

[0065] In this embodiment of the invention, the process of performing gradient-sensitive structured pruning on the student model after knowledge distillation is as follows: First, using the first-order gradient magnitude statistical method, a predetermined number of grayscale image samples (e.g., 1000 images) from the chip surface in the validation set are input into the knowledge-distilled student model for forward propagation. For each network layer in the backbone network of the knowledge-distilled student model... and each channel Calculate the feature map output by this channel for the final detection loss. The mean magnitude of the first-order gradient is defined as the gradient sensitivity. : ; in, The representation layer index refers to a specific layer in the backbone network, such as the network layer in the first CBS module or the second CSP module. This represents the channel index, referring to a specific channel in a feature map of a certain layer. ; Indicates the index of the grayscale image sample on the surface of the chip in the verification set; This represents the total number of grayscale image samples on the surface of the chip in the verification set; Indicates the first The grayscale image samples of the surface of the verification set chip in the first... Layer The corresponding values ​​on the feature map output by each channel.

[0066] Gradient sensitivity The meaning is the first Layer The impact of each channel on the detection results; a higher gradient sensitivity indicates a greater impact, and a lower gradient sensitivity indicates a smaller impact. See also Figure 3 The concept of sparsification of weight levels is illustrated by removing connections, which means setting the weights of channels with low gradient sensitivity to 0; removing neurons means physically removing channels with low gradient sensitivity and their associated weights, performing structured pruning, thereby achieving stable acceleration without relying on sparse operators.

[0067] For each layer, channels are sorted in ascending order of gradient sensitivity. The channels with low sensitivity that are in the bottom X% (e.g., 15%) are physically removed, and the channels that contribute the most to defect detection are retained.

[0068] In order to eliminate the accuracy oscillation caused by pruning after pruning, the remaining network parameters are subjected to short-term fine-tuning in this embodiment of the invention. This allows the remaining channel parameters to quickly adapt to the pruned topology, thereby restoring the detection accuracy of the model without significantly increasing the training cost.

[0069] After testing, the model achieved a single-frame inference latency of 7.8ms (128FPS) on existing AI (Artificial Intelligence) accelerator cards after gradient-sensitive structured pruning, which is 3.6 times faster than the original model. At the same time, the mAP@0.5 only decreased by 1.2%. It maintained a detection accuracy of 95.6% in key scenarios such as 0.3μm-level die edge chipping and TSV hole defects. It successfully overcame the technical contradiction between high accuracy and low latency in chip defect detection, and provided 3D packaging production lines with a real-time full inspection capability of 34 dies per second.

[0070] In this embodiment of the invention, after structured pruning, quantization-aware training is performed to convert the model from floating-point calculations to integer calculations.

[0071] In this embodiment of the invention, pseudo-quantization nodes are inserted between each convolutional layer and the activation function of the structured pruned student model. Through forward propagation, pseudo-quantization nodes simulate the truncation and rounding operations of INT8 (8-bit integers), quantizing floating-point weights and activation values ​​into low-precision values, then dequantizing them back into floating-point numbers, which are then used as inputs for the next layer.

[0072] By using backpropagation and employing a straight-through estimator (STE) to update the weights, the model adapts to the low-precision computing environment during the training phase, thereby achieving lossless INT8 inference during subsequent formal deployment.

[0073] A calibration set is constructed using real chip images (e.g., 500 images) from the production line. The MinMax quantization strategy is used to count the maximum and minimum values ​​of activation values ​​in each layer, and the quantization scaling factor and zero point are determined.

[0074] After training, the model weights and activation values ​​are formally converted from FP32 (32-bit floating-point numbers) to INT8 format to obtain the final lightweight chip defect detection model.

[0075] The size of the lightweight chip defect detection model has been further compressed from 14.7MB after pruning to 7.2MB, significantly reducing floating-point operations. Testing on the Ascend 310B hardware shows that the inference accuracy (mAP@0.5) of the lightweight chip defect detection model is only 0.9% lower than that of the FP32 model, fully meeting industrial-grade detection standards. Simultaneously, the single-frame inference latency has been reduced to 7.8ms, achieving high real-time detection at 128FPS.

[0076] Step S105: Deploy the lightweight defect detection model to a heterogeneous computing platform to perform defect detection on the grayscale image of the input chip to be detected.

[0077] In this embodiment of the invention, a lightweight defect detection model is deployed to a heterogeneous computing platform to perform real-time inference detection of defects in chips under test on an actual production line. The heterogeneous computing platform is equipped with an AI acceleration processor supporting INT8 fixed-point computing power and its supporting heterogeneous computing toolchain.

[0078] Specifically, this embodiment of the invention relies on the MindSpore deep learning framework and the CANN (Compute Architecture for Neural Networks) heterogeneous computing toolchain to achieve end-to-end adaptation of algorithms, frameworks, and hardware. MindSpore is an open-source deep learning framework.

[0079] Optionally, before inputting the grayscale image of the chip surface to be inspected into the lightweight defect detection model, the preprocessing operation described above needs to be performed, and then defect detection is performed based on the preprocessed grayscale image.

[0080] In actual deployment, the lightweight defect detection model is first converted from its original framework format (such as PyTorch) to an offline model format (such as Ascend Tensor Compiler) supported by the target accelerator processor of the heterogeneous computing platform, using an intermediate representation such as ONNX (Open Neural Network Exchange), via an intermediate representation. After conversion, the model is then compared using the msquickcmp tool to verify that the output error between the offline model and the original model under the same input conditions is strictly less than a preset threshold, such as 1e-5, ensuring lossless feature mapping during model migration. The msquickcmp tool is a one-click, end-to-end accuracy comparison tool.

[0081] To address the instruction set characteristics of the target accelerator processor, operator-level compilation optimizations are implemented. The continuous sequence of "convolution operation + batch normalization + activation function" in the backbone network and dual detector heads is fused into a single custom fusion operator, eliminating repeated read / write operations of intermediate tensors in GPU memory and improving computational throughput. For non-standard operators not natively supported by heterogeneous computing toolchains, such as learnable channel mask operations in dynamic mask training or custom channel / spatial normalization functions in the CBAM module, custom operators are developed using the tensor acceleration engine to ensure complete transfer of the model topology.

[0082] By enabling the built-in memory optimizer of the heterogeneous computing platform, the system automatically parses the data dependencies of the model computation graph and intelligently reuses and schedules the memory space of the feature map. This reduces peak inference memory usage by 32% while ensuring computational correctness, and adapts to the on-chip cache resources of the accelerator processor. Simultaneously, it utilizes the processor's built-in AI preprocessing module to directly perform normalization, size scaling, and color space conversion of the input chip image at the hardware level, completely eliminating the data transfer latency and bus bandwidth bottlenecks between the central processing unit and the accelerator processor in traditional solutions.

[0083] The lightweight defect detection model achieves an end-to-end inference latency of 7.8ms (equivalent to 128 FPS) on the Ascend 310B platform, a 3.6x speedup compared to the original unoptimized model. Compared to existing GPU (Graphics Processing Unit) solutions with equivalent detection accuracy, the overall power consumption is reduced by 68%, achieving synergistic optimization of high precision, low latency, and low power consumption. This provides semiconductor packaging production lines with a full inspection solution that allows data to remain within the factory and computing power to be independently controlled.

[0084] In this embodiment of the invention, the improved dual-detection-head defect detection model is constructed by fusing dual-branch features through the global and local paths of the feature pyramid network. Combined with the collaborative detection of the global and local detection heads, it can simultaneously and accurately identify both macroscopic and micro-defects, significantly improving the detection accuracy of multi-scale defects.

[0085] By employing a joint optimization strategy that includes attention distillation, structured pruning, and INT8 quantization, using both teacher and student models, the number of model parameters and computational load are significantly reduced while maintaining high accuracy, achieving millisecond-level single-frame inference speed and meeting the high-throughput real-time full inspection requirements of semiconductor production lines.

[0086] By deploying a lightweight defect detection model to a heterogeneous computing platform, inference latency and overall power consumption are significantly reduced, enabling independent control of the entire chip defect detection process and ensuring data security without leaving the factory.

[0087] Based on the same inventive concept, embodiments of the present invention also provide a chip defect detection device based on a heterogeneous computing architecture and a deep learning model, see [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of a chip defect detection device based on a heterogeneous computing architecture and a deep learning model provided in an embodiment of the present invention. The chip defect detection device includes: Module 401 is used to construct an improved dual-detection-head defect detection model. The improved dual-detection-head defect detection model includes a backbone network, a feature pyramid network, and dual detectors. The feature pyramid network includes a global path and a local path. The dual detectors include a local detector and a global detector. Training module 402 is used to train the improved dual-detection head defect detection model with the total detection loss of the dual detection heads as the optimization objective, so as to obtain the trained improved dual-detection head defect detection model. The joint optimization module 403 is used to take the trained improved dual-detector defect detection model as the teacher model and construct a student model with fewer parameters than the teacher model. The distillation loss is defined based on the difference between the teacher model and the student model in the CBAM attention map of the intermediate feature layer. The student model is then jointly optimized and trained in combination with the total detection loss of the student model to obtain the student model after knowledge distillation. The pruning and quantization module 404 is used to perform gradient-sensitive structured pruning on the student model after knowledge distillation, and then obtain a lightweight defect detection model through quantization-aware training and INT8 quantization. Deployment module 405 is used to deploy a lightweight defect detection model to a heterogeneous computing platform to perform defect detection on the grayscale image of the input chip to be tested.

[0088] In this embodiment of the invention, the improved dual-detection-head defect detection model is constructed by fusing dual-branch features through the global and local paths of the feature pyramid network. Combined with the collaborative detection of the global and local detection heads, it can simultaneously and accurately identify both macroscopic and micro-defects, significantly improving the detection accuracy of multi-scale defects.

[0089] By employing a joint optimization strategy that includes attention distillation, structured pruning, and INT8 quantization, using both teacher and student models, the number of model parameters and computational load are significantly reduced while maintaining high accuracy, achieving millisecond-level single-frame inference speed and meeting the high-throughput real-time full inspection requirements of semiconductor production lines.

[0090] By deploying a lightweight defect detection model to a heterogeneous computing platform, inference latency and overall power consumption are significantly reduced, enabling independent control of the entire chip defect detection process and ensuring data security without leaving the factory.

[0091] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. Memory 503 is used to store computer programs; When the processor 501 executes the program stored in the memory 503, it implements the method steps of any of the above-mentioned chip defect detection methods based on heterogeneous computing architecture and deep learning models.

[0092] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0093] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0094] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0095] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0096] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements the method steps of any of the above-described chip defect detection methods based on heterogeneous computing architecture and deep learning models.

[0097] Optionally, the computer-readable storage medium may be non-volatile memory (NVM), such as at least one disk storage device.

[0098] Optionally, the aforementioned computer-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0099] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the steps of the method described in any of the above-described chip defect detection methods based on heterogeneous computing architecture and deep learning models.

[0100] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0101] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0102] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0103] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0104] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0105] It should be noted that the device, electronic device and storage medium in the embodiments of the present invention are respectively the device, electronic device and storage medium for applying the above-mentioned chip defect detection method based on heterogeneous computing architecture and deep learning model. Therefore, all embodiments of the above-mentioned chip defect detection method based on heterogeneous computing architecture and deep learning model are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0106] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A chip defect detection method based on heterogeneous computing architecture and deep learning model, characterized in that, The chip defect detection method includes: An improved dual-detection-head defect detection model is constructed; the improved dual-detection-head defect detection model includes a backbone network, a feature pyramid network, and dual detection heads; the feature pyramid network includes a global path and a local path; the dual detection heads include a local detection head and a global detection head; The improved dual-detection head defect detection model is trained with the total detection loss of the dual detectors as the optimization objective, resulting in a trained improved dual-detection head defect detection model. The improved dual-detector defect detection model that has been trained is used as the teacher model, and a student model with fewer parameters is constructed. The distillation loss is defined based on the difference between the teacher model and the student model in the CBAM attention map of the intermediate feature layer. The student model is then jointly optimized and trained in combination with the total detection loss of the student model to obtain the student model after knowledge distillation. The student model after knowledge distillation is subjected to gradient-sensitive structured pruning, and then a lightweight defect detection model is obtained through quantization-aware training and INT8 quantization. The lightweight defect detection model is deployed to a heterogeneous computing platform to perform defect detection on the grayscale image of the input chip surface to be tested. Perform gradient-sensitive structured pruning on the knowledge-distilled student model, including: Using the first-order gradient magnitude statistical method, a preset number of grayscale image samples from the chip surface in the validation set are input into the knowledge-distilled student model for forward propagation; for each network layer in the backbone network of the knowledge-distilled student model... and each channel Calculate the feature map output by this channel for the detection loss. The mean magnitude of the first-order gradient is defined as the gradient sensitivity. : ； in, Presentation layer index; Indicates the channel index; The index represents the grayscale image sample on the surface of the chip in the verification set; This represents the total number of grayscale image samples on the surface of the chip in the verification set; Indicates the first The grayscale image samples of the surface of the verification set chip in the first... Layer The corresponding values ​​on the feature map output by each channel.

2. The chip defect detection method according to claim 1, characterized in that, The backbone network is used to extract feature maps of four different scales based on the preprocessed grayscale image of the surface of the chip to be detected. The feature pyramid network is used to fuse the feature maps of the four different scales using the global path and the local path to obtain a low-resolution fused feature map and a high-resolution fused feature map. The dual detection head is used to perform defect detection based on the low-resolution fused feature map and the high-resolution fused feature map.

3. The chip defect detection method according to claim 2, characterized in that, The global path is used to fuse the deep low-resolution feature maps in the four feature maps of different scales to output a low-resolution fused feature map; the local path is used to fuse the shallow high-resolution feature maps in the four feature maps of different scales to output a high-resolution fused feature map.

4. The chip defect detection method according to claim 2, characterized in that, The global detection head is used to detect macroscopic defects based on the low-resolution fused feature map, and outputs a first predicted category, a first bounding box, and a first confidence score. The local detection head is used to detect minute defects based on the high-resolution fused feature map, and outputs a second predicted category, a second bounding box, and a second confidence score.

5. The chip defect detection method according to claim 2, characterized in that, The preprocessing method for the grayscale image of the surface of the chip to be detected includes: Obtain an initial grayscale image of the surface of the chip to be inspected; The initial grayscale image is scaled to a fixed resolution using bilinear interpolation, and then the pixel values ​​are normalized to obtain a normalized grayscale image. The normalized grayscale image is subjected to noise filtering using a Gaussian filtering algorithm to obtain a grayscale image.

6. The chip defect detection method according to claim 1, characterized in that, The improved dual-detector defect detection model is trained using the total detection loss of the dual detectors as the optimization objective, resulting in a trained improved dual-detector defect detection model, including: Acquire multiple grayscale image samples of defects on the chip surface and their corresponding ground truth labels; the ground truth labels corresponding to the defect grayscale image samples include the chip's actual defect category and actual bounding box; After preprocessing each defect grayscale image sample, it is input into the improved dual-detection head defect detection model for defect detection, and the predicted defect category, predicted bounding box and predicted confidence score are output. Using the real labels corresponding to each defect grayscale image sample, as well as the predicted category, predicted bounding box, and predicted confidence of the defect, the total detection loss of the dual detection head is calculated. The improved dual-detection head defect detection model is optimized in the direction of reducing the total detection loss until the preset training conditions are met, resulting in a trained improved dual-detection head defect detection model.

7. The chip defect detection method according to claim 6, characterized in that, Calculate the total detection loss of the dual-detector head, including: Calculate the global loss corresponding to the global detection head, wherein the global loss includes global classification loss, global bounding box regression loss, and global target confidence loss; Calculate the local loss corresponding to the local detection head, whereby the local loss includes local classification loss, local bounding box regression loss, and local target confidence loss; The total detection loss is obtained by weighting and summing the global loss and the local loss based on preset weight coefficients.

8. A chip defect detection device based on heterogeneous computing architecture and deep learning model, characterized in that, The chip defect detection device includes: A construction module is used to build an improved dual-detection-head defect detection model; the improved dual-detection-head defect detection model includes a backbone network, a feature pyramid network, and dual detectors; the feature pyramid network includes a global path and a local path; the dual detectors include a local detector and a global detector. The training module is used to train the improved dual-detection head defect detection model with the total detection loss of the dual detection heads as the optimization objective, so as to obtain the trained improved dual-detection head defect detection model. The joint optimization module is used to take the trained improved dual-detector defect detection model as the teacher model and construct a student model with fewer parameters than the teacher model; the distillation loss is defined based on the difference between the teacher model and the student model in the CBAM attention map of the intermediate feature layer, and the student model is subjected to joint optimization training in combination with the total detection loss of the student model to obtain the knowledge-distilled student model. The pruning and quantization module is used to perform gradient-sensitive structured pruning on the student model after knowledge distillation, and then obtain a lightweight defect detection model through quantization-aware training and INT8 quantization. The deployment module is used to deploy the lightweight defect detection model to a heterogeneous computing platform to perform defect detection on the grayscale image of the input chip surface to be detected. The pruning and quantization module performs gradient-sensitive structured pruning on the knowledge-distilled student model, including: Using the first-order gradient magnitude statistical method, a preset number of grayscale image samples from the chip surface in the validation set are input into the knowledge-distilled student model for forward propagation; for each network layer in the backbone network of the knowledge-distilled student model... and each channel Calculate the feature map output by this channel for the detection loss. The mean magnitude of the first-order gradient is defined as the gradient sensitivity. : ； in, Presentation layer index; Indicates the channel index; The index represents the grayscale image sample on the surface of the chip in the verification set; This represents the total number of grayscale image samples on the surface of the chip in the verification set; Indicates the first The grayscale image samples of the surface of the verification set chip in the first... Layer The corresponding values ​​on the feature map output by each channel.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a computer program stored in memory, implements the chip defect detection method based on heterogeneous computing architecture and deep learning model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the chip defect detection method based on heterogeneous computing architecture and deep learning model as described in any one of claims 1-7.

Images