Chip surface defect detection method and system based on deep learning, and computer readable medium

Abstract

The invention discloses a chip surface defect detection method and system based on deep learning, and a computer readable medium. The method comprises the following steps: acquiring a bright field image and a dark field image of a chip; fusing the bright field image and the dark field image to generate a fused image; performing coarse positioning on a chip area in the fused image to obtain an external rectangular frame of the chip area; in the external rectangular frame, accurately positioning the edge of the chip to obtain an accurate area of the chip; dividing the accurate area into a plurality of overlapped grid image blocks; performing defect detection on each grid image block to obtain a local defect detection result; all local defect detection results are mapped to an original image coordinate system, and chip surface defect information is output after combination. According to the invention, high-precision and high-robustness automatic detection of chip surface defects can be realized, manual intervention is reduced, and the configuration efficiency and detection capability of an AOI system are improved.

Description

Technical Field

[0001] This invention relates to the field of chip surface inspection technology, and in particular to a chip surface defect detection method and system based on deep learning, as well as a computer-readable medium. Background Technology

[0002] With the continuous improvement of semiconductor packaging processes, chip surface defects (such as chipping, cracks, pits, bubbles, scratches, and foreign matter) have an increasingly significant impact on yield and reliability. Automated optical inspection (AOI), as the core quality inspection method in the packaging process, directly determines production line yield and capacity based on its inspection accuracy, stability, and speed. Especially in high-density packaging, ultra-thin chips, and high-precision processes, surface defects often exhibit characteristics such as small size, weak texture, low contrast, and diverse shapes, posing a significant challenge to traditional AOI inspection algorithms.

[0003] Currently, industrial sites commonly use a combination of white coaxial lighting and ring lighting to acquire images of chip surfaces. While this lighting method provides uniform illumination, it often encounters problems such as feature obscuration, texture loss, and overlapping reflections when dealing with weak reflections, light-colored micro-defects, or anomalies with extremely low contrast to the background. These issues make it difficult for both deep learning models and traditional algorithms to fully identify defects. Furthermore, the same defect often exhibits significant differences under different angles and lighting conditions, leading to poor model generalization and a high false negative rate.

[0004] In image processing algorithms, traditional methods often rely on edge detection, threshold segmentation, and connected component analysis. However, in the semiconductor packaging field, due to interference from chamfers, blemishes, and reflective textures at chip edges, simply using traditional algorithms often fails to achieve stable and accurate chip boundary localization, nor can it effectively extract small-sized defect areas.

[0005] In recent years, deep learning technology has been widely used in industrial visual inspection. However, for complex scenarios such as chip surfaces with strong reflective backgrounds, small-sized defects, and weak texture features, directly using deep learning models still has the following shortcomings: 1. Weak texture defects are difficult for the model to learn, leading to missed detections.

[0006] 2. The proportion of small targets is extremely low, the model training is insufficient, and the detection performance is unstable.

[0007] 3. Large input images slow down inference speed and the model does not pay enough attention to local details.

[0008] 4. Lack of precise localization of chip areas, excessively broad inference range, and excessive noise interference.

[0009] 5. Traditional single-model inference speed is limited and cannot meet the needs of high-speed production lines.

[0010] Therefore, existing technologies still have significant shortcomings in areas such as weak texture extraction, small target detection, precise edge localization, and real-time inference efficiency. Summary of the Invention

[0011] According to a first aspect of the present invention, a chip surface defect detection method based on deep learning is provided, comprising the following steps: Acquire bright-field and dark-field images of the chip; The bright-field image and the dark-field image are fused to generate a fused image; Coarsely locate the chip region in the fused image to obtain the bounding rectangle of the chip region; Within the circumscribed rectangle, the chip edge is precisely located to obtain the precise area of ​​the chip; Divide the precise region into multiple overlapping grid image blocks; Defect detection is performed on each grid image block to obtain local defect detection results; Map all local defect detection results back to the original coordinate system, merge them, and output the chip surface defect information.

[0012] Furthermore, the bright field image and the dark field image are fused to generate a fused image. Specifically, the bright field image is used as the first channel, the dark field image is used as the second channel, and the difference image between the bright field and the dark field images is used as the third channel. The three channels are combined to generate a fused image.

[0013] Furthermore, the coarse localization of the chip region in the fused image is specifically performed by using a trained deep learning object detection model to detect the chip region in the fused image and output the bounding rectangle of the chip; the deep learning object detection model is robust to changes in illumination after training.

[0014] Furthermore, within the circumscribed rectangular frame, the chip edge is precisely located to obtain the chip's precise region: Within the circumscribed rectangle, multiple one-dimensional profile lines are projected vertically along the edge of the chip. Gray-level transition points on each profile line are detected, and the point with the maximum gray-level gradient is taken as the edge point, forming a set of edge points. The least squares method is used to fit a straight line to the set of edge points to obtain the precise four edges of the chip. The precise region of the chip is obtained through the intersection of the four edges.

[0015] Furthermore, the grid size and overlap rate of the grid image blocks are configurable, ensuring that there are overlapping areas between adjacent grid image blocks.

[0016] Furthermore, defect detection is performed on each grid image block to obtain local defect detection results. Specifically, a trained deep learning object detection model is used to identify defects in each grid image block and output the defect category, location, and confidence level.

[0017] Furthermore, all local defect detection results are mapped back to the original image coordinate system and merged as follows: Based on the position of the mesh in the original image, convert the defect coordinates within the mesh to the coordinates in the original image; Non-maximum suppression is applied to the detection results of all grids, and overlapping detection boxes are merged.

[0018] Furthermore, the chip surface defect information includes one or more combinations of defect type, defect location, defect quantity, and defect visualization contour map.

[0019] According to a second aspect of the present invention, a chip surface defect detection system based on deep learning is provided, comprising: The acquisition module is used to acquire bright-field and dark-field images of the chip; The fusion module is used to fuse bright-field images and dark-field images to generate a fused image; The coarse localization module is used to coarsely locate the chip region in the fused image and obtain the bounding rectangle of the chip region. The precision positioning module is used to accurately locate the chip edge within the outer rectangular frame, thereby obtaining the precise area of ​​the chip. The region segmentation module is used to divide a precise region into multiple overlapping grid image blocks; The detection module is used to perform defect detection on each grid image block and obtain local defect detection results; The output module is used to map all local defect detection results back to the original coordinate system and output the chip surface defect information after merging.

[0020] According to a third aspect of the present invention, a computer-readable medium having processor-executable non-volatile program code is provided, the program code causing the processor to perform a deep learning-based chip surface defect detection method according to the first aspect.

[0021] A chip surface defect detection method based on deep learning according to an embodiment of the present invention has the following beneficial effects: 1. Solving the problem of insufficient information in traditional white light detection and achieving high-precision identification of complex defects: This invention innovatively proposes an enhancement strategy that fuses bright field and dark field images. By combining multi-source information through a deep learning feature extraction network, the visibility of fine surface texture defects is significantly improved. Compared with the traditional method that relies solely on white coaxial light and ring light, it can effectively solve the problem of easy omission of shallow scratches, micro-dimples, and weak reflection defects, which is conducive to the promotion and implementation of automatic surface quality inspection after semiconductor packaging in industrial scenarios.

[0022] 2. Lightweight model, high-efficiency inference, and adaptable to various industrial equipment platforms: This invention adopts a lightweight deep learning model structure and combines it with a regional gridding mechanism to reduce the overall inference load without sacrificing image resolution, and has a stronger ability to perceive small defects (micrometer level). The model can be flexibly deployed on different hardware platforms such as GPUs, NPUs, and edge computing units to meet the requirements of high-speed detection and multi-channel parallel processing.

[0023] 3. Standardized process design, compatible with multiple chip packaging and process environments: By constructing a generalizable region localization model and a scalable gridded inference strategy, this invention can adapt to chips of different sizes, packaging forms, and surface structures. The system can be quickly migrated to new process platforms with only a small amount of training data replacement, exhibiting extremely high reusability and significantly reducing subsequent maintenance and expansion costs.

[0024] 4. High detection accuracy, strong robustness, and good interpretability: After coarse detection using a deep learning model, boundary correction is enhanced through caliper geometric positioning and result mapping mechanisms, achieving sub-pixel-level precise positioning of the chip area. Simultaneously, gridded inference reduces global error propagation, resulting in more stable detection performance for small target defects. The overall detection process is transparent and structured, with good interpretability, facilitating debugging and verification by engineers.

[0025] 5. Modular algorithm architecture, easy for engineering integration and functional expansion: This invention divides all processing steps such as image fusion, region localization, fine edge localization, mesh segmentation, and small defect detection into independent and reusable modules. Each module can be optimized and replaced individually, achieving standardized engineering integration. This architecture is not only suitable for chip surface defect detection, but can also be transferred to other vision tasks such as package scratch detection, metal device appearance inspection, and foreign object contamination identification, possessing broad scalability.

[0026] 6. A defect detection technology system combining deep learning and traditional image processing is proposed: This invention is the first to combine bright / dark field fusion, deep learning coarse detection, caliper fine calibration, and small target gridded inference into a unified defect detection framework. Feature enhancement and large-area recognition are achieved through a deep learning model, and then precise positioning is performed in key boundary areas using a caliper tool. This gives the overall detection system both high intelligence and geometric stability, enabling reliable detection of small defects under complex lighting conditions.

[0027] 7. Configurable parameters, dynamically optimized based on chip size and defect type: In this invention, parameters such as mesh size, overlap ratio, fusion factor, number and length of calipers, and defect inference threshold all support configurable adjustment and can be dynamically optimized according to different chip structures, surface processes, and defect distribution characteristics. This feature enables the system to adapt well to changes in chip size, differences in defect scale, and differences in reflection intensity.

[0028] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0029] Figure 1 This is a flowchart of a chip surface defect detection method based on deep learning according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of bright-field and dark-field images of a chip in a chip surface defect detection method based on deep learning according to an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of the fused image in a deep learning-based chip surface defect detection method according to an embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram of the meshing and inference results in a deep learning-based chip surface defect detection method according to an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram of the output results in a chip surface defect detection method based on deep learning according to an embodiment of the present invention.

[0034] Figure 6 This is a schematic diagram of a chip surface defect detection system based on deep learning according to an embodiment of the present invention. Detailed Implementation

[0035] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, further illustrating the present invention.

[0036] First, combine Figures 1-5 This invention describes a deep learning-based chip surface defect detection method according to a first aspect of an embodiment of the present invention, which is used for chip surface inspection and has a wide range of applications.

[0037] like Figures 1-5 As shown in the figure, a chip surface defect detection method based on deep learning according to an embodiment of the present invention includes the following steps: like Figures 1-2As shown, in S1, the bright-field and dark-field images of the chip are acquired. A dedicated AOI industrial camera can be used to acquire the bright-field and dark-field images of the chip separately (e.g., ...). Figure 2 (As shown). Bright field image: emphasizes the macroscopic structural contours of the region with less noise. Dark field image: more sensitive to shallow texture defects such as minor scratches, dents, and foreign objects. By fusing the two types of images, the algorithm combines structural edge information and micro-texture information.

[0038] like Figures 1-3 As shown, in S2, the bright-field image and the dark-field image are fused to generate a fused image. Specifically, the bright-field image is used as the first channel, the dark-field image as the second channel, and the difference image between the bright-field and dark-field images as the third channel, and these are combined to generate a three-channel fused image.

[0039] Channel 1: Brightfield image Used for chip edge and contour recognition; Channel 2: Dark Field Image Used for texture defect enhancement; Channel 3: Bright-dark field difference image It enhances low-contrast defects such as pits and chipped edges.

[0040] Generate fused image: ; The three channels retain different feature dimensions, enabling the feature extraction model to simultaneously acquire structural, textural, and differential information.

[0041] Compose is essentially a pixel-by-pixel composition: ; Compared to a single light source, fused images (such as...) Figure 3 (As shown) It contains richer structures and textures, which greatly improves the accuracy of subsequent detection.

[0042] like Figure 1 As shown, in S3, the chip region in the fused image is coarsely located to obtain the bounding rectangle of the chip region. This bounding box is used to limit the search area for subsequent precise edge localization, improving the stability of the caliper tool. Specifically, a trained deep learning object detection model is used to detect the chip region in the fused image and output the bounding rectangle of the chip; the trained deep learning object detection model is robust to changes in illumination. The trained deep learning object detection model can use YOLOv11 or a lightweight model of equivalent precision. Training process: The chip region in the fused image is labeled with bounding boxes or semantic regions, and then data augmentation (rotation, brightness change, blending enhancement, etc.) and training of the region detection model are performed using YOLOv11 or a lightweight model of equivalent precision. Training objective: Only used for coarse localization of the chip's position and approximate boundary, and robust to changes in illumination.

[0043] like Figure 1 As shown, in step S4, the chip edge is precisely located within the circumscribed rectangle to obtain the precise area of ​​the chip. In this embodiment, the precise positioning uses a geometric caliper algorithm to accurately extract the four sides of the chip. Specifically: Multiple one-dimensional profile lines are projected along the direction perpendicular to the edge, and gray-level jump points (gradient extrema points) are detected on each profile line.

[0044] For each profile line : ; The edge points are: ; Collect all edge point sets .

[0045] For the set of edge points Perform linear fitting: Equation of the straight line: ; Least squares solution: ; Determine the precise boundaries of the four sides of the chip The precise chip region is obtained by finding the intersection points of the four sides. The advantages of precise positioning are as follows: sub-pixel accuracy; unaffected by texture interference from fused images; stable and reliable edge results.

[0046] like Figure 1 , 4 As shown, in S5, the precise region is divided into multiple overlapping grid image blocks. In this embodiment, the grid size and overlap rate of the grid image blocks are configurable to ensure that there are overlapping areas between adjacent grid image blocks, avoiding the omission of cross-boundary defects.

[0047] Let the chip area size be The grid size is defined as The overlap rate is .

[0048] The grid step size is calculated as follows: Horizontal step size: ; Vertical step size: ; Calculation of the coordinates of the top left corner of the grid ; ; Grid area: ; This approach ensures complete coverage of all regions, guarantees that no small defects are missed in overlapping areas, and facilitates high-resolution inference of small targets by large models.

[0049] like Figure 1 , 4 As shown, in S6, defect detection is performed on each grid image patch to obtain local defect detection results. Defects such as scratches, chipped edges, contamination, foreign objects, and surface pits are labeled in the gridded image. YOLOv11 or other instance segmentation / small object enhancement models are used for training to improve small object perception, texture enhancement recognition, and multi-scale feature fusion capabilities. Defect detection model inference is performed on each grid image separately to obtain local defect detection results. ; Each defect includes its category, location (relative to the grid), and confidence level.

[0050] like Figure 1 , 5 As shown, in S7, all local defect detection results are mapped back to the original image coordinate system, and the chip surface defect information is output after merging. Specifically, local defects are mapped back to the original image based on the coordinates of the upper left corner of the grid, with the grid offset set to... The defect bounding box within the mesh is: ; The original frame after mapping: ; All grid inference results are merged using nonmaximum suppression (NMS).

[0051] Furthermore, the chip surface defect information includes one or more combinations of defect type, defect location, defect quantity, and defect visualization contour map.

[0052] As described above, a chip surface defect detection method based on deep learning according to an embodiment of the present invention achieves high-precision and robust automatic detection of chip surface defects by employing bright-field and dark-field image fusion to enhance texture information, combining coarse chip region localization with precise edge localization of traditional algorithms, and multi-scale mesh cropping to improve defect detection accuracy. This reduces manual intervention and improves the configuration efficiency and detection capability of the AOI system.

[0053] The above combined with the appendix Figures 1-5 A deep learning-based chip surface defect detection method according to an embodiment of the present invention is described. Furthermore, the present invention can also be applied to a deep learning-based chip surface defect detection system.

[0054] like Figure 6 As shown, according to a second aspect of the present invention, a chip surface defect detection system based on deep learning is provided, comprising: The acquisition module 100 is used to acquire the bright field image and dark field image of the chip; The fusion module 200 is used to fuse the bright-field image and the dark-field image to generate a fused image; The coarse positioning module 300 is used to coarsely locate the chip region in the fused image and obtain the bounding rectangle of the chip region. The precision positioning module 400 is used to accurately locate the chip edge within an outer rectangular frame to obtain the precise area of ​​the chip. The region division module 500 is used to divide a precise region into multiple overlapping grid image blocks; The detection module 600 is used to perform defect detection on each grid image block and obtain local defect detection results; The output module 700 is used to map all local defect detection results back to the original coordinate system and output the chip surface defect information after merging.

[0055] The above-described chip surface defect detection system based on deep learning is the system structure corresponding to the first aspect of the chip surface defect detection method based on deep learning. Therefore, the details will not be elaborated further, as this is fully clear to those skilled in the art.

[0056] The above combined with the appendix Figure 6 A deep learning-based chip surface defect detection system according to an embodiment of the present invention is described. Furthermore, the present invention can also be applied to a computer-readable medium having processor-executable non-volatile program code.

[0057] According to a third aspect of the present invention, a computer-readable medium having processor-executable non-volatile program code is provided, the program code causing the processor to perform a deep learning-based chip surface defect detection method according to the first aspect.

[0058] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of a computer program from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the ASIC can reside within a device. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. The present invention also provides a program product comprising executable instructions stored in the readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the deep learning-based chip surface defect detection method provided in the various embodiments described above. In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0059] It should be noted that, in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0060] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A chip surface defect detection method based on deep learning, characterized in that, It includes the following steps: Acquire bright-field and dark-field images of the chip; The bright-field image and the dark-field image are fused to generate a fused image; Coarsely locate the chip region in the fused image to obtain the bounding rectangle of the chip region; Within the circumscribed rectangular frame, the chip edge is precisely located to obtain the precise area of ​​the chip; The precise region is divided into multiple overlapping grid image blocks; Defect detection is performed on each grid image block to obtain local defect detection results; Map all local defect detection results back to the original coordinate system, merge them, and output the chip surface defect information.

2. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, The process of fusing the bright field image and the dark field image to generate a fused image involves using the bright field image as the first channel, the dark field image as the second channel, and the difference image between the bright field and dark field images as the third channel, and combining them to generate a three-channel fused image.

3. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, The coarse localization of the chip region in the fused image is specifically performed by using a trained deep learning object detection model to detect the chip region in the fused image and outputting the bounding rectangle of the chip; the deep learning object detection model is robust to changes in illumination after training.

4. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, Within the circumscribed rectangular frame, the chip edge is precisely located to obtain the chip's precise region: Within the outer rectangular frame, multiple one-dimensional profile lines are projected vertically along the chip edge. Gray-level jump points on each profile line are detected, and the point with the maximum gray-level gradient is taken as the edge point to form an edge point set. The least squares method is used to fit a straight line to the set of edge points to obtain the precise four edges of the chip. The precise region of the chip is obtained through the intersection of the four edges.

5. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, The grid size and overlap rate of the grid image blocks are configurable, ensuring that there are overlapping areas between adjacent grid image blocks.

6. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, Defect detection is performed on each grid image block to obtain local defect detection results. Specifically, a trained deep learning object detection model is used to identify defects in each grid image block and output the defect category, location, and confidence level.

7. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, Map all local defect detection results back to the original image coordinate system and merge them as follows: Based on the position of the mesh in the original image, convert the defect coordinates within the mesh to the coordinates in the original image; Non-maximum suppression is applied to the detection results of all grids, and overlapping detection boxes are merged.

8. The chip surface defect detection method based on deep learning as described in claim 1, characterized in that, The chip surface defect information includes one or more combinations of defect type, defect location, defect quantity, and defect visualization contour map.

9. A chip surface defect detection system based on deep learning, characterized in that, Include: The acquisition module is used to acquire bright-field and dark-field images of the chip; The fusion module is used to fuse the bright-field image and the dark-field image to generate a fused image; The coarse localization module is used to coarsely locate the chip region in the fused image and obtain the bounding rectangle of the chip region. The precision positioning module is used to accurately position the chip edge within the outer rectangular frame to obtain the precise area of ​​the chip. The region segmentation module is used to divide the precise region into multiple overlapping grid image blocks; The detection module is used to perform defect detection on each grid image block and obtain local defect detection results; The output module is used to map all local defect detection results back to the original coordinate system and output the chip surface defect information after merging.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to run the deep learning-based chip surface defect detection method according to any one of claims 1-8.

Images