Chip defect visual detection method and system and storage medium

Abstract

The invention discloses a chip defect visual detection method and system and a storage medium, and the method comprises the steps: obtaining and preprocessing an original image of a chip, and obtaining a preprocessed image; dividing the preprocessed image into a plurality of equal-size image sub-blocks, and calculating feature vectors of the image sub-blocks to obtain feature vectors of the plurality of sub-blocks; inputting the feature vectors of the plurality of sub-blocks into a defect detection model to obtain sub-block categories; wherein the defect detection model is constructed based on a deep learning algorithm, and the sub-block category comprises a normal category and a defect category. The invention relates to the technical field of visual detection, and solves the technical problem that a traditional detection method is easy to cause missing detection of small defects.

Description

Technical Field

[0001] The present invention belongs to the field of chip quality inspection and relates to computer vision inspection technology, specifically a chip defect visual inspection method, system and storage medium. Background Art

[0002] Existing chip defect visual inspection technology faces multiple technical bottlenecks in industrial applications. In industrial scenarios, chip images are often affected by background factors such as substrate texture and uneven lighting. Traditional inspection methods lack a precise mechanism for segmenting the background and chip regions. During global inspection, defect signals are easily overwhelmed by noise, resulting in reduced detection reliability. At the feature analysis level, most methods rely on global features or single manually designed features, making it difficult to capture texture mutations and edge anomalies in local areas of the chip. They are also insufficiently sensitive to subtle defects, resulting in insufficient chip defect detection accuracy and prone to missed and false detections. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a chip defect visual detection method, system and storage medium to solve the technical problem that traditional detection methods are prone to miss detection of small defects.

[0004] To achieve the above objectives, a first aspect of the present invention provides a chip defect visual detection method, comprising:

[0005] Acquire the original image of the chip and preprocess it to obtain a preprocessed image;

[0006] Divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the eigenvectors of the image sub-blocks to obtain the eigenvectors of several sub-blocks;

[0007] The feature vectors of several sub-blocks are input into a defect detection model to obtain sub-block categories; wherein the defect detection model is constructed based on a deep learning algorithm, and the sub-block categories include normal categories and defect categories.

[0008] It should be noted that the sub-block category is used to indicate whether the current sub-block has a defect. If so, the category of the current sub-block is a defective category, otherwise it is a normal category.

[0009] Furthermore, the feature vector of the image sub-block is calculated based on the grayscale variance and gradient average of the sub-block, and the formula is: Among them, σ 2 represents the sub-block grayscale variance, and They represent the gradient values ​​of the image sub-block in the x and y directions at point (i, j), a×b represents the size of the image sub-block, and a and b represent the length and width of the image sub-block, respectively.

[0010] Furthermore, the training process of the defect detection model includes a first pre-training and a second pre-training; wherein,

[0011] The input of the first pre-training is the pre-processed historical chip image, and the output is the pixel-level background area or chip area, which is used to generate a chip area mask to separate the chip area from the background area;

[0012] The input of the second pre-training is several sub-block feature vectors of the pre-processed and divided historical chip image, and the output is the chip area or defect area of ​​the sub-block, which is used to identify defect features in the chip area.

[0013] Furthermore, the preprocessing includes but is not limited to grayscale, noise reduction processing using Gaussian filtering or median filtering, and histogram equalization.

[0014] Furthermore, the first pre-training includes:

[0015] Collect several historical chip images and perform pixel-level annotation on the background and chip regions in the historical chip images to obtain training and validation datasets. The annotation label of the background region is 0, and the annotation label of the chip region is 1. Both the training and validation datasets contain historical chip images and corresponding annotation labels.

[0016] Constructing a first initial model based on a deep learning algorithm, inputting a training data set into the first initial model for iterative training, outputting a pixel-level classification probability map, and obtaining a prediction result; wherein the first initial model adopts an encoder-decoder architecture;

[0017] Optimize the parameters of the first initial model through back propagation and minimize the loss between the predicted results and the labeled labels to obtain the first model after iterative training;

[0018] The evaluation index of the first model is calculated using the validation data set. When the evaluation index reaches a preset first threshold, the iterative training is completed to obtain a first pre-trained model; wherein the evaluation index of the first model includes but is not limited to intersection-over-union ratio, precision, accuracy, and Dice coefficient.

[0019] Furthermore, the second pre-training includes:

[0020] Extracting chip regions from historical chip images using the first pre-trained model to obtain a number of chip regions;

[0021] Divide the chip area into several equal-sized sub-blocks, and calculate the feature vector of each sub-block to obtain several historical feature vectors;

[0022] Based on whether several chip regions have defects, they are categorized and labeled to construct sub-block-level training and validation datasets. Normal sub-blocks are labeled with a value of 2, while defective sub-blocks are labeled with a value of 3. Both sub-block-level training and validation datasets contain historical feature vectors and corresponding labeling.

[0023] Reusing the encoder part of the first pre-trained model, freezing the parameters of the first several layers, and adding a sub-block feature extraction branch after the encoder to obtain a second initial model;

[0024] Input the sub-block level training data set into the second initial model for iterative training, and output the predicted label value of the sub-block;

[0025] Optimize the parameters of the second initial model through back propagation and minimize the loss between the predicted label value and the labeled label of the sub-block to obtain the second model after iterative training;

[0026] The evaluation index of the second model is calculated using the validation data set. When the evaluation index reaches a preset second threshold, the iterative training is completed to obtain a defect detection model; wherein the evaluation index of the second model includes but is not limited to Accuracy, Precision, Recall, and F1-Score.

[0027] Furthermore, the sub-block feature extraction branch includes at least one global average pooling layer, at least one convolutional layer and at least one fully connected layer, which is used to process the feature map output by the encoder, wherein,

[0028] The global average pooling layer is used to compress the feature map output by the encoder into spatial dimensions to obtain channel-level statistical features;

[0029] The convolution layer is used to perform convolution processing on the channel-level statistical features through a convolution operation to obtain refined features;

[0030] The fully connected layer is used to map the refined features output by the convolutional layer to the classification space, and output the probability distribution of whether the sub-block belongs to a normal sub-block or a defective sub-block through the Softmax activation function.

[0031] A second aspect of the present invention provides a chip defect visual inspection system, comprising:

[0032] Data preprocessing module: used to obtain the original image of the chip and preprocess it to obtain the preprocessed image;

[0033] Image division module: used to divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the feature vectors of the image sub-blocks to obtain the feature vectors of several sub-blocks;

[0034] Defect detection module: used to input the feature vectors of several sub-blocks into the defect detection model to obtain sub-block categories; wherein, the defect detection model is constructed based on a deep learning algorithm, and the sub-block categories include normal categories and defect categories.

[0035] A third aspect of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, a chip defect visual detection method as described in the first aspect above is implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This method uses a first pre-trained model to achieve semantic segmentation of the chip area and background, eliminating background interference such as substrate texture and uneven lighting. It precisely locks the detection range to the main chip area, avoiding the problem of defect signals being obscured by background noise during global detection. Furthermore, through sub-block partitioning and local feature extraction, the chip area is decomposed into microscopic units. Features such as grayscale variance and gradient mean are used to capture subtle differences in defects, improving the accuracy and efficiency of chip defect detection.

[0038] In terms of model training and architecture design, this invention optimizes the performance and efficiency of deep learning models through a phased training strategy and parameter reuse mechanism. The first pre-training stage only needs to distinguish between background and chip, without relying on scarce defect samples, and can quickly converge using a large amount of defect-free data. The second pre-training stage reuses the encoder features of the first stage and, by adding a sub-block feature extraction branch, reduces the number of parameters while enhancing the abstraction ability of defect features. This reduces the complexity of the model training process and improves the model's generalization ability for complex defect patterns through feature hierarchical learning. It is particularly suitable for industrial detection scenarios where defect samples are scarce and the categories are unbalanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A schematic diagram of the framework of a chip defect visual inspection system provided by the present invention;

[0041] Figure 2 A schematic diagram of the process of a chip defect visual detection method provided by the present invention;

[0042] Figure 3A schematic diagram of the process of training and applying the defect detection model provided by the present invention;

[0043] Figure 4 A schematic structural diagram of a chip defect visual inspection device provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the hardware structure of a chip defect visual detection device provided in an embodiment of the present application; DETAILED DESCRIPTION

[0045] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] The chip defect visual detection method provided in the embodiment of the present application can be applied to Figure 1 In a chip defect visual inspection system shown in FIG. Figure 1 As shown, the communication system includes: a data preprocessing module, an image segmentation module and a defect detection module.

[0047] The data preprocessing module is used to obtain the original image of the chip and preprocess it to obtain the preprocessed image;

[0048] Image division module: used to divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the feature vectors of the image sub-blocks to obtain the feature vectors of several sub-blocks;

[0049] Defect detection module: used to input the feature vectors of several sub-blocks into the defect detection model built based on the deep learning algorithm to obtain sub-block categories, and the sub-block categories include normal categories and defect categories.

[0050] Furthermore, the data preprocessing module, the image segmentation module and the defect detection module are communicatively connected.

[0051] To solve the technical problems of low efficiency and insufficient accuracy in chip defect visual detection in the prior art, the present invention provides a chip defect visual detection method, which includes:

[0052] Acquire the original image of the chip and preprocess it to obtain a preprocessed image;

[0053] Divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the eigenvectors of the image sub-blocks to obtain the eigenvectors of several sub-blocks;

[0054] The feature vectors of several sub-blocks are input into a defect detection model to obtain defect classification of the sub-blocks; wherein the defect detection model is constructed based on a deep learning algorithm, and the defect classification includes a normal category and a defect category.

[0055] Based on this, efficient and accurate detection of chip defects can be achieved.

[0056] like Figure 2 As shown, an embodiment of the present application provides a chip defect visual detection method, comprising:

[0057] S1. Obtain the original image of the chip and preprocess it to obtain a preprocessed image.

[0058] The original image is a chip image taken by an image acquisition device (such as an industrial camera).

[0059] In some implementations, pre-processing operations may include image noise reduction, contrast enhancement, edge sharpening, etc.

[0060] It should be noted that the purpose of preprocessing is to eliminate noise interference in the original image and improve image quality for subsequent processing. For example, a median filter algorithm can be used to reduce noise in the original image, or a histogram equalization method can be used to enhance image contrast.

[0061] S2. Divide the preprocessed image into a plurality of image sub-blocks of equal size, and calculate the feature vectors of the image sub-blocks to obtain feature vectors of the plurality of sub-blocks.

[0062] The image sub-blocks are small image blocks obtained by evenly dividing the pre-processed image according to a preset size.

[0063] In some implementations, the size of the image sub-block may be set to a×b, such as 8×8, 16×16, etc., in pixels, according to the chip size and defect detection accuracy requirements.

[0064] It should be pointed out that the feature vector is used to characterize the texture and structural information of the image sub-block, and can be calculated based on grayscale statistical features, texture features, gradient features, frequency domain features, etc., or the image sub-block can be directly input into the deep learning algorithm model, and the network model can be used to extract the feature vector of the image and perform defect detection. It can also be calculated based on the grayscale variance and gradient average of the sub-block as in the present invention.

[0065] Among them, grayscale statistical features include grayscale mean, grayscale variance, grayscale entropy, etc., which are used to describe the distribution characteristics of pixel grayscale within a sub-block. For example, the mean reflects the brightness level, and the variance reflects the degree of pixel dispersion.

[0066] Texture features: including gray-level co-occurrence matrix GLCM, local binary pattern LBP, etc., used to capture the spatial relationship between pixels and texture patterns, such as roughness and directionality.

[0067] Gradient features: including Histogram of Oriented Gradients (HOG) and edge density, used to highlight edge and contour information in images, such as calculating gradient values ​​through the Sobel operator and Prewitt operator.

[0068] Frequency domain features: This includes converting the image into the frequency domain through Fourier transform, wavelet transform, etc., and extracting the statistical values ​​of high-frequency components (corresponding to details and defects) or low-frequency components (corresponding to the overall structure).

[0069] For example, when detecting edge defects such as chip surface scratches, gradient features are preferred; when detecting texture defects, LBP or GLCM features are used; and when detecting brightness defects, a combination of grayscale mean and variance features is used. The dimension of the feature vector can be set according to actual needs. For example, the above features can be concatenated to form a multidimensional vector (such as grayscale mean, grayscale variance, and HOG feature vector) to improve the comprehensiveness of defect characterization.

[0070] S3. Input the feature vectors of several sub-blocks into the defect detection model to obtain the defect classification of the sub-blocks.

[0071] Among them, the defect detection model is a model built based on a deep learning algorithm, which is used to analyze and classify the feature vectors of sub-blocks.

[0072] In some implementations, the defect detection model can use deep learning architectures such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Defect classification typically includes a normal category and a defect category, indicating whether the chip area corresponding to the sub-block has a defect.

[0073] For example, after the model is trained, the Softmax activation function can be used to output the probability distribution of whether the sub-block belongs to the normal category or the defect category, and the final classification result can be determined based on a preset threshold.

[0074] Based on the above technical solution, the chip defect visual detection method provided in this application realizes automated and high-precision detection of chip defects by preprocessing the chip image, dividing it into sub-blocks and extracting feature vectors, combined with a defect detection model based on deep learning, effectively improving the detection efficiency and accuracy, and reducing the cost and error of manual inspection.

[0075] In a possible implementation of the embodiment of the present application, the above S1 can be specifically implemented by the following S101, S102 and S103, which are specifically described below:

[0076] S101, obtaining an original image of the chip.

[0077] Among them, the original image is the chip surface image taken by image acquisition devices such as industrial cameras and line array cameras. The image format can be RGB or grayscale, and the resolution must meet the defect detection accuracy requirements (such as ≥5 million pixels).

[0078] In some implementations, a camera mounted on an automated robotic arm can be used to dynamically capture chips on a conveyor belt, or a camera at a fixed station can be used to statically capture stationary chips.

[0079] It is important to note that the image acquisition process requires controlled illumination uniformity. Ringlights, backlights, and other lighting solutions can be used to reduce reflections and shadows. For example, for highly reflective chip surfaces, a polarized light source combined with a diffuse reflector can be used to obtain a clear image.

[0080] S102: grayscale the original image.

[0081] Grayscaling converts an RGB three-channel image into a single-channel grayscale image to reduce computational complexity and highlight brightness features. For example, a weighted summation of the three-channel channels can be performed to obtain a single-channel grayscale image, or the maximum value method can be used to convert the grayscale value by taking the maximum value of the three channels.

[0082] It should be pointed out that grayscale processing does not change the texture and edge information of the image, but can simplify subsequent feature calculations.

[0083] For example, if the original image is a color chip image, grayscale conversion can be quickly achieved through the cvtColor function of the OpenCV library.

[0084] S103: performing noise reduction and contrast enhancement on the grayscale image.

[0085] Among them, noise reduction processing can use Gaussian filtering or median filtering to suppress salt and pepper noise or Gaussian noise; contrast enhancement can use histogram equalization to improve the brightness and darkness levels of the image.

[0086] In some implementations, the kernel size of the Gaussian filter can be set to 3×3 or 5×5, and the standard deviation is adjusted according to the noise intensity; the median filter is suitable for salt and pepper noise, and the kernel size is usually an odd number, such as 3×3 or 5×5.

[0087] It should be pointed out that histogram equalization can expand the image grayscale range to the full dynamic range (0-255) by redistributing pixel grayscale values, thereby improving the contrast between the defect area and the background area.

[0088] For example, the GaussianBlur function of OpenCV can be used directly to perform Gaussian noise reduction, and the equalizeHist function can be used to implement histogram equalization.

[0089] In a possible implementation of the embodiment of the present application, combined with Figure 2 The above S2 can be implemented by the following S201, S202 and S203, which are described in detail below:

[0090] S201: Divide the preprocessed image into a plurality of image sub-blocks of equal size.

[0091] The size of the image sub-block a×b is set according to the chip size and defect detection accuracy, such as 8×8 and 16×16 pixels, and the division method is a non-overlapping sliding window, that is, the step size is equal to the sub-block size.

[0092] In some implementations, if defects are likely to occur in the chip edge area, overlapping partitioning may be used, such as with a step size of 50% of the sub-block size, to ensure complete extraction of edge features.

[0093] It should be pointed out that before division, the chip area mask generated by the first pre-training model is used to only divide the chip area into sub-blocks to exclude interference from the background area.

[0094] For example, assuming that the size of the pre-processed image is 256×256 pixels and the sub-block size is set to 16×16, it can be divided into 16×16 image sub-blocks.

[0095] S202, calculate the grayscale variance σ of each image sub-block 2 .

[0096] Among them, the grayscale variance is used to measure the discrete degree of pixel grayscale within a sub-block, and the calculation formula is: Among them, μ represents the grayscale mean of the sub-block pixels, and f(i,j) represents the grayscale value of point (i,j).

[0097] In some implementations, the variance can be quickly calculated through matrix operations, such as first calculating the mean matrix, then calculating the sum of the squares of the differences between the pixel values ​​and the mean, and finally finding the average.

[0098] It should be pointed out that the larger the grayscale variance, the greater the grayscale difference of pixels within the sub-block, which may indicate defects such as scratches and stains.

[0099] S203: Calculate the average gradient of each image sub-block.

[0100] The gradient average is calculated by the gradient values ​​in the x and y directions, and the formula is: in, and It can be calculated using gradient operators such as the Sobel operator and the Prewitt operator.

[0101] It should be pointed out that the gradient average reflects the edge density within the sub-block, and defect areas, such as cracks and pin deformation, usually have higher gradient values.

[0102] For example, the gradient magnitude is calculated using the Sobel function of OpenCV, and then the average value within the sub-block is calculated as the gradient feature.

[0103] S204: Concatenate the grayscale variance and the gradient average to obtain a feature vector of the sub-block.

[0104] In a possible implementation of the embodiment of the present application, combined with Figure 2 ,like Figure 3 As shown, the above S3 can be specifically implemented through the following S301, S302, S303 and S304, which are specifically described below:

[0105] S301. Construct a training data set for a defect detection model.

[0106] The training dataset includes historical chip images and their annotation information, which is divided into two stages:

[0107] Semantic segmentation stage, i.e. the first pre-training stage: use the LabelMe tool to label the background area (label 0) and chip area (label 1) in the chip image to obtain the label image corresponding to each chip image.

[0108] The classification stage, also known as the second pre-training stage, is as follows: within the chip area, depending on whether the sub-block has defects, the sub-block file name is labeled with a unique identifier containing the image ID, sub-block ID, and label value. The specific format is "image ID_sub-block ID_label value", and the label value corresponds to the normal category (label value is 2) or the defect category (label value is 3).

[0109] Among them, the image ID is the unique identifier of the original chip image, which can be in the form of "acquisition timestamp" (such as "20250519_143002"), "device number + serial number" (such as "CAM01_000123"), or "chip batch + location code" (such as "LOT202505_AB12"), which is used to trace the source of the original image corresponding to the sub-block; the sub-block ID is the coordinate index of the sub-block in the original image, which can be expressed as "row number_column number" (such as "row5_col3") or a continuous integer (such as "block0015"), which is used to locate the position of the sub-block in the image; the label value directly corresponds to the category.

[0110] It's important to note that file names containing image IDs enable three-level traceability: sub-block, image, and actual chip. This is particularly useful in industrial inspection scenarios involving multi-batch, multi-device acquisition. For example, if a defect is detected in a sub-block, the image ID can be used to quickly locate the original acquisition device, acquisition time, and corresponding chip batch. This can then be combined with production logs to analyze the process link that caused the defect.

[0111] For example, if the chip image ID is "CHIP20250519A001" and its sub-block is located at row 10, column 8 in the image, and it is detected as a normal sub-block, the file name is "CHIP20250519A001_10_8_2.png." If it is a defective sub-block, the file name is "CHIP20250519A001_10_8_3.png." When loading data, the file name can be parsed using a regular expression (such as (\w+)_(\d+_\d+)_(\d+)\.\w+$) to extract the image ID, sub-block coordinates, and label values, respectively, to ensure the correlation between the training data and the original image.

[0112] In this embodiment, the first pre-training can quickly converge using a large number of defect-free chip images, and the second pre-training can complete fine-tuning with a small number of defective samples.

[0113] It should be pointed out that data enhancement techniques (such as rotation, scaling, and noise addition) can be used to expand the training data set and improve the generalization ability of the model.

[0114] S302: Train the defect detection model in stages.

[0115] Among them, model training is divided into two stages:

[0116] The first pre-training stage: adopt an encoder-decoder architecture, input preprocessed historical images, output pixel-level classification probability maps, optimize parameters through back-propagation, minimize cross-entropy loss, and evaluate indicators such as intersection over union (IoU) and Dice coefficient. When the indicator reaches the preset threshold (such as IoU ≥ 0.9), training is stopped to obtain the chip area segmentation model, which is the first pre-training model.

[0117] Second pre-training stage: reuse the encoder of the first pre-training, freeze the parameters of the first N layers (such as the first 10 layers), add a sub-block feature extraction branch (including global average pooling layer, convolution layer, and fully connected layer), input the sub-block feature vector after calculation, output the classification probability, use the Softmax activation function and cross entropy loss, and evaluate the indicators including accuracy, F1 score, etc. The training is completed when the indicator reaches the preset first threshold (such as accuracy ≥ 0.95).

[0118] In some implementations, the encoder may adopt a classic network architecture such as ResNet and VGG, and the number of convolutional layers of the feature extraction branch is set according to the feature complexity (e.g., 1-3 layers).

[0119] It should be pointed out that the phased training strategy can reduce the dependence on defective samples and improve training efficiency.

[0120] S303: Input the sub-block feature vector into the trained defect detection model.

[0121] Among them, the model input is a two-dimensional feature vector F = [σ 2 , GradAvg], or concatenate other features, such as grayscale mean and texture features, to form a multi-dimensional vector.

[0122] In some implementations, the feature vector needs to be normalized before input, such as to the range [-1, 1] or [0, 1], to avoid the influence of numerical range differences on model convergence.

[0123] It should be pointed out that the model compresses the spatial dimension of the feature map through the global average pooling layer, and then maps it to the classification space through the fully connected layer, and outputs the probability distribution of whether the sub-block belongs to the normal category or the defect category.

[0124] For example, the feature vector is standardized using Scikit-learn's StandardScaler, and the classification probability is output through the model's predict interface.

[0125] The sub-block category is determined based on the probability distribution. A second threshold value, such as 0.5, is preset to determine the sub-block category: if the normal class probability ≥ the threshold value, the sub-block is determined to be a normal sub-block; otherwise, the sub-block is determined to be a defective sub-block.

[0126] In some implementations, dynamic threshold adjustment can be used, such as setting a weighted threshold based on the category distribution of the training set, to alleviate the category imbalance problem.

[0127] It's important to note that the algorithmic code for calculating the sub-block feature vector and determining the sub-block category using a probability threshold can be included in the defect detection model. That is, the defect detection model can directly take the sub-block image as input, and output the predicted sub-block category. Alternatively, the defect detection model can take the sub-block feature vector as input, and output the predicted sub-block category.

[0128] In one possible implementation, combining Figure 2 After S3, the chip defect visual detection method provided in the embodiment of the present application further includes the following S401 to S403:

[0129] S401: Generate an overall chip defect report based on the sub-block defect classification results.

[0130] The defect report contains information such as the number of defective sub-blocks, location coordinates (such as row number, column number), confidence score, etc., which can be displayed through a visual interface.

[0131] In some implementations, the defect sub-blocks can be marked with rectangular boxes on the original image and superimposed on the inspection result image. For example, the Matplotlib library can be used to draw the defect area box on the pre-processed image to generate an annotated inspection result image.

[0132] It should be noted that the report can be exported in formats such as CSV and JSON to facilitate quality traceability and statistical analysis.

[0133] S402: Locate the physical location of the chip corresponding to the defective sub-block.

[0134] Among them, the sub-block coordinates are converted into the physical coordinates (x, y) of the chip surface through the mapping relationship between the image acquisition coordinate system and the chip physical coordinate system (such as pixel size converted into millimeters).

[0135] In some implementations, the defect location can be automatically located in conjunction with the robotic arm motion coordinate system for subsequent manual review or automatic repair.

[0136] Exemplarily, the specific operations of converting the sub-block coordinates into the physical coordinates (x, y) of the chip surface may include:

[0137] First, coordinate system calibration is performed. A calibration plate of known physical size (such as a checkerboard calibration plate) is placed at the image acquisition station. After capturing the calibration image, the visual library is used to calculate the conversion relationship between the pixel coordinate system and the physical coordinate system to obtain the pixel size factor K (mm / pixel). Next, the pixel coordinates of the sub-block in the original image are obtained. When dividing the sub-blocks, their indexes or corner coordinates are recorded, and the pixel coordinates of the sub-block center are calculated as the positioning reference point. The pixel coordinates are then converted into the physical coordinates of the chip. The pixel size factor K and the origin of the image coordinate system are used to convert the pixel coordinates of the sub-block center (u center , v center ) is converted into physical coordinates (x, y), and the coordinate axis direction is adjusted according to the chip's physical coordinate system; finally, the defect location is visualized and output, and a rectangular frame is drawn in the detection result image with sub-block pixel coordinates to mark the defect area. A report containing physical coordinates, dimensions, and confidence levels is generated. If automated processing is required, the physical coordinates are connected to the robotic arm coordinate system.

[0138] For example, assuming that the calibration plate grid side length L = 1 mm, and the pixel spacing between adjacent corner points is 100 pixels, we get K = 1 / 100 = 0.01 mm / pixel;

[0139] The sub-block size is 16×16 pixels and is located at the 5th row and 3rd column of the image (counting starts from 0). The coordinates of the upper left corner pixel are (3×16, 5×16) = (48, 80).

[0140] The coordinates of the sub-block center pixel are (48+8,80+8)=(56,88); the origin of the image coordinate system (u_0,v_0)=(0,0), corresponding to the upper left corner of the chip.

[0141] Physical coordinate calculation: X = 56 × 0.01 = 0.56 mm, Y = 88 × 0.01 = 0.88 mm

[0142] That is, the defect is located in the area 0.56 mm to the right and 0.88 mm downward from the upper left corner of the chip.

[0143] S403: Store the detection result and the original image in a computer-readable storage medium.

[0144] The storage media include hard disks, solid-state drives, optical disks, etc., and the storage contents include pre-processed images, sub-block feature vectors, model prediction results, defect reports, etc.

[0145] In some implementations, a database (such as MySQL) can be used to manage test records, supporting retrieval by time, chip model, and other criteria. For example, the Pandas library can be used to store test results in a CSV file, and the original image can be stored in PNG format, with the file naming convention being "chip number_test time.png".

[0146] Based on the above technical solution, this application realizes the automation of the entire process from image acquisition to result output through phased model training, multi-feature fusion and defect location storage, thereby improving the efficiency, accuracy and traceability of chip defect detection, and is particularly suitable for large-scale mass production detection scenarios of high-precision chips.

[0147] In the case of an integrated unit, Figure 4 A possible structural diagram of the chip defect visual detection device (denoted as device 40 ) involved in the above embodiment is provided. The device 40 includes a processing unit 401 and a communication unit 402 , and may also include a storage unit 403 . Figure 4 The structural schematic diagram shown can be used to illustrate the structure of the detection system involved in the above embodiments.

[0148] when Figure 4The structural schematic diagram shown is used to illustrate the structure of the chip defect visual detection device involved in the above-mentioned embodiment. The processing unit 401 is used to control and manage the operation of the device, the communication unit 402 is used for the device to communicate with other equipment (such as industrial cameras, robotic arms), and the storage unit 403 is used to store the program code and data of the device (such as image data, model parameters, and detection reports).

[0149] For example, the communication unit 402 is used to obtain the original image of the chip from the industrial camera and output the defect detection report and positioning results to the external device;

[0150] The processing unit 401 is used to perform image preprocessing (grayscale, Gaussian filtering, histogram equalization), divide the image into sub-blocks, calculate the sub-block feature vectors (grayscale variance and gradient average), and call the defect detection model for classification.

[0151] In a possible implementation, the processing unit 401 is further configured to perform two-stage training of the defect detection model: a first pre-training is to separate the chip from the background through semantic segmentation, and a second pre-training is to train a classification model based on sub-block feature vectors.

[0152] In one possible implementation, the communication unit 402 is further used to receive external calibration instructions and transmit calibration plate image data, and the processing unit 401 is further used to calculate the conversion factor between pixels and the physical coordinate system based on the calibration data to achieve physical coordinate mapping of the defect sub-block position.

[0153] The processing unit 401 may be a processor or controller (e.g., a CPU, GPU, or ASIC chip), and the communication unit 402 may be a communication interface, a transceiver, an input / output circuit, or the like. The storage unit 403 may be a memory (e.g., RAM, ROM, or flash memory). When the device 40 is a chip, the processing unit 401 may be a processor core or controller, the communication unit 402 may be a pin or a high-speed interface, and the storage unit 403 may be a register or cache within the chip.

[0154] Figure 4 If the integrated units in the embodiment of the present application are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (such as a server, an industrial control machine) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The storage medium for storing computer software products includes: various media that can store program codes, such as USB flash drives, mobile hard drives, solid-state drives, and optical disks.

[0155] The present application also provides a hardware structure diagram of a chip defect visual detection device (denoted as device 50), see Figure 5 The device 50 includes a processor 501 and, optionally, a memory 502 connected to the processor 501 .

[0156] In the first possible implementation, see Figure 5 The apparatus 50 further includes a transceiver 503. The processor 501, the memory 502, and the transceiver 503 are connected via a bus. The transceiver 503 is used to communicate with other devices or communication networks (e.g., to access a factory Internet of Things to transmit detection data). Optionally, the transceiver 503 may include a transmitter and a receiver, respectively, for receiving image data and transmitting detection results.

[0157] Based on the first possible implementation, Figure 5 The structural diagram shown can be used to illustrate the structure of the detection device involved in the above embodiments, wherein the processor 501 performs preprocessing, feature extraction and model inference, the memory 502 stores the operating system, detection algorithm and trained model, and the transceiver 503 realizes data interaction with external devices.

[0158] During implementation, each step of the method provided in this embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. For example, image preprocessing and feature calculation can be implemented by a software algorithm of the CPU, while deep learning model reasoning can be completed by a hardware acceleration unit of a GPU or FPGA to improve detection efficiency.

[0159] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, and other types of computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform operations or processing. The processor may be a separate semiconductor chip, or it may be integrated into a semiconductor chip together with other circuits. For example, it may form an SoC (system on a chip) with other circuits (such as a codec circuit, a hardware acceleration circuit, or various bus and interface circuits), or it may be integrated into the ASIC as a built-in processor of the ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the core for executing software instructions to perform operations or processing, the processor may further include necessary hardware accelerators, such as a field programmable gate array (FPGA), a PLD (programmable logic device), or a logic circuit that implements dedicated logic operations.

[0160] The memory in the embodiments of the present application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0161] An embodiment of the present application also provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute any of the above methods.

[0162] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above methods.

[0163] An embodiment of the present application also provides a chip, which includes a processor and an interface circuit, the interface circuit is coupled to the processor, the processor is used to run a computer program or instruction to implement the above method, and the interface circuit is used to communicate with other modules outside the chip.

[0164] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line digital subscriber line, abbreviated DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more media that can be integrated. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks, SSDs).

[0165] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0166] Working principle of the present invention:

[0167] First, the original chip image is acquired through an industrial camera. The image quality is improved through preprocessing operations such as grayscale conversion, Gaussian filtering for noise reduction, and histogram equalization. The preprocessed image is then divided into multiple equal-sized sub-blocks according to a preset size. The grayscale variance and gradient mean of each sub-block are calculated to generate a feature vector containing texture and edge information.

[0168] Classification is then performed using a deep learning-based defect detection model, which is trained in two stages. The first pre-training phase uses an encoder-decoder architecture to generate a chip region mask through semantic segmentation, separating the chip region from the background. The second pre-training phase reuses the encoder from the first phase, freezing some parameters and adding a sub-block feature extraction branch. The classification model is trained based on the sub-block feature vectors, outputting either a normal or defective category.

[0169] During detection, the sub-block feature vector is input into the model, and the defect category is determined according to the Softmax probability distribution to obtain the chip defect detection result.

[0170] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A chip defect visual detection method, characterized in that: include: Acquire the original image of the chip and preprocess it to obtain a preprocessed image; Divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the eigenvectors of the image sub-blocks to obtain the eigenvectors of several sub-blocks; The feature vectors of several sub-blocks are input into a defect detection model built based on a deep learning algorithm to obtain sub-block categories; the sub-block categories include normal categories and defect categories.

2. A chip defect visual detection method according to claim 1, characterized in that: The eigenvector of the image sub-block is calculated based on the grayscale variance and gradient average of the sub-block, and the formula is: Among them, σ 2 represents the sub-block grayscale variance, and They represent the gradient values ​​of the image sub-block in the x and y directions at point (i, j), a and b represent the length and width of the image sub-block respectively, and both a and b are integers.

3. The chip defect visual detection method according to claim 1, characterized in that: The training process of the defect detection model includes a first pre-training and a second pre-training; wherein, The input of the first pre-training is the pre-processed historical chip image, and the output is the pixel-level background area or chip area, which is used to generate a chip area mask to separate the chip area from the background area; The input of the second pre-training is several sub-block feature vectors of the pre-processed and divided historical chip image, and the output is the predicted category of the sub-block, which is used to identify the sub-block with defects in the chip area.

4. A chip defect visual detection method according to claim 3, characterized in that: The preprocessing includes at least one of grayscale conversion, noise reduction processing using Gaussian filtering or median filtering, and histogram equalization.

5. The chip defect visual detection method according to claim 3, characterized in that: The first pre-training includes: Collect several historical chip images, perform pixel-level annotation on the background area and chip area in the historical chip images, and obtain a training dataset and a validation dataset; wherein the training dataset and the validation dataset both contain historical chip images and corresponding annotation labels; Constructing a first initial model based on a deep learning algorithm, inputting a training data set into the first initial model for iterative training, outputting a pixel-level classification probability map, and obtaining a prediction result; wherein the first initial model adopts an encoder-decoder architecture; Optimize the parameters of the first initial model through back propagation and minimize the loss between the predicted results and the labeled labels to obtain the first model after iterative training; The evaluation index of the first model is calculated using the validation data set. When the evaluation index reaches a preset first threshold, the iterative training is completed to obtain a first pre-trained model; wherein the evaluation index of the first model includes at least one of the intersection over union (IoU), precision, accuracy, and Dice coefficient.

6. A chip defect visual detection method according to claim 5, characterized in that: The second pre-training includes: Extracting chip regions from historical chip images using the first pre-trained model to obtain a number of chip regions; Divide the chip area into several equal-sized sub-blocks, and calculate the feature vector of each sub-block to obtain several historical feature vectors; Based on whether there are defects in several chip areas, the categories are labeled and sub-block-level training and verification datasets are constructed. Both sub-block-level training and verification datasets contain historical feature vectors and corresponding annotation labels. Reusing the encoder part of the first pre-trained model, freezing the parameters of the first several layers, and adding a sub-block feature extraction branch after the encoder to obtain a second initial model; Input the sub-block level training data set into the second initial model for iterative training, and output the predicted label value of the sub-block; Optimize the parameters of the second initial model through back propagation and minimize the loss between the predicted label value and the labeled label of the sub-block to obtain the second model after iterative training; The evaluation index of the second model is calculated using the validation data set. When the evaluation index reaches a preset second threshold, the iterative training is completed to obtain a defect detection model; wherein the evaluation index of the second model includes at least one of accuracy, precision, recall, and F1 score.

7. A chip defect visual detection method according to claim 6, characterized in that: The sub-block feature extraction branch includes at least one global average pooling layer, at least one convolutional layer and at least one fully connected layer, which is used to process the feature map output by the encoder.

8. The chip defect visual detection method according to claim 7, characterized in that: The global average pooling layer is used to compress the feature map output by the encoder into spatial dimensions to obtain channel-level statistical features; The convolution layer is used to perform convolution processing on the channel-level statistical features through a convolution operation to obtain refined features; The fully connected layer is used to map the refined features output by the convolutional layer to the classification space, and output the probability distribution of whether the sub-block belongs to the normal category or the defect category through the Softmax activation function.

9. A chip defect visual detection system, applied to a chip defect visual detection method according to any one of claims 1 to 8, characterized in that: include: Data preprocessing module: used to obtain the original image of the chip and preprocess it to obtain the preprocessed image; Image division module: used to divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the feature vectors of the image sub-blocks to obtain the feature vectors of several sub-blocks; Defect detection module: used to input the feature vectors of several sub-blocks into the defect detection model built based on the deep learning algorithm to obtain sub-block categories; wherein the sub-block categories include normal categories and defect categories.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the chip defect visual detection method according to any one of claims 1 to 8 are implemented, specifically including: Acquire the original image of the chip and preprocess it to obtain a preprocessed image; Divide the preprocessed image into multiple image sub-blocks of equal size, and calculate the eigenvectors of the image sub-blocks to obtain the eigenvectors of several sub-blocks; The feature vectors of several sub-blocks are input into a defect detection model built based on a deep learning algorithm to obtain sub-block categories; the sub-block categories include normal categories and defect categories.

Images