An intelligent detection method and system for chip processing based on machine vision

Abstract

The present invention discloses an intelligent detection method and system for chip processing based on machine vision. By acquiring a standardized image of the chip and identifying and positioning the chip body and pins, a spatial positioning template is formulated; within a detection cycle, chip image data is acquired and preprocessed, and the body and pin regions are extracted using the spatial positioning template; based on a defect recognition module, defect information in these regions is recognized and recorded; the body region and the pin region are combined into a first chip analysis region, and secondary analysis is performed after chip packaging and processing to obtain a second chip analysis region and second defect information; based on the SSIM similarity index, the similarity index between the first and second chip analysis regions is calculated; the associated regions are judged according to the similarity index and a preset threshold, and are associated with the defect information to generate an associated feature set. Through the associated feature set, subsequent processing and classification of defective chips can be achieved, improving the information-based manufacturing detection ability of chips.

Description

Technical Field

[0001] The present invention relates to the field of chip processing analysis, and more specifically, to an intelligent detection method and system for chip processing based on machine vision. Background Art

[0002] With the continuous progress of integrated circuit manufacturing technology, the integration of chips is getting higher and higher, and the requirements for chip processing quality are also getting higher and higher. Traditional chip detection methods mainly rely on manual visual inspection or simple image processing techniques, which cannot meet the needs of modern high-precision chip manufacturing. Moreover, in the existing chip defect image detection, defects are often only detected in the chip processing finished products, ignoring the defect associations in processing steps such as welding and packaging, making it difficult to achieve full-process tracking of processing defects and effective defect review and evaluation, thus reducing the chip manufacturing quality. Summary of the Invention

[0003] The present invention overcomes the defects of the prior art and provides an intelligent detection method and system for chip processing based on machine vision.

[0004] The first aspect of the present invention provides an intelligent detection method for chip processing based on machine vision, including:

[0005] Extracting a standardized chip image from system data, identifying and positioning the chip body and pins in the standardized chip image, and formulating a spatial positioning template;

[0006] During a detection cycle, real-time acquiring chip image data before chip packaging and processing and performing image preprocessing, positioning the chip body and pins in the chip image data through the spatial positioning template, and extracting the main body area and the pin area through the positioning;

[0007] Based on the defect recognition module, identifying and recording chip defects in the main body area and the pin area to obtain the first defect information;

[0008] Combining the main body area and the pin area into a first chip analysis area, acquiring chip image data after chip packaging and processing and performing chip body and pin positioning analysis and defect recognition, and extracting the corresponding second chip analysis area and the second defect information;

[0009] Based on the SSIM similarity index, calculating the image information of the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. Through the brightness, contrast, and structure information, calculating the similarity between the first chip analysis area and the second chip analysis area. The calculation process sets a window based on the size of the pin area, performs local area comparison based on the set window, and obtains the similarity index of each window;

[0010] Based on multiple similarity indices and a preset similarity threshold, determine the main body area and pin area where the first chip analysis area and the second chip analysis area are associated, and mark them to obtain the associated area. Through the associated area, extract features from the chip image data before and after chip packaging and processing, and associate them with the first defect information and the second defect information to generate an associated feature set before and after processing.

[0011] In this solution, extract the standardized chip image from the system data, perform identification, positioning analysis on the chip main body and pins of the standardized chip image, and formulate a spatial positioning template. Specifically:

[0012] Obtain the standardized chip image from the system data;

[0013] Through region identification of the chip main body and pins on the standardized chip image, label and position the corresponding regions to obtain the positioning position information of the chip main body and pins;

[0014] Generate a spatial positioning template based on the image size, scale parameters of the standardized chip image and the positioning position information of the chip main body and pins.

[0015] In this solution, within a detection cycle, obtain the chip image data before chip packaging and processing in real time and perform image preprocessing. Use the spatial positioning template to position the chip main body and pins on the chip image data, and extract the main body area and pin area through positioning. Specifically:

[0016] Within a detection cycle, obtain the chip image data before chip packaging and processing;

[0017] Perform preprocessing such as denoising, grayscale conversion, and image enhancement on the chip image data;

[0018] Based on the spatial positioning template, perform standardization processing on the preprocessed chip image data through image scaling, and position the chip main body and pin areas through the region position information of the spatial positioning template to obtain the main body area and pin area.

[0019] In this solution, based on the defect recognition module, perform chip defect recognition on the main body area and pin area and record to obtain the first defect information. Specifically:

[0020] Construct a defect recognition module based on CNN;

[0021] Obtain historical defect features from the system database and import them into the defect recognition module for pre-training;

[0022] Based on the main body area and pin area, extract features of the corresponding areas from the chip image data before chip packaging and processing, and import the extracted features into the defect recognition module for defect recognition and recording to obtain the first defect information.

[0023] In this solution, the main area and the pin area are combined into the first chip analysis area. The chip image data after chip packaging and processing is obtained, and chip body and pin positioning analysis and defect identification are carried out. The corresponding second chip analysis area and second defect information are extracted, specifically as follows:

[0024] Based on combining the area information of the main area and the pin area into a large area, the first chip analysis area is formed;

[0025] Based on the spatial positioning template, the chip body and pins of the chip image data after chip packaging and processing are positioned, and the corresponding second chip analysis area is generated;

[0026] The chip image data after chip packaging and processing is imported into the defect recognition module for defect recognition and recording to obtain the second defect information.

[0027] In this solution, based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. Through the brightness, contrast, and structure information, the similarity between the first chip analysis area and the second chip analysis area is calculated. In the calculation process, a window is set according to the size of the pin area, and local area comparison is carried out based on the set window, and the similarity index of each window is obtained, specifically as follows:

[0028] Based on the first chip analysis area and the second chip analysis area, the image information extraction and loading of the chip image data before and after chip packaging and processing are respectively carried out to obtain the first image loading information and the second image loading information;

[0029] A window is set according to the size of the pin area through the SSIM similarity index calculation method;

[0030] Based on the window size, the local image information in the first image loading information and the second image loading information is calculated, and the local image information of each window is calculated by moving the window. The local image information includes brightness, contrast, and structure information;

[0031] Based on the brightness, contrast, and structure information of each window, the similarity index of each window is calculated in combination with the SSIM similarity index.

[0032] In this solution, based on multiple similarity indexes and a preset similarity threshold, the main area and pin area where the first chip analysis area and the second chip analysis area are associated are judged and marked to obtain the associated area. Through the associated area, the feature extraction of the chip image data before and after chip packaging and processing is carried out, and it is associated with the first defect information and the second defect information to generate the pre- and post-processing associated feature set, specifically as follows:

[0033] Compare the similarity index of each window with a preset similarity threshold, mark the windows higher than the preset similarity threshold, and obtain multiple marked windows;

[0034] Based on one of the marked windows, extract the corresponding local regions in the first chip analysis region and the second chip analysis region to obtain a first marked region and a second marked region;

[0035] Extract image features from the chip image data before chip packaging and processing based on the first marked region to obtain a first feature;

[0036] Extract image features from the chip image data after chip packaging and processing based on the second marked region to obtain a second feature;

[0037] Associate the first feature, the second feature, the first defect information, and the second defect information to form an associated data record;

[0038] Based on multiple marked windows, form multiple associated data records;

[0039] Integrate multiple associated data records to obtain an associated feature set before and after processing.

[0040] In this solution, the associated feature set before and after processing further includes:

[0041] Import the associated feature set before and after processing into the system database;

[0042] In the second detection cycle, obtain real-time chip image data;

[0043] Extract features and perform defect analysis on the real-time chip image data to form real-time features and real-time defect evaluation information;

[0044] Based on the real-time features and real-time defect evaluation information, perform associated feature and processing impact analysis from the associated feature set before and after processing to generate processing impact evaluation data;

[0045] Set the chip classification for the chips detected in real time through the processing impact evaluation data.

[0046] The second aspect of the present invention also provides an intelligent chip processing detection system based on machine vision. The system includes: a memory and a processor. The memory includes an intelligent chip processing detection program based on machine vision. When the intelligent chip processing detection program based on machine vision is executed by the processor, the following steps are implemented:

[0047] Extract the chip standard image from the system data, identify and position the chip body and pins in the chip standard image, and formulate a spatial positioning template;

[0048] During a detection period, chip image data before chip packaging and processing is obtained in real time and image preprocessing is performed. The chip body and pins are located through a spatial positioning template for the chip image data, and the main body area and pin area are extracted through positioning.

[0049] Based on a defect recognition module, chip defects in the main body area and pin area are recognized and recorded to obtain first defect information.

[0050] The main body area and pin area are combined into a first chip analysis area. Chip image data after chip packaging and processing is obtained and chip body and pin positioning analysis and defect recognition are performed, and the corresponding second chip analysis area and second defect information are extracted.

[0051] Based on the SSIM similarity index, image information calculation is performed on the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. Through brightness, contrast, and structure information, similarity calculation is performed on the first chip analysis area and the second chip analysis area. The calculation process sets a window based on the size of the pin area, and local area comparison is performed based on the set window, and the similarity index of each window is obtained.

[0052] Based on multiple similarity indexes and a preset similarity threshold, the main body area and pin area where the first chip analysis area and the second chip analysis area are associated are judged and marked to obtain an associated area. Through the associated area, feature extraction is performed on the chip image data before and after chip packaging and processing, and it is associated with the first defect information and the second defect information to generate an associated feature set before and after processing.

[0053] The third aspect of the present invention also provides a computer-readable storage medium, which includes an intelligent detection program for chip processing based on machine vision. When the intelligent detection program for chip processing based on machine vision is executed by a processor, the steps of the intelligent detection method for chip processing based on machine vision as described in any one of the above are realized.

[0054] The present invention discloses an intelligent detection method and system for chip processing based on machine vision. By acquiring a standardized image of the chip and identifying and positioning the chip body and pins, a spatial positioning template is formulated; within a detection cycle, chip image data is acquired and preprocessed, and the body and pin regions are extracted using the spatial positioning template; based on a defect recognition module, defect information in these regions is identified and recorded; the body region and the pin region are combined into a first chip analysis region, and secondary analysis is performed after chip packaging and processing to obtain a second chip analysis region and second defect information; based on the SSIM similarity index, the similarity index between the first and second chip analysis regions is calculated; the associated regions are judged according to the similarity index and a preset threshold, and are associated with the defect information to generate an associated feature set. Through the associated feature set, subsequent processing classification of defective chips can be achieved, improving the information-based manufacturing detection ability of chips. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 FIG. shows a flowchart of an intelligent detection method for chip processing based on machine vision according to the present invention;

[0056] Figure 2 FIG. shows a block diagram of an intelligent detection system for chip processing based on machine vision according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to more clearly understand the above objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0058] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0059] Figure 1 FIG. shows a flowchart of an intelligent detection method for chip processing based on machine vision according to the present invention.

[0060] As Figure 1 shown, in a first aspect of the present invention, an intelligent detection method for chip processing based on machine vision is provided, including:

[0061] S102. Extract a standardized image of the chip from the system data, perform identification and positioning analysis on the chip body and pins of the standardized image of the chip, and formulate a spatial positioning template;

[0062] S104. During a detection period, obtain the chip image data before chip packaging and processing in real time and perform image preprocessing. Use a spatial positioning template to locate the chip body and pins in the chip image data, and extract the main body area and pin area through positioning.

[0063] S106. Based on the defect recognition module, identify and record chip defects in the main body area and pin area to obtain the first defect information.

[0064] S108. Combine the main body area and pin area into the first chip analysis area, obtain the chip image data after chip packaging and processing, perform chip body and pin positioning analysis and defect recognition, and extract the corresponding second chip analysis area and second defect information.

[0065] S110. Based on the SSIM similarity index, calculate the image information of the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. Through the brightness, contrast, and structure information, calculate the similarity between the first chip analysis area and the second chip analysis area. The calculation process sets a window based on the size of the pin area, performs local area comparison based on the set window, and obtains the similarity index of each window.

[0066] S112. Based on multiple similarity indexes and a preset similarity threshold, judge the associated main body area and pin area between the first chip analysis area and the second chip analysis area and mark them to obtain the associated area. Through the associated area, extract the features of the chip image data before and after chip packaging and processing, and associate them with the first defect information and the second defect information to generate an associated feature set before and after processing.

[0067] According to the embodiments of the present invention, extracting the chip standardized image from the system data, identifying and positioning the chip body and pins in the chip standardized image, and formulating a spatial positioning template specifically include:

[0068] Obtain the chip standardized image from the system data;

[0069] Through region recognition of the chip body and pins in the chip standardized image, label and locate the corresponding regions to obtain the chip body and pin positioning position information;

[0070] Generate a spatial positioning template based on the image size, scale parameters of the chip standardized image, and the chip body and pin positioning position information.

[0071] It should be noted that the standardized chip image is a standardized comparison image, which can be based on user settings or system screening and has a better comparison effect. The chip body and pin positioning position information includes the chip body position, the positions of each pin, the main body and pin areas, and the relative position relationship information between the areas, etc. with respect to the entire standardized chip image. The spatial positioning template is a positioning template based on a certain image standardization, which is used to quickly locate the area of the chip image collected in real time. Before positioning, operations such as rotation and scaling need to be performed on the chip image collected in real time to make its image size, ratio parameters, etc. consistent with those of the standardized chip image. This process is a standardization process.

[0072] According to an embodiment of the present invention, within one detection cycle, chip image data before chip packaging and processing is acquired in real time and image preprocessing is performed. The chip body and pins are positioned for the chip image data through the spatial positioning template, and the main body area and pin area are extracted through positioning. Specifically:

[0073] Within one detection cycle, chip image data before chip packaging and processing is acquired;

[0074] Denoising, grayscale conversion, and image enhancement preprocessing are performed on the chip image data;

[0075] Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin areas are positioned through the area position information of the spatial positioning template to obtain the main body area and pin area.

[0076] According to an embodiment of the present invention, based on the defect recognition module, chip defects are recognized and recorded for the main body area and pin area to obtain the first defect information. Specifically:

[0077] A defect recognition module based on CNN is constructed;

[0078] Historical defect features are obtained from the system database and imported into the defect recognition module for pre-training;

[0079] Based on the main body area and pin area, feature extraction is performed on the chip image data before chip packaging and processing for the corresponding areas, and the extracted features are imported into the defect recognition module for defect recognition and recording to obtain the first defect information.

[0080] It should be noted that the first defect information is defect information before packaging, including information such as defect category and defect location. The first defect information is specifically defect information before chip packaging and processing, mainly judging the defects of pins. The main chip defects are generally raw material defects, foreign objects, scratches, printing defects, etc. Pin defects include pin bending, missing, etc.

[0081] According to an embodiment of the present invention, merging the main body area and the pin area into a first chip analysis area, obtaining chip image data after chip packaging and processing, performing chip main body and pin positioning analysis and defect identification, and extracting the corresponding second chip analysis area and second defect information, specifically:

[0082] Based on merging the area information of the main body area and the pin area into a large area to form a first chip analysis area;

[0083] Based on a spatial positioning template, performing row chip main body and pin positioning on the chip image data after chip packaging and processing and generating a corresponding second chip analysis area;

[0084] Importing the chip image data after chip packaging and processing into a defect identification module for defect identification and recording to obtain second defect information.

[0085] It should be noted that the purpose of forming the first chip analysis area is to extract the main chip image research through this area and propose irrelevant background areas. The analysis processes of the second chip analysis area and the second defect information are the same as those of the first chip analysis area and the first defect information. In addition, the second defect information is the defect identified for the chip after packaging and processing. There is a certain correlation between its defect categories and defect characteristics and the first defect information. Therefore, the present invention performs association mining of processing defects through subsequent similarity analysis, and analyzes and stores the defect feature association and defect type association in continuous processing steps.

[0086] According to an embodiment of the present invention, based on the SSIM similarity index, performing image information calculation on the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. Through brightness, contrast, and structure information, performing similarity calculation on the first chip analysis area and the second chip analysis area. The calculation process sets a window based on the size of the pin area, performs local area comparison based on the set window, and obtains the similarity index of each window, specifically:

[0087] Based on the first chip analysis area and the second chip analysis area, respectively performing image information extraction and loading of the analysis areas on the chip image data before and after chip packaging and processing to obtain first image loading information and second image loading information;

[0088] Setting a window based on the size of the pin area through the SSIM similarity index calculation method;

[0089] Based on the window size, calculating the local image information in the first image loading information and the second image loading information, and calculating the local image information of each window by moving the window. The local image information includes brightness, contrast, and structure information;

[0090] Based on the brightness, contrast, and structural information of each window, combined with the SSIM similarity index, the similarity index of each window is calculated.

[0091] It should be noted that the first chip analysis area is the corresponding area based on the chip image data before chip packaging and processing, and the second chip analysis area is the corresponding area based on the chip image data after chip packaging and processing. The sizes of the two analysis areas are the same. The window size is the same as the pin area size, for example, 10×10 pixels. In the similarity index, the number of similarity indices is the same as the number of windows. The brightness, contrast, and structural information respectively refer to the average pixel intensity, the local variance of pixel intensity, and the covariance between two images, which respectively reflect the average pixel intensity, the degree of pixel intensity change, and the consistency of pixel intensity change within each window. Since the set size of the window is the size of the pin area, therefore, when performing local analysis, a comparison analysis will be carried out based on each pin area. Since the area of the main body area is generally larger than the area of the pin area, therefore, the main body area will be evaluated for the similarity index based on multiple windows. Each window corresponds to a local area.

[0092] According to an embodiment of the present invention, the main body area and the pin area where the first chip analysis area and the second chip analysis area are associated are determined based on multiple similarity indices and a preset similarity threshold and marked to obtain an associated area. Through the associated area, image feature extraction is performed on the chip image data before and after chip packaging and processing, and is associated with the first defect information and the second defect information to generate an associated feature set before and after processing. Specifically:

[0093] Compare the similarity index of each window with the preset similarity threshold, and mark the windows higher than the preset similarity threshold to obtain multiple marked windows;

[0094] Based on one marked window, extract the local areas corresponding in the first chip analysis area and the second chip analysis area to obtain a first marked area and a second marked area;

[0095] Perform image feature extraction from the chip image data before chip packaging and processing based on the first marked area to obtain a first feature;

[0096] Perform image feature extraction from the chip image data after chip packaging and processing based on the second marked area to obtain a second feature;

[0097] Associate the first feature, the second feature, the first defect information, and the second defect information and form an associated data record;

[0098] Based on multiple marked windows, form multiple associated data records;

[0099] Integrate multiple associated data records to obtain an associated feature set before and after processing.

[0100] According to the embodiments of the present invention, the associated feature set before and after processing further includes:

[0101] Import the associated feature set before and after processing into the system database;

[0102] In the second detection period, obtain real-time chip image data;

[0103] Perform feature extraction and defect analysis on the real-time chip image data to form real-time features and real-time defect evaluation information;

[0104] Based on the real-time features and real-time defect evaluation information, perform associated feature and processing impact analysis from the associated feature set before and after processing to generate processing impact evaluation data;

[0105] Through the processing impact evaluation data, perform chip classification setting on the chips detected in real time.

[0106] It should be noted that each window corresponds to a local area, and this local area exists in the first chip analysis area and the second chip analysis area. Each window corresponds to a pin area or a main body area. The first marking area and the second marking area are the pin area or the main body area. The first defect information and the second defect information include the defect information before and after processing of different pin areas and the main body area. Image feature extraction includes features such as contour edges, textures, and colors. The processing impact evaluation data evaluates the impact on the current processing step (such as before packaging) and the impact on the next processing step (such as after packaging). The chip classification setting is based on the impact of defects, and sets whether the chip continues to be processed or stops processing and further analyzes the cause of the defects or detects the parameters of the processing equipment, etc. for classification setting. The first feature and the second feature are associated features.

[0107] It is worth mentioning that in the existing chip defect image detection, often only the finished chip products are detected for defects, ignoring the defect associations existing in the processing steps such as welding and packaging. It is difficult to achieve the full-process tracking of processing defects, reducing the chip manufacturing quality. Therefore, in the present invention, within a set detection period, the chip images before and after processing such as packaging and welding are preprocessed, the main body and pin regions are identified based on the corresponding spatial positioning templates, local similarity analysis based on SSIM is performed through the images before and after processing, the associated image regions before and after processing are screened out through the SSIM index, and the corresponding defect features and information are statistically analyzed to form an associated feature set. Through the associated feature set, in the subsequent chip manufacturing, it is possible to evaluate in real time whether there are corresponding defects in the manufacturing process of the chip, and based on the defect association information, evaluate and predict the impact of the defect in the subsequent processing steps, and further classify the subsequent processing of the defective chips, improving the information-based manufacturing detection ability of the chips and realizing the refined chip processing detection.

[0108] According to an embodiment of the present invention, it further includes:

[0109] Obtain the associated feature set before and after processing from the system database;

[0110] Set a type of defect, screen out the associated features that meet the said type of defect in the associated feature set before and after processing, and obtain an associated feature data set;

[0111] Construct a GAN-based generation model and set a loss function. The generation model includes a generator and a discriminator;

[0112] Import the associated feature data set as real data into the generation model. Through the generator, learn the real data and generate simulated features, import the simulated features into the discriminator for discrimination, and optimize the generator and the discriminator based on the discrimination result and the loss function;

[0113] Perform adversarial training of the generator and the discriminator in a loop until the generator and the discriminator reach Nash equilibrium;

[0114] Through the trained generation model, generate a preset amount of simulated features and use them as the training data for the said type of defect.

[0115] It should be noted that the training data is obtained by performing feature analysis on the processing images of the current chip manufacturing, and is obtained through adversarial learning training simulation of the corresponding associated data by GAN simulation. It is worth mentioning that during the manufacturing process of different chips, the generation of defects often requires training an image recognition model with a high accuracy rate, and generally based on a machine learning model. Training through a machine learning model requires a large amount of training data to improve the accuracy rate. However, in the actual manufacturing of chips, the image feature data of different chip products is different, and the defect feature data generated during the production process is also different. The amount of defect feature data collected is often small. In the processing and detection tasks with a fast chip product iteration cycle, it is difficult to achieve effective data collection and the construction of a recognition model with a high accuracy rate.

[0116] In the present invention, the associated feature data stores the feature data with certain defect associations in two consecutive processing steps. Further, using the associated features as real data and simulating through a GAN adversarial neural network can simulate training data similar to the associated features, effectively increasing the amount of training data. In the real-time construction of a chip recognition model, based on the training data, it is possible to quickly construct and train a model for existing defect types, and the accuracy rate can reach a relatively high expected value, realizing the processing recognition and intelligent analysis of different chip products. In addition, training the training data based on the associated features can further enhance the recognition accuracy of associated defects in subsequent detection cycles.

[0117] Figure 2 The block diagram of an intelligent chip processing detection system based on machine vision according to the present invention is shown.

[0118] The second aspect of the present invention also provides an intelligent chip processing detection system 2 based on machine vision. The system includes: a memory 21 and a processor 22. The memory includes an intelligent chip processing detection program based on machine vision. When the intelligent chip processing detection program based on machine vision is executed by the processor, the following steps are implemented:

[0119] Extract the standardized chip image from the system data, perform identification and positioning analysis on the chip body and pins of the standardized chip image, and formulate a spatial positioning template;

[0120] During a detection cycle, obtain the chip image data before chip packaging processing in real time and perform image preprocessing. Use the spatial positioning template to position the chip body and pins of the chip image data, and extract the main body area and the pin area through the positioning;

[0121] Based on the defect recognition module, perform chip defect recognition on the main body area and the pin area and record to obtain the first defect information;

[0122] Merge the main body area and the pin area into the first chip analysis area, obtain the chip image data after chip packaging and processing, perform chip main body and pin positioning analysis and defect recognition, and extract the corresponding second chip analysis area and second defect information;

[0123] Based on the SSIM similarity index, calculate the image information of the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. Through the brightness, contrast, and structure information, calculate the similarity between the first chip analysis area and the second chip analysis area. In the calculation process, set a window based on the size of the pin area, perform local area comparison based on the set window, and obtain the similarity index of each window;

[0124] Based on multiple similarity indexes and a preset similarity threshold, judge the associated main body area and pin area between the first chip analysis area and the second chip analysis area and mark them to obtain the associated area. Through the associated area, extract the features of the chip image data before and after chip packaging and processing, and associate them with the first defect information and the second defect information to generate an associated feature set before and after processing.

[0125] According to the embodiments of the present invention, extracting the standardized chip image from the system data, performing identification, positioning analysis on the chip main body and pins of the standardized chip image, and formulating a spatial positioning template specifically includes:

[0126] Obtain the standardized chip image from the system data;

[0127] Through region identification of the chip main body and pins on the standardized chip image, label and position the corresponding regions to obtain the chip main body and pin positioning position information;

[0128] Generate a spatial positioning template based on the image size, scale parameters of the standardized chip image, and the chip main body and pin positioning position information.

[0129] It should be noted that the standardized chip image is a standardized comparison image, which can be based on user settings or system screening and has a better comparison effect. The chip main body and pin positioning position information includes the chip main body position, the positions of each pin, the main body and pin areas, the relative position relationship information between the areas, etc. relative to the entire standardized chip image. The spatial positioning template is a positioning template based on a standardized image, which is used for rapid region positioning of the chip image collected in real time. Before positioning, it is necessary to perform operations such as rotation and scaling on the chip image collected in real time to make it consistent with the image size, scale parameters, etc. of the standardized chip image. This process is a standardization process.

[0130] According to an embodiment of the present invention, within one detection cycle, chip image data before chip packaging and processing is acquired in real time and image preprocessing is performed. The chip body and pins are located for the chip image data through a spatial positioning template, and the main body area and the pin area are extracted through positioning, specifically as follows:

[0131] Within one detection cycle, chip image data before chip packaging and processing is acquired;

[0132] Denoising, grayscale conversion, and image enhancement preprocessing are performed on the chip image data;

[0133] Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin areas are located through the regional position information of the spatial positioning template to obtain the main body area and the pin area.

[0134] According to an embodiment of the present invention, based on the defect recognition module, chip defects in the main body area and the pin area are recognized and recorded to obtain first defect information, specifically as follows:

[0135] A defect recognition module based on CNN is constructed;

[0136] Historical defect features are obtained from the system database and imported into the defect recognition module for pre-training;

[0137] Based on the main body area and the pin area, feature extraction is performed on the chip image data before chip packaging and processing for the corresponding areas, and the extracted features are imported into the defect recognition module for defect recognition and recording to obtain first defect information.

[0138] It should be noted that the first defect information is defect information before packaging, including information such as defect category and defect location. The first defect information is specifically defect information before chip packaging and processing, mainly judging defects of pins. Main chip defects are generally raw material defects, foreign objects, scratches, printing defects, etc., and pin defects include pin bending, missing, etc.

[0139] According to an embodiment of the present invention, the main body area and the pin area are combined into a first chip analysis area, chip image data after chip packaging and processing is acquired, and chip body and pin positioning analysis and defect recognition are performed, and the corresponding second chip analysis area and second defect information are extracted, specifically as follows:

[0140] Based on combining the regional information of the main body area and the pin area into a large area, a first chip analysis area is formed;

[0141] Based on the spatial positioning template, chip body and pin positioning are performed on the chip image data after chip packaging and processing, and a corresponding second chip analysis area is generated;

[0142] Import the chip image data after chip packaging and processing into the defect recognition module for defect recognition and recording to obtain the second defect information.

[0143] It should be noted that the purpose of forming the first chip analysis area is to extract the main chip image research through this area and exclude the irrelevant background area. The analysis processes of the second chip analysis area and the second defect information are the same as those of the first chip analysis area and the first defect information. In addition, the second defect information is the defect recognized for the chip after packaging and processing. There is a certain correlation between its defect categories and defect characteristics and the first defect information. Therefore, in the present invention, similarity analysis is performed subsequently to mine the association of processing defects, and the defect feature association and defect type association in continuous processing steps are analyzed and stored.

[0144] According to an embodiment of the present invention, based on the SSIM similarity index, image information calculation is performed on the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. Through the brightness, contrast, and structure information, similarity calculation is performed on the first chip analysis area and the second chip analysis area. The calculation process sets a window based on the size of the pin area, and local area comparison is performed based on the set window, and the similarity index of each window is obtained. Specifically:

[0145] Based on the first chip analysis area and the second chip analysis area, image information extraction and loading of the analysis area are respectively performed on the chip image data before and after chip packaging and processing to obtain the first image loading information and the second image loading information;

[0146] Set a window based on the size of the pin area through the SSIM similarity index calculation method;

[0147] Based on the window size, calculate the local image information in the first image loading information and the second image loading information, and calculate the local image information of each window by moving the window. The local image information includes brightness, contrast, and structure information;

[0148] Based on the brightness, contrast, and structure information of each window, combine the SSIM similarity index to calculate the similarity index of each window.

[0149] It should be noted that the first chip analysis area is the corresponding area based on the chip image data before chip packaging and processing, and the second chip analysis area is the corresponding area based on the chip image data after chip packaging and processing. The sizes of the two analysis areas are the same. The window size is the same as the pin area size, for example, 10×10 pixels. In the similarity index, the number of similarity indices is the same as the number of windows. The brightness, contrast, and structural information respectively refer to the average pixel intensity, the local variance of pixel intensity, and the covariance between two images, and the three respectively reflect the average pixel intensity, the degree of pixel intensity change, and the consistency of pixel intensity change within each window. Since the set window size is the pin area size, when performing local analysis, comparison analysis will be carried out based on each pin area. Since the area of the main body area is generally larger than the pin area, the main body area will be evaluated for similarity indices based on multiple windows. Each window corresponds to a local area.

[0150] According to an embodiment of the present invention, the main body area and the pin area where the first chip analysis area and the second chip analysis area are associated are determined based on a plurality of similarity indices and a preset similarity threshold and marked to obtain an associated area. Through the associated area, image feature extraction is performed on the chip image data before and after chip packaging and processing, and is associated with the first defect information and the second defect information to generate an associated feature set before and after processing. Specifically:

[0151] Compare the similarity index of each window with the preset similarity threshold, and mark the windows higher than the preset similarity threshold to obtain a plurality of marked windows;

[0152] Based on one marked window, extract the local areas corresponding in the first chip analysis area and the second chip analysis area to obtain a first marked area and a second marked area;

[0153] Perform image feature extraction on the chip image data before chip packaging and processing based on the first marked area to obtain a first feature;

[0154] Perform image feature extraction on the chip image data after chip packaging and processing based on the second marked area to obtain a second feature;

[0155] Associate the first feature, the second feature, the first defect information, and the second defect information and form an associated data record;

[0156] Based on a plurality of marked windows, form a plurality of associated data records;

[0157] Integrate the plurality of associated data records to obtain an associated feature set before and after processing.

[0158] According to an embodiment of the present invention, the associated feature set before and after processing further includes:

[0159] Import the associated feature sets before and after processing into the system database;

[0160] In the second detection cycle, obtain real-time chip image data;

[0161] Extract features and analyze defects from the real-time chip image data to form real-time features and real-time defect evaluation information;

[0162] Based on the real-time features and real-time defect evaluation information, conduct associated feature and processing impact analysis from the associated feature sets before and after processing to generate processing impact evaluation data;

[0163] Through the processing impact evaluation data, perform chip classification setting on the chips being detected in real time.

[0164] It should be noted that each of the windows corresponds to a local area, which exists in the first chip analysis area and the second chip analysis area, and each window corresponds to a pin area or a main body area. The first marking area and the second marking area are the pin area or the main body area. The first defect information and the second defect information include the defect information before and after processing of different pin areas and the main body area. Image feature extraction includes features such as contour edges, textures, and colors. The processing impact evaluation data evaluates the impact on the current processing step (such as before encapsulation) and the impact on the next processing step (such as after encapsulation). The chip classification setting is based on the impact of the defects, and sets whether the chip continues to be processed or stops processing and further analyzes the cause of the defects or detects the processing equipment parameters, etc. for classification setting. The first feature and the second feature are associated features.

[0165] It is worth mentioning that in the existing chip defect image detection, often only the finished products of chip processing are defect-detected, ignoring the defect associations existing in processing steps such as welding and encapsulation, making it difficult to achieve full-process tracking of processing defects and reducing the chip manufacturing quality. Therefore, in the present invention, within a set detection cycle, the chip images before and after processing such as encapsulation and welding are preprocessed, the main body and pin areas are identified based on the corresponding spatial positioning templates, local similarity analysis based on SSIM is performed through the images before and after processing, the image areas before and after processing with associations are screened out through the SSIM index, and the corresponding defect features and information are statistically analyzed to form an associated feature set. Through the associated feature set, in subsequent chip manufacturing, it can be used to evaluate in real time whether there are corresponding defects in the chip manufacturing process, and based on the defect association information, evaluate and predict the impact of the defect on subsequent processing steps, and further classify the chips with defects for subsequent processing, improving the information-based manufacturing detection ability of the chips and achieving refined chip processing detection.

[0166] In a third aspect of the present invention, there is also provided a computer-readable storage medium, which includes an intelligent detection program for chip processing based on machine vision. When the intelligent detection program for chip processing based on machine vision is executed by a processor, the steps of the intelligent detection method for chip processing based on machine vision as described in any one of the above are implemented.

[0167] The present invention discloses an intelligent detection method and system for chip processing based on machine vision. By acquiring a standardized image of a chip and identifying and positioning the chip body and pins, a spatial positioning template is formulated; within a detection cycle, chip image data is acquired and preprocessed, and the body and pin regions are extracted using the spatial positioning template; based on a defect recognition module, the defect information of these regions is identified and recorded; the body region and the pin region are combined into a first chip analysis region, and a secondary analysis is performed after chip packaging and processing to obtain a second chip analysis region and second defect information; based on the SSIM similarity index, the similarity index between the first and second chip analysis regions is calculated; the associated regions are judged according to the similarity index and a preset threshold, and are associated with the defect information to generate an associated feature set. Through the associated feature set, subsequent processing classification of defective chips can be achieved, improving the information-based manufacturing detection ability of chips.

[0168] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they may be located in one place, or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional unit in the embodiments of the present invention can be all integrated in one processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in one unit; the above integrated units can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0171] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including those of the above method embodiments. The aforementioned storage medium includes various media that can store program codes, such as removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0172] Alternatively, if the above integrated units of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program codes, such as removable storage devices, ROM, RAM, magnetic disks, or optical discs.

[0173] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A chip processing intelligent detection method based on machine vision, characterized in that: include: Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template; In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the preprocessed chip image data through a spatial positioning template, and the body area and pin area are extracted through positioning. Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information; The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information; Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window; Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

2. According to claim 1, a chip processing intelligent detection method based on machine vision is characterized in that: The chip standardized image is extracted from the system data, the chip body and pins are identified and located on the chip standardized image, and a spatial positioning template is formulated, specifically: Get chip standardized images from system data; By identifying the chip body and pin regions on the standardized chip image, marking and locating the corresponding regions, the chip body and pin location information is obtained; A spatial positioning template is generated based on the image size, scale parameters of the chip standardized image and the positioning position information of the chip body and pins.

3. The chip processing intelligent detection method based on machine vision according to claim 2 is characterized in that: In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed, the chip body and pins are positioned on the preprocessed chip image data through a spatial positioning template, and the body area and pin area are extracted through positioning, specifically: In one inspection cycle, the chip image data before chip packaging processing is obtained; Perform denoising, grayscale conversion, and image enhancement preprocessing on chip image data; Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin area are located through the regional position information of the spatial positioning template to obtain the body area and the pin area.

4. The chip processing intelligent detection method based on machine vision according to claim 3 is characterized in that: Based on the defect recognition module, the chip defect recognition is performed on the main body area and the pin area and the first defect information is recorded, specifically: Build a CNN-based defect recognition module; Obtain historical defect features from the system database and import them into the defect recognition module for pre-training; Based on the main body area and the pin area, feature extraction of corresponding areas is performed on chip image data before chip packaging processing, and the extracted features are imported into a defect recognition module for defect recognition and recording to obtain first defect information.

5. The chip processing intelligent detection method based on machine vision according to claim 4 is characterized in that: The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information, specifically: Based on merging the regional information of the main body region and the pin region into a large region, a first chip analysis region is formed; Based on the spatial positioning template, the chip image data after the chip packaging process is used to locate the chip body and pins and generate a corresponding second chip analysis area; The chip image data after chip packaging processing is imported into the defect recognition module for defect recognition and recording to obtain second defect information.

6. The chip processing intelligent detection method based on machine vision according to claim 5 is characterized in that: Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated through brightness, contrast, and structure information. The calculation process sets the window with the size of the pin area, compares the local area based on the set window, and obtains the similarity index of each window, which is specifically: Based on the first chip analysis area and the second chip analysis area, image information of the analysis area is extracted and loaded for chip image data before and after chip packaging processing, respectively, to obtain first image loading information and second image loading information; By using the SSIM similarity index calculation method, a window is set based on the size of the pin area; Based on the window size, calculate the local image information in the first image loading information and the second image loading information, and calculate the local image information of each window by moving the window, the local image information including brightness, contrast, and structure information; Based on the brightness, contrast, and structure information of each window, the similarity index of each window is calculated in combination with the SSIM similarity index.

7. The chip processing intelligent detection method based on machine vision according to claim 6 is characterized in that: The method determines the main body area and the pin area associated with the first chip analysis area and the second chip analysis area based on multiple similarity indexes and preset similarity thresholds and marks them to obtain the associated area. Through the associated area, the chip image data before and after the chip packaging process is extracted and associated with the first defect information and the second defect information to generate the associated feature set before and after the process, specifically: Compare the similarity index of each window with a preset similarity threshold, mark the windows with similarity index higher than the preset similarity threshold, and obtain multiple marked windows; Based on a marked window, extracting the local area corresponding to the first chip analysis area and the second chip analysis area to obtain a first marked area and a second marked area; Extracting image features from chip image data before chip packaging based on the first marking area to obtain a first feature; Extracting image features from chip image data after chip packaging based on the second marking area to obtain a second feature; Associating the first feature, the second feature, the first defect information and the second defect information to form an associated data record; Based on multiple marking windows, multiple associated data records are formed; Integrate multiple related data records to obtain the related feature set before and after processing.

8. The chip processing intelligent detection method based on machine vision according to claim 7 is characterized in that: The pre- and post-processing associated feature set also includes: Import the associated feature sets before and after processing into the system database; In the second detection cycle, real-time chip image data is acquired; Perform feature extraction and defect analysis on real-time chip image data to form real-time feature and real-time defect assessment information; Based on real-time features and real-time defect assessment information, the associated features and processing impact analysis are performed from the associated features before and after processing to generate processing impact assessment data; By processing impact assessment data, chip classification settings are made for chips detected in real time.

9. A chip processing intelligent detection system based on machine vision, characterized in that: The system includes: a memory and a processor, wherein the memory includes a chip processing intelligent detection program based on machine vision, and when the chip processing intelligent detection program based on machine vision is executed by the processor, the following steps are implemented: Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template; In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the preprocessed chip image data through a spatial positioning template, and the body area and pin area are extracted through positioning. Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information; The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information; Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window; Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a chip processing intelligent detection program based on machine vision. When the chip processing intelligent detection program based on machine vision is executed by a processor, the steps of the chip processing intelligent detection method based on machine vision as described in any one of claims 1 to 8 are implemented.

Images