Chip processing intelligent detection method and system based on machine vision

Abstract

The invention discloses a chip processing intelligent detection method and system based on machine vision. The method comprises the following steps: acquiring a chip standardized image and identifying and positioning a chip main body and a pin to formulate a space positioning template; the method comprises the following steps: in a detection period, acquiring and preprocessing chip image data, and extracting main body and pin areas by using a space positioning template; identifying and recording defect information of the areas based on a defect identification module; combining the main body area and the pin area into a first chip analysis area, and performing secondary analysis after chip packaging processing to obtain a second chip analysis area and second defect information; based on the SSIM similarity index, calculating a similarity index of the first chip analysis area and the second chip analysis area; and according to the similarity index and a preset threshold value, judging an associated region, and associating the associated region with the defect information to generate an associated feature set. And by associating the feature set, subsequent processing classification can be performed on the chips with defects, and the informatization manufacturing detection capability of the chips is improved.

Description

Technical Field

[0001] The present invention relates to the field of chip processing analysis, and more specifically, to a chip processing intelligent detection method and system based on machine vision. Background Art

[0002] With the continuous advancement of integrated circuit manufacturing technology, the integration of chips is getting higher and higher, and the requirements for chip processing quality are also getting higher and higher. Traditional chip detection methods mainly rely on manual visual inspection or simple image processing technology, which cannot meet the needs of modern high-precision chip manufacturing. In addition, in the existing chip defect image detection, defect detection is often only performed on the finished chip processing products, ignoring the defect associations in the processing steps such as welding and packaging. It is difficult to achieve full-process tracking of processing defects and effective defect film evaluation, which reduces the quality of chip manufacturing. Summary of the invention

[0003] The present invention overcomes the defects of the prior art and proposes a chip processing intelligent detection method and system based on machine vision.

[0004] The first aspect of the present invention provides a chip processing intelligent detection method based on machine vision, comprising:

[0005] Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template;

[0006] In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the chip image data through the spatial positioning template, and the body area and pin area are extracted through positioning.

[0007] Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information;

[0008] The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information;

[0009] Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window;

[0010] Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

[0011] In this solution, the chip standardized image is extracted from the system data, the chip body and pins are identified and located in the chip standardized image, and a spatial positioning template is formulated, specifically:

[0012] Get chip standardized images from system data;

[0013] By identifying the chip body and pin regions on the standardized chip image, marking and locating the corresponding regions, the chip body and pin location information is obtained;

[0014] A spatial positioning template is generated based on the image size, scale parameters of the chip standardized image and the positioning position information of the chip body and pins.

[0015] In this solution, within one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed, the chip body and pins are located on the chip image data through a spatial positioning template, and the body area and pin area are extracted through positioning, specifically:

[0016] In one inspection cycle, the chip image data before chip packaging processing is obtained;

[0017] Perform denoising, grayscale conversion, and image enhancement preprocessing on chip image data;

[0018] Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin area are located through the regional position information of the spatial positioning template to obtain the body area and the pin area.

[0019] In this solution, the defect recognition module is used to identify chip defects in the main body area and the pin area and record the first defect information, specifically:

[0020] Build a CNN-based defect recognition module;

[0021] Obtain historical defect features from the system database and import them into the defect recognition module for pre-training;

[0022] Based on the main body area and the pin area, feature extraction of corresponding areas is performed on chip image data before chip packaging processing, and the extracted features are imported into a defect recognition module for defect recognition and recording to obtain first defect information.

[0023] In this solution, the main body area and the pin area are merged into the first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information, specifically:

[0024] Based on merging the area information of the main body area and the pin area into a large area, a first chip analysis area is formed;

[0025] Based on the spatial positioning template, the chip image data after the chip packaging process is used to locate the chip body and pins and generate a corresponding second chip analysis area;

[0026] The chip image data after chip packaging processing is imported into the defect recognition module for defect recognition and recording to obtain second defect information.

[0027] In this solution, based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated through brightness, contrast, and structure information. The calculation process sets the window with the size of the pin area, compares the local area based on the set window, and obtains the similarity index of each window, which is specifically:

[0028] Based on the first chip analysis area and the second chip analysis area, image information of the analysis area is extracted and loaded for chip image data before and after chip packaging processing, respectively, to obtain first image loading information and second image loading information;

[0029] By using the SSIM similarity index calculation method, a window is set based on the size of the pin area;

[0030] Based on the window size, calculate the local image information in the first image loading information and the second image loading information, and calculate the local image information of each window by moving the window, the local image information including brightness, contrast, and structure information;

[0031] Based on the brightness, contrast, and structure information of each window, the similarity index of each window is calculated in combination with the SSIM similarity index.

[0032] In this solution, based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area associated with the first chip analysis area and the second chip analysis area are judged and marked to obtain the associated area. Through the associated area, the chip image data before and after the chip packaging process is feature extracted and associated with the first defect information and the second defect information to generate the associated feature set before and after the process, specifically:

[0033] Compare the similarity index of each window with a preset similarity threshold, mark the windows with similarity index higher than the preset similarity threshold, and obtain multiple marked windows;

[0034] Based on a marked window, local areas corresponding to the first chip analysis area and the second chip analysis area are extracted to obtain a first marked area and a second marked area;

[0035] Extracting image features from chip image data before chip packaging based on the first marking area to obtain a first feature;

[0036] Extracting image features from chip image data after chip packaging based on the second marking area to obtain a second feature;

[0037] Associating the first feature, the second feature, the first defect information and the second defect information to form an associated data record;

[0038] Based on multiple marking windows, multiple associated data records are formed;

[0039] Integrate multiple related data records to obtain the related feature set before and after processing.

[0040] In this solution, the pre- and post-processing associated feature set also includes:

[0041] Import the associated feature sets before and after processing into the system database;

[0042] In the second detection cycle, real-time chip image data is acquired;

[0043] Perform feature extraction and defect analysis on real-time chip image data to form real-time feature and real-time defect assessment information;

[0044] Based on real-time features and real-time defect assessment information, the associated features and processing impact analysis are performed from the associated features before and after processing to generate processing impact assessment data;

[0045] By processing impact assessment data, chip classification settings are made for chips detected in real time.

[0046] The second aspect of the present invention further provides a chip processing intelligent detection system based on machine vision, the system comprising: a memory, a processor, the memory comprising a chip processing intelligent detection program based on machine vision, the chip processing intelligent detection program based on machine vision being executed by the processor to implement the following steps:

[0047] Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template;

[0048] In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the chip image data through the spatial positioning template, and the body area and pin area are extracted through positioning.

[0049] Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information;

[0050] The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information;

[0051] Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window;

[0052] Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

[0053] The third aspect of the present invention also provides a computer-readable storage medium, which includes a chip processing intelligent detection program based on machine vision. When the chip processing intelligent detection program based on machine vision is executed by a processor, the steps of the chip processing intelligent detection method based on machine vision as described in any one of the above are implemented.

[0054] The present invention discloses a chip processing intelligent detection method and system based on machine vision. By acquiring a standardized chip image and identifying and locating the chip body and pins, a spatial positioning template is formulated; within a detection cycle, chip image data is acquired and preprocessed, and the body and pin areas are extracted using the spatial positioning template; based on a defect recognition module, the defect information of these areas is identified and recorded; the body area and the pin area are merged into a first chip analysis area, and a secondary analysis is performed after the chip is packaged and processed to obtain a second chip analysis area and second defect information; based on the SSIM similarity index, the similarity index of the first and second chip analysis areas is calculated; the associated area is judged according to the similarity index and a preset threshold, and is associated with the defect information to generate an associated feature set. Through the associated feature set, subsequent processing classification can be achieved for chips with defects, and the information manufacturing detection capability of the chip can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flow chart of a chip processing intelligent detection method based on machine vision of the present invention is shown;

[0056] Figure 2 A block diagram of a chip processing intelligent detection system based on machine vision of the present invention is shown. DETAILED DESCRIPTION

[0057] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0059] Figure 1 The flowchart of the chip processing intelligent detection method based on machine vision of the present invention is shown.

[0060] like Figure 1 As shown, the first aspect of the present invention provides a chip processing intelligent detection method based on machine vision, comprising:

[0061] S102, extracting a standardized chip image from the system data, performing chip body and pin identification and positioning analysis on the standardized chip image, and formulating a spatial positioning template;

[0062] S104, within one detection cycle, obtaining chip image data before chip packaging and processing in real time and performing image preprocessing, locating the chip body and pins of the chip image data using a spatial positioning template, and extracting the body area and the pin area through positioning;

[0063] S106, based on the defect recognition module, performing chip defect recognition on the main body area and the pin area and recording to obtain first defect information;

[0064] S108, merging the main body area and the pin area into a first chip analysis area, acquiring chip image data after chip packaging and processing, performing chip body and pin positioning analysis and defect recognition, and extracting a corresponding second chip analysis area and second defect information;

[0065] S110, based on the SSIM similarity index, image information calculation is performed on the first chip analysis area and the second chip analysis area, the image information includes three dimensions of brightness, contrast, and structure, similarity calculation is performed on the first chip analysis area and the second chip analysis area based on the brightness, contrast, and structure information, the calculation process sets a window based on the size of the pin area, compares the local area based on the set window, and obtains a similarity index for each window;

[0066] S112, based on multiple similarity indexes and preset similarity thresholds, determine the main body area and pin area that are associated with the first chip analysis area and the second chip analysis area and mark them to obtain the associated area, and extract features of the chip image data before and after the chip packaging process through the associated area, and associate them with the first defect information and the second defect information to generate a set of associated features before and after processing.

[0067] According to an embodiment of the present invention, the chip standardized image is extracted from the system data, the chip body and the pins are identified and positioned on the chip standardized image, and a spatial positioning template is formulated, specifically:

[0068] Get chip standardized images from system data;

[0069] By identifying the chip body and pin regions on the standardized chip image, marking and locating the corresponding regions, the chip body and pin location information is obtained;

[0070] A spatial positioning template is generated based on the image size, scale parameters of the chip standardized image and the positioning position information of the chip body and pins.

[0071] It should be noted that the chip standardized image is a standardized comparison image, which can be based on user settings or system screening and has a better comparison effect. The chip body and pin positioning position information includes the chip body position relative to the entire chip standardized image, the position of each pin, the body and pin area, and the relative position relationship information between the areas. The spatial positioning template is a positioning template based on an image standardization situation, which is used to quickly locate the chip image collected in real time. Before positioning, the chip image collected in real time needs to be rotated, scaled, etc. to keep it consistent with the image size, scale parameters, etc. of the chip standardized image. This process is a standardization process.

[0072] According to an embodiment of the present invention, within one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed, the chip body and pins are located on the chip image data through a spatial positioning template, and the body area and the pin area are extracted through positioning, specifically:

[0073] In one inspection cycle, the chip image data before chip packaging processing is obtained;

[0074] Perform denoising, grayscale conversion, and image enhancement preprocessing on chip image data;

[0075] Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin area are located through the regional position information of the spatial positioning template to obtain the body area and the pin area.

[0076] According to an embodiment of the present invention, the defect recognition module is used to identify chip defects in the main body area and the pin area and record the first defect information, specifically:

[0077] Build a CNN-based defect recognition module;

[0078] Obtain historical defect features from the system database and import them into the defect recognition module for pre-training;

[0079] Based on the main body area and the pin area, feature extraction of corresponding areas is performed on chip image data before chip packaging processing, and the extracted features are imported into a defect recognition module for defect recognition and recording to obtain first defect information.

[0080] It should be noted that the first defect information is the defect information before packaging, including defect type, defect location, etc. The first defect information is specifically the defect information before chip packaging processing, which mainly judges the defects of the pins. The main chip defects are generally poor raw materials, foreign matter, scratches, printing defects, etc., and the pin defects include bent and missing pins, etc.

[0081] According to an embodiment of the present invention, the main body area and the pin area are merged into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed, and the corresponding second chip analysis area and second defect information are extracted, specifically:

[0082] Based on merging the area information of the main body area and the pin area into a large area, a first chip analysis area is formed;

[0083] Based on the spatial positioning template, the chip image data after the chip packaging process is used to locate the chip body and pins and generate a corresponding second chip analysis area;

[0084] The chip image data after chip packaging processing is imported into the defect recognition module for defect recognition and recording to obtain second defect information.

[0085] It should be noted that the purpose of forming the first chip analysis area is to extract the main chip image research through this area and propose irrelevant background areas. The analysis process of the second chip analysis area and the second defect information is consistent with the analysis process of the first chip analysis area and the first defect information. In addition, the second defect information is the chip identification defect after packaging processing. Its defect category and defect characteristics have a certain correlation with the first defect information. Therefore, the present invention performs correlation mining of processing defects through subsequent similarity analysis, and analyzes and stores the defect feature correlation and defect type correlation in continuous processing steps.

[0086] According to an embodiment of the present invention, based on the SSIM similarity index, image information calculation is performed on the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. The similarity calculation is performed on the first chip analysis area and the second chip analysis area through brightness, contrast, and structure information. The calculation process sets the window with the pin area size, compares the local area based on the set window, and obtains the similarity index of each window, which is specifically:

[0087] Based on the first chip analysis area and the second chip analysis area, image information of the analysis area is extracted and loaded for chip image data before and after chip packaging processing, respectively, to obtain first image loading information and second image loading information;

[0088] By using the SSIM similarity index calculation method, a window is set based on the size of the pin area;

[0089] Based on the window size, calculate the local image information in the first image loading information and the second image loading information, and calculate the local image information of each window by moving the window, the local image information including brightness, contrast, and structure information;

[0090] Based on the brightness, contrast, and structure information of each window, the similarity index of each window is calculated in combination with the SSIM similarity index.

[0091] It should be noted that the first chip analysis area is the corresponding area based on the chip image data before the chip packaging process, and the second chip analysis area is the corresponding area based on the chip image data after the chip packaging process, and the sizes of the two analysis areas are the same. The window size is consistent with the pin area size, for example, 10×10 pixels. In the similarity index, the number of similarity indexes is consistent with the number of windows. The brightness, contrast, and structural information refer to the average pixel intensity, the local variance of the pixel intensity, and the covariance between the two images, respectively, which reflect the average pixel intensity in each window, the degree of pixel intensity change, and the consistency of pixel intensity change. Since the size of the window setting is the size of the pin area, when performing local analysis, a comparative analysis will be performed based on each pin area. Since the area of ​​the main area is generally larger than the area of ​​the pin area, the main area will be evaluated for similarity index based on multiple windows. Each window corresponds to a local area.

[0092] According to an embodiment of the present invention, based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area associated with the first chip analysis area and the second chip analysis area are judged and marked to obtain the associated area. Through the associated area, the chip image data before and after the chip packaging process is feature extracted and associated with the first defect information and the second defect information to generate a pre- and post-processing associated feature set, specifically:

[0093] Compare the similarity index of each window with a preset similarity threshold, mark the windows with similarity index higher than the preset similarity threshold, and obtain multiple marked windows;

[0094] Based on a marked window, extracting the local area corresponding to the first chip analysis area and the second chip analysis area to obtain a first marked area and a second marked area;

[0095] Extracting image features from chip image data before chip packaging based on the first marking area to obtain a first feature;

[0096] Extracting image features from chip image data after chip packaging based on the second marking area to obtain a second feature;

[0097] Associating the first feature, the second feature, the first defect information and the second defect information to form an associated data record;

[0098] Based on multiple marking windows, multiple associated data records are formed;

[0099] Integrate multiple related data records to obtain the related feature set before and after processing.

[0100] According to an embodiment of the present invention, the pre- and post-processing associated feature set further includes:

[0101] Import the associated feature sets before and after processing into the system database;

[0102] In the second detection cycle, real-time chip image data is acquired;

[0103] Perform feature extraction and defect analysis on real-time chip image data to form real-time feature and real-time defect assessment information;

[0104] Based on real-time features and real-time defect assessment information, the associated features and processing impact analysis are performed from the associated features before and after processing to generate processing impact assessment data;

[0105] By processing impact assessment data, chip classification settings are made for chips detected in real time.

[0106] It should be noted that each of the windows corresponds to a local area, which exists in the first chip analysis area and the second chip analysis area, and each window corresponds to the pin area or the main body area. The first marking area and the second marking area are the pin area or the main body area. The first defect information and the second defect information include defect information of different pin areas and the main body area before and after processing. Image feature extraction includes features such as contour edges, textures, and colors. The processing impact assessment data evaluates the impact of the current processing step (for example, before packaging) and the impact on the next processing step (for example, after packaging). The chip classification setting, based on the impact of the defect, sets the chip to continue processing or stop processing and further analyzes the cause of the defect or detects processing equipment parameters and other classification settings. The first feature and the second feature are associated features.

[0107] It is worth mentioning that in the existing chip defect image detection, defect detection is often only performed on the finished chip, ignoring the defect associations in the processing steps such as welding and packaging, making it difficult to achieve full-process tracking of processing defects, thereby reducing the chip manufacturing quality. Therefore, the present invention pre-processes the chip images before and after processing such as packaging and welding within a set detection cycle, identifies the main body and pin area based on the corresponding spatial positioning template, performs local similarity analysis based on SSIM through the images before and after processing, selects the image areas before and after processing with associations through the SSIM index, and performs statistics on the corresponding defect features and information to form an associated feature set. Through the associated feature set, in the subsequent chip manufacturing, the associated feature set can be used to evaluate in real time whether the chip has corresponding defects in the manufacturing process, and based on the defect association information, evaluate and predict the impact of the defect in the subsequent processing steps, and further implement subsequent processing classification for the chip with defects, improve the information manufacturing detection capability of the chip, and realize refined chip processing detection.

[0108] According to an embodiment of the present invention, it also includes:

[0109] Obtain the associated feature set before and after processing from the system database;

[0110] A defect type is set, and associated features that meet the defect type are screened out from the associated feature sets before and after processing, and an associated feature data set is obtained;

[0111] Build a GAN-based generative model and set the loss function. The generative model includes a generator and a discriminator.

[0112] Import the associated feature data set as real data into the generative model, learn the real data through the generator and generate simulated features, import the simulated features into the discriminator for discrimination, and optimize the generator and discriminator based on the discrimination results and loss function;

[0113] The adversarial training of the generator and the discriminator is repeated until the generator and the discriminator reach Nash equilibrium.

[0114] The trained generation model is used to generate simulated features of a preset data volume and used as training data for the defect type.

[0115] It should be noted that the training data is obtained by feature analysis based on the processing images of the current chip manufacturing, and is obtained by adversarial learning training simulation of the corresponding related data through GAN simulation. It is worth mentioning that in the manufacturing process of different chips, the generation of defects often requires the training of image recognition models with higher accuracy, and is generally based on machine learning models. A large amount of training data is required to improve the accuracy through machine learning model training. However, in the actual manufacturing of chips, different chip products have different image feature data, and the defect features generated in the production process also differ. The amount of defect feature data collected is often small. In the processing and inspection tasks with fast chip product iteration cycles, it is difficult to achieve effective data collection and high-accuracy recognition model construction.

[0116] In the present invention, the associated feature data stores feature data with certain defect associations in two consecutive processing steps. Furthermore, by using the associated features as real data and simulating the GAN adversarial neural network, it is possible to simulate training data similar to the associated features, effectively increasing the amount of training data. In the real-time chip recognition model construction, the existing defect type model can be quickly constructed and trained based on the training data, and the accuracy can reach a higher expected value, realizing the processing recognition and intelligent analysis of different chip products. In addition, training data based on associated features can further enhance the recognition accuracy of associated defects in subsequent detection cycles.

[0117] Figure 2 A block diagram of a chip processing intelligent detection system based on machine vision of the present invention is shown.

[0118] The second aspect of the present invention further provides a chip processing intelligent detection system 2 based on machine vision, the system comprising: a memory 21, a processor 22, the memory comprising a chip processing intelligent detection program based on machine vision, the chip processing intelligent detection program based on machine vision being executed by the processor to implement the following steps:

[0119] Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template;

[0120] In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the chip image data through the spatial positioning template, and the body area and pin area are extracted through positioning.

[0121] Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information;

[0122] The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information;

[0123] Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window;

[0124] Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

[0125] According to an embodiment of the present invention, the chip standardized image is extracted from the system data, the chip body and the pins are identified and positioned on the chip standardized image, and a spatial positioning template is formulated, specifically:

[0126] Get chip standardized images from system data;

[0127] By identifying the chip body and pin regions on the standardized chip image, marking and locating the corresponding regions, the chip body and pin location information is obtained;

[0128] A spatial positioning template is generated based on the image size, scale parameters of the chip standardized image and the positioning position information of the chip body and pins.

[0129] It should be noted that the chip standardized image is a standardized comparison image, which can be based on user settings or system screening and has a better comparison effect. The chip body and pin positioning position information includes the chip body position relative to the entire chip standardized image, the position of each pin, the body and pin area, and the relative position relationship information between the areas. The spatial positioning template is a positioning template based on an image standardization situation, which is used to quickly locate the chip image collected in real time. Before positioning, the chip image collected in real time needs to be rotated, scaled, etc. to keep it consistent with the image size, scale parameters, etc. of the chip standardized image. This process is a standardization process.

[0130] According to an embodiment of the present invention, within one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed, the chip body and pins are located on the chip image data through a spatial positioning template, and the body area and the pin area are extracted through positioning, specifically:

[0131] In one inspection cycle, the chip image data before chip packaging processing is obtained;

[0132] Perform denoising, grayscale conversion, and image enhancement preprocessing on chip image data;

[0133] Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin area are located through the regional position information of the spatial positioning template to obtain the body area and the pin area.

[0134] According to an embodiment of the present invention, the defect recognition module is used to identify chip defects in the main body area and the pin area and record the first defect information, specifically:

[0135] Build a CNN-based defect recognition module;

[0136] Obtain historical defect features from the system database and import them into the defect recognition module for pre-training;

[0137] Based on the main body area and the pin area, feature extraction of corresponding areas is performed on chip image data before chip packaging processing, and the extracted features are imported into a defect recognition module for defect recognition and recording to obtain first defect information.

[0138] It should be noted that the first defect information is the defect information before packaging, including defect type, defect location, etc. The first defect information is specifically the defect information before chip packaging processing, which mainly judges the defects of the pins. The main chip defects are generally poor raw materials, foreign matter, scratches, printing defects, etc., and the pin defects include bent and missing pins, etc.

[0139] According to an embodiment of the present invention, the main body area and the pin area are merged into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed, and the corresponding second chip analysis area and second defect information are extracted, specifically:

[0140] Based on merging the regional information of the main body region and the pin region into a large region, a first chip analysis region is formed;

[0141] Based on the spatial positioning template, the chip image data after the chip packaging process is used to locate the chip body and pins and generate a corresponding second chip analysis area;

[0142] The chip image data after chip packaging processing is imported into the defect recognition module for defect recognition and recording to obtain second defect information.

[0143] It should be noted that the purpose of forming the first chip analysis area is to extract the main chip image research through this area and propose irrelevant background areas. The analysis process of the second chip analysis area and the second defect information is consistent with the analysis process of the first chip analysis area and the first defect information. In addition, the second defect information is the chip identification defect after packaging processing. Its defect category and defect characteristics have a certain correlation with the first defect information. Therefore, the present invention performs correlation mining of processing defects through subsequent similarity analysis, and analyzes and stores the defect feature correlation and defect type correlation in continuous processing steps.

[0144] According to an embodiment of the present invention, based on the SSIM similarity index, image information calculation is performed on the first chip analysis area and the second chip analysis area. The image information includes three dimensions: brightness, contrast, and structure. The similarity calculation is performed on the first chip analysis area and the second chip analysis area through brightness, contrast, and structure information. The calculation process sets the window with the pin area size, compares the local area based on the set window, and obtains the similarity index of each window, which is specifically:

[0145] Based on the first chip analysis area and the second chip analysis area, image information of the analysis area is extracted and loaded for chip image data before and after chip packaging processing, respectively, to obtain first image loading information and second image loading information;

[0146] By using the SSIM similarity index calculation method, a window is set based on the size of the pin area;

[0147] Based on the window size, calculate the local image information in the first image loading information and the second image loading information, and calculate the local image information of each window by moving the window, the local image information including brightness, contrast, and structure information;

[0148] Based on the brightness, contrast, and structure information of each window, the similarity index of each window is calculated in combination with the SSIM similarity index.

[0149] It should be noted that the first chip analysis area is the corresponding area based on the chip image data before the chip packaging process, and the second chip analysis area is the corresponding area based on the chip image data after the chip packaging process, and the sizes of the two analysis areas are the same. The window size is consistent with the pin area size, for example, 10×10 pixels. In the similarity index, the number of similarity indexes is consistent with the number of windows. The brightness, contrast, and structural information refer to the average pixel intensity, the local variance of the pixel intensity, and the covariance between the two images, respectively, which reflect the average pixel intensity in each window, the degree of pixel intensity change, and the consistency of pixel intensity change. Since the size of the window setting is the size of the pin area, when performing local analysis, a comparative analysis will be performed based on each pin area. Since the area of ​​the main area is generally larger than the area of ​​the pin area, the main area will be evaluated for similarity index based on multiple windows. Each window corresponds to a local area.

[0150] According to an embodiment of the present invention, based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area associated with the first chip analysis area and the second chip analysis area are judged and marked to obtain the associated area. Through the associated area, the chip image data before and after the chip packaging process is feature extracted and associated with the first defect information and the second defect information to generate a pre- and post-processing associated feature set, specifically:

[0151] Compare the similarity index of each window with a preset similarity threshold, mark the windows with similarity index higher than the preset similarity threshold, and obtain multiple marked windows;

[0152] Based on a marked window, extracting the local area corresponding to the first chip analysis area and the second chip analysis area to obtain a first marked area and a second marked area;

[0153] Extracting image features from chip image data before chip packaging based on the first marking area to obtain a first feature;

[0154] Extracting image features from chip image data after chip packaging based on the second marking area to obtain a second feature;

[0155] Associating the first feature, the second feature, the first defect information and the second defect information to form an associated data record;

[0156] Based on multiple marking windows, multiple associated data records are formed;

[0157] Integrate multiple related data records to obtain the related feature set before and after processing.

[0158] According to an embodiment of the present invention, the pre- and post-processing associated feature set further includes:

[0159] Import the associated feature sets before and after processing into the system database;

[0160] In the second detection cycle, real-time chip image data is acquired;

[0161] Perform feature extraction and defect analysis on real-time chip image data to form real-time feature and real-time defect assessment information;

[0162] Based on real-time features and real-time defect assessment information, the associated features and processing impact analysis are performed from the associated features before and after processing to generate processing impact assessment data;

[0163] By processing impact assessment data, chip classification settings are made for chips detected in real time.

[0164] It should be noted that each of the windows corresponds to a local area, which exists in the first chip analysis area and the second chip analysis area, and each window corresponds to the pin area or the main body area. The first marking area and the second marking area are the pin area or the main body area. The first defect information and the second defect information include defect information of different pin areas and the main body area before and after processing. Image feature extraction includes features such as contour edges, textures, and colors. The processing impact assessment data evaluates the impact of the current processing step (for example, before packaging) and the impact on the next processing step (for example, after packaging). The chip classification setting, based on the impact of the defect, sets the chip to continue processing or stop processing and further analyzes the cause of the defect or detects processing equipment parameters and other classification settings. The first feature and the second feature are associated features.

[0165] It is worth mentioning that in the existing chip defect image detection, defect detection is often only performed on the finished chip, ignoring the defect associations in the processing steps such as welding and packaging, making it difficult to achieve full-process tracking of processing defects, thereby reducing the chip manufacturing quality. Therefore, the present invention pre-processes the chip images before and after processing such as packaging and welding within a set detection cycle, identifies the main body and pin area based on the corresponding spatial positioning template, performs local similarity analysis based on SSIM through the images before and after processing, selects the image areas before and after processing with associations through the SSIM index, and performs statistics on the corresponding defect features and information to form an associated feature set. Through the associated feature set, in the subsequent chip manufacturing, the associated feature set can be used to evaluate in real time whether the chip has corresponding defects in the manufacturing process, and based on the defect association information, evaluate and predict the impact of the defect in the subsequent processing steps, and further implement subsequent processing classification for the chip with defects, improve the information manufacturing detection capability of the chip, and realize refined chip processing detection.

[0166] The third aspect of the present invention also provides a computer-readable storage medium, which includes a chip processing intelligent detection program based on machine vision. When the chip processing intelligent detection program based on machine vision is executed by a processor, the steps of the chip processing intelligent detection method based on machine vision as described in any one of the above are implemented.

[0167] The present invention discloses a chip processing intelligent detection method and system based on machine vision. By acquiring a standardized chip image and identifying and locating the chip body and pins, a spatial positioning template is formulated; within a detection cycle, chip image data is acquired and preprocessed, and the body and pin areas are extracted using the spatial positioning template; based on a defect recognition module, the defect information of these areas is identified and recorded; the body area and the pin area are merged into a first chip analysis area, and a secondary analysis is performed after the chip is packaged and processed to obtain a second chip analysis area and second defect information; based on the SSIM similarity index, the similarity index of the first and second chip analysis areas is calculated; the associated area is judged according to the similarity index and a preset threshold, and is associated with the defect information to generate an associated feature set. Through the associated feature set, subsequent processing classification can be achieved for chips with defects, and the information manufacturing detection capability of the chip can be improved.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0170] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0171] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and other media that can store program codes.

[0172] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0173] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A chip processing intelligent detection method based on machine vision, characterized in that: include: Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template; In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the chip image data through the spatial positioning template, and the body area and pin area are extracted through positioning. Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information; The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information; Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window; Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

2. According to claim 1, a chip processing intelligent detection method based on machine vision is characterized in that: The chip standardized image is extracted from the system data, the chip body and pins are identified and located on the chip standardized image, and a spatial positioning template is formulated, specifically: Get chip standardized images from system data; By identifying the chip body and pin regions on the standardized chip image, marking and locating the corresponding regions, the chip body and pin location information is obtained; A spatial positioning template is generated based on the image size, scale parameters of the chip standardized image and the positioning position information of the chip body and pins.

3. The chip processing intelligent detection method based on machine vision according to claim 2 is characterized in that: In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed, the chip body and pins are located on the chip image data through a spatial positioning template, and the body area and pin area are extracted through positioning, specifically: In one inspection cycle, the chip image data before chip packaging processing is obtained; Perform denoising, grayscale conversion, and image enhancement preprocessing on chip image data; Based on the spatial positioning template, the preprocessed chip image data is standardized through image scaling, and the chip body and pin area are located through the regional position information of the spatial positioning template to obtain the body area and the pin area.

4. The chip processing intelligent detection method based on machine vision according to claim 3 is characterized in that: Based on the defect recognition module, the chip defect recognition is performed on the main body area and the pin area and the first defect information is recorded, specifically: Build a CNN-based defect recognition module; Obtain historical defect features from the system database and import them into the defect recognition module for pre-training; Based on the main body area and the pin area, feature extraction of corresponding areas is performed on chip image data before chip packaging processing, and the extracted features are imported into a defect recognition module for defect recognition and recording to obtain first defect information.

5. The chip processing intelligent detection method based on machine vision according to claim 4 is characterized in that: The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information, specifically: Based on merging the area information of the main body area and the pin area into a large area, a first chip analysis area is formed; Based on the spatial positioning template, the chip image data after the chip packaging process is used to locate the chip body and pins and generate a corresponding second chip analysis area; The chip image data after chip packaging processing is imported into the defect recognition module for defect recognition and recording to obtain second defect information.

6. The chip processing intelligent detection method based on machine vision according to claim 5 is characterized in that: Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated through brightness, contrast, and structure information. The calculation process sets the window with the size of the pin area, compares the local area based on the set window, and obtains the similarity index of each window, which is specifically: Based on the first chip analysis area and the second chip analysis area, image information of the analysis area is extracted and loaded for chip image data before and after chip packaging processing, respectively, to obtain first image loading information and second image loading information; By using the SSIM similarity index calculation method, a window is set based on the size of the pin area; Based on the window size, calculate the local image information in the first image loading information and the second image loading information, and calculate the local image information of each window by moving the window, the local image information including brightness, contrast, and structure information; Based on the brightness, contrast, and structure information of each window, the similarity index of each window is calculated in combination with the SSIM similarity index.

7. The chip processing intelligent detection method based on machine vision according to claim 6 is characterized in that: The method determines the main body area and the pin area associated with the first chip analysis area and the second chip analysis area based on multiple similarity indexes and preset similarity thresholds and marks them to obtain the associated area. Through the associated area, the chip image data before and after the chip packaging process is extracted and associated with the first defect information and the second defect information to generate the associated feature set before and after the process, specifically: Compare the similarity index of each window with a preset similarity threshold, mark the windows with similarity index higher than the preset similarity threshold, and obtain multiple marked windows; Based on a marked window, local areas corresponding to the first chip analysis area and the second chip analysis area are extracted to obtain a first marked area and a second marked area; Extracting image features from chip image data before chip packaging based on the first marking area to obtain a first feature; Extracting image features from chip image data after chip packaging based on the second marking area to obtain a second feature; Associating the first feature, the second feature, the first defect information and the second defect information to form an associated data record; Based on multiple marking windows, multiple associated data records are formed; Integrate multiple related data records to obtain the related feature set before and after processing.

8. The chip processing intelligent detection method based on machine vision according to claim 7 is characterized in that: The pre- and post-processing associated feature set also includes: Import the associated feature sets before and after processing into the system database; In the second detection cycle, real-time chip image data is acquired; Perform feature extraction and defect analysis on real-time chip image data to form real-time feature and real-time defect assessment information; Based on real-time features and real-time defect assessment information, the associated features and processing impact analysis are performed from the associated features before and after processing to generate processing impact assessment data; By processing impact assessment data, chip classification settings are made for chips detected in real time.

9. A chip processing intelligent detection system based on machine vision, characterized in that: The system includes: a memory and a processor, wherein the memory includes a chip processing intelligent detection program based on machine vision, and when the chip processing intelligent detection program based on machine vision is executed by the processor, the following steps are implemented: Extract standardized chip images from system data, identify and locate the chip body and pins on the standardized chip images, and develop a spatial positioning template; In one detection cycle, the chip image data before chip packaging processing is acquired in real time and image preprocessing is performed. The chip body and pins are located on the chip image data through the spatial positioning template, and the body area and pin area are extracted through positioning. Based on the defect recognition module, chip defects are recognized in the main body area and the pin area and recorded to obtain first defect information; The main body area and the pin area are combined into a first chip analysis area, the chip image data after chip packaging processing is obtained, and the chip body and pin positioning analysis and defect identification are performed to extract the corresponding second chip analysis area and second defect information; Based on the SSIM similarity index, the image information of the first chip analysis area and the second chip analysis area is calculated. The image information includes three dimensions: brightness, contrast, and structure. The similarity of the first chip analysis area and the second chip analysis area is calculated based on the brightness, contrast, and structure information. The calculation process sets the window based on the pin area size, compares the local area based on the set window, and obtains the similarity index of each window; Based on multiple similarity indexes and preset similarity thresholds, the main body area and the pin area that are associated with the first chip analysis area and the second chip analysis area are determined and marked to obtain the associated area. Through the associated area, features are extracted from the chip image data before and after the chip packaging process, and associated with the first defect information and the second defect information to generate a set of associated features before and after processing.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a chip processing intelligent detection program based on machine vision. When the chip processing intelligent detection program based on machine vision is executed by a processor, the steps of the chip processing intelligent detection method based on machine vision as described in any one of claims 1 to 8 are implemented.

Images