A multi-classification method for interconnect indium pillar defects based on transfer learning

Abstract

A method for multi-classification of interconnected indium pillar defects based on transfer learning, which relates to a method for multi-classification of interconnected indium pillar defects. The invention aims to solve the problem of detecting structural defects and highly abnormal defects of focal plane connected pixels. The steps of the invention include step 1, generating defect data; step 2, establishing a transfer learning model; step 3, data dimension reduction; step 4, defect classification. The invention belongs to the field of image processing technology.

Description

Technical Field

[0001] The invention relates to a multi-classification method for interconnected indium pillar defects and belongs to the technical field of image processing. Background Art

[0002] Infrared technology has the advantages of high resolution, high confidentiality, strong recognition ability, strong anti-electronic interference ability, and can work all-weather and all-season. It is widely used in night vision imaging, precision guidance, fire control tracking, weapon thermal aiming, reconnaissance and early warning, ship navigation and other fields; infrared detectors are key components in infrared systems. Their function is to convert incident infrared signals into electrical signals for output, which are used for target detection and identification; the performance level of infrared detectors is related to the development level of infrared technology. With the continuous development of infrared detection technology, infrared detectors have also gone through several stages, starting from single-point HgCdTe photoconductive detectors, gradually developing to the second generation of infrared detectors represented by 288×4 HgCdTe photovoltaic focal plane detectors, and then to the most widely used HgCdTe, InSb and other infrared detectors. The improvement of detector chip performance has continuously broken through the performance limit at the cost of the complexity of functional microstructures, which is specifically manifested in the smaller and smaller pixel size, more complex structure, and a significant increase in the number of pixels, thereby improving resolution and sensitivity. The focal plane scale has expanded from 320×256, 640×512, 1280×1024 to 2560×2048. As the focal plane size increases, the pixel center distance continues to shrink from 30μm, 15μm to 10μm. The detection of structural defects and highly abnormal defects connecting pixels in the focal plane has become a problem that must be urgently addressed. Summary of the invention

[0003] The present invention solves the problem of detecting structural defects and highly abnormal defects of focal plane connected pixels, and further proposes a multi-classification method for interconnected indium pillar defects based on transfer learning.

[0004] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:

[0005] Step 1: Generate defect data;

[0006] Step 2: Establish a transfer learning model;

[0007] Step 3: Data dimensionality reduction;

[0008] Step 4: Defect classification.

[0009] Furthermore, the method for generating defect data in step 1 is: using a method for generating defect data of unpaired interconnected indium pillars based on a convolutional neural network, using a model to simulate an interconnected indium pillar model, and then performing style conversion to achieve data generation.

[0010] Furthermore, the method of establishing a transfer learning model in step 2 is to use the existing knowledge and experience of the Google-Net network to improve the performance of the model, which is suitable for scenarios with small data volumes and the need for rapid iteration.

[0011] Furthermore, the method of data dimensionality reduction in step 3 is: using a nonlinear data dimensionality reduction method to map the high-dimensional defect features extracted from the transfer learning network to a low-dimensional space, so as to facilitate visualization and improve the performance of the classification network.

[0012] Furthermore, the defect classification method in step 4 is: using an improved support vector machine to classify the defect features after dimensionality reduction, thereby ensuring classification accuracy while reducing classification time.

[0013] The beneficial effects of the present invention are as follows: 1. The present invention can realize the process of simulating the style conversion of interconnected indium pillar images to laser confocal images, solving the problem of low diversity of defect data;

[0014] 2. The present invention realizes rapid convergence of the interconnected indium pillar defect extraction network and improves its accuracy. The SVM algorithm improved based on error correction output coding is used as a classifier, which can solve the contradiction between accuracy and training time brought about by the traditional SVM algorithm training classifier mode. On the basis of ensuring the accuracy of interconnected indium pillar defect classification, the training time is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the present invention;

[0016] Figure 2 is a schematic diagram of the normalization layer. Figure 2 a is a schematic diagram of the BN layer. Figure 2 b is a schematic diagram of the IN layer;

[0017] Figure 3 This is a schematic diagram of the VGG-19 network structure;

[0018] Figure 4 It is the network structure and schematic diagram of convolutional neural network style transfer;

[0019] Figure 5 It is a schematic diagram of transfer learning;

[0020] Figure 6 This is a schematic diagram of different Inception convolution computing unit modules;

[0021] Figure 7 This is a schematic diagram of transfer learning based on feature extraction;

[0022] Figure 8 It is the flow chart of the ECOC-SVM interconnect indium pillar defect classification. DETAILED DESCRIPTION

[0023] Specific implementation method 1: Combination Figures 1 to 8 This embodiment is described. The steps of a multi-classification method for interconnected indium pillar defects based on transfer learning in this embodiment include:

[0024] Step 1: Generate defect data;

[0025] Step 2: Establish a transfer learning model;

[0026] Step 3: Data dimensionality reduction;

[0027] Step 4: Defect classification.

[0028] Specific implementation method 2: Combination Figures 1 to 8 To illustrate this embodiment, the method for generating defect data in step 1 of a multi-classification method of interconnected indium pillar defects based on transfer learning described in this embodiment is: adopting an unpaired interconnected indium pillar defect data generation method based on a convolutional neural network, using a model to simulate an interconnected indium pillar model, and then performing style conversion to achieve data generation.

[0029] The VGG-19 network is selected as the backbone network for style migration data generation, and the VGG-19 network is improved. First, the normalization layer is replaced with InstanceNormalization, referred to as the IN layer. The IN layer normalizes each sample separately instead of the entire batch. Therefore, it is not affected by the batch size and has a smaller amount of calculation and faster processing speed. At the same time, the IN layer can better retain the characteristics of the sample itself, so it is more suitable for image generation tasks than the BN layer. The calculation process is:

[0030] Step 1: For x∈R N*C*H*W , IN only calculates the mean and variance of the (H, W) dimension data of each sample, and retains the N and C dimension data;

[0031] Step 2: Normalize the mean and variance of the H and W dimensions;

[0032] Step 3, adding scaling parameter γ and translation parameter β;

[0033]

[0034]

[0035]

[0036] Compared with the BN layer, the IN layer can not only accelerate the convergence of the model, but also perform normalization while maintaining the independence of each image instance; the IN layer processes the data in each channel separately, and its calculation results are not affected by the channel and batch size, making the model training process more stable;

[0037] Secondly, the average pooling operation is used to improve the maximum pooling operation to more effectively improve the problem of gradient reconstruction. At the same time, the entire network uses convolution kernel (3×3) and pooling size (2×2), and uses small filter convolution layers to improve large filters to deepen the network structure and improve network performance. The network structure is as follows Figure 3 As shown;

[0038] The interconnect indium pillar defect data generation method based on convolutional neural network is used to calculate the Figure 4 As shown, the calculation process is:

[0039] Define x and p as the original image and the generated image. x will pass through the convolutional neural network and calculate the activation value. Since each layer in the network can be understood as a nonlinear filter group, its response image will form a set of filter feature maps.

[0040] Assume that there are N l There are N different convolution kernels. l The size is m l The feature maps are stored in the matrix is the activation value of the jth convolution kernel at position k on the lth layer, P l and F l For their respective feature representations in layer l, the loss function between two feature representations is expressed as: The method of generating interconnected indium pillars can continue to iterate through optimization P until the same response as the original image is obtained in a certain layer of the network;

[0041] On top of the responses of each layer of the network, a new representation is constructed to calculate the correlation between the responses of different filters. The feature correlation is given by the Gram matrix in is the inner product of feature maps i and j at layer l:

[0042]

[0043] A series of Gram matrices: G1, G2...G lIt is the network's description of the background texture. In order to generate a new texture for an image, it is necessary to continuously optimize and iterate until another image that matches the Gram matrix of the original image is found. The optimization iteration is completed by the mean square error distance between the Gram matrix of the original image and the Gram matrix of the generated image. Define G l , A l Represents the Gram matrix corresponding to the original image and the generated image of the lth layer, then the loss function of the lth layer is:

[0044]

[0045] Define a as the network, the total loss function of background texture reconstruction is:

[0046]

[0047] The loss function of the entire network is:

[0048] L total (p,a,x)=αL content +βL style (a,x),

[0049] Where α is the weighting factor for content reconstruction and β is the weighting factor for style reconstruction.

[0050] Specific implementation method three: Combination Figures 1 to 8 To illustrate this embodiment, the method of establishing a transfer learning model in step 2 of a multi-classification method of interconnected indium pillar defects based on transfer learning described in this embodiment is to use the existing knowledge and experience of the Google-Net network to improve the performance of the model, which is suitable for scenarios with small data volume and the need for fast iteration.

[0051] In this embodiment, transfer learning based on InceptionV3 is used to realize the feature extraction of interconnected indium pillar defect data, and the output of its convolutional layer is used as the feature extractor for the new task. On this basis, a new classifier or regressor is added for fine-tuning. Figure 7 As shown in the figure, using the feature extraction capability of the pre-trained model and the starting point provided to improve the training effect of the target task can provide better performance in the target task, thereby improving model accuracy, reducing training time, and also reducing data requirements. The specific implementation steps are:

[0052] Step (1), model pre-training: Use the ImageNet dataset to complete pre-training on InceptionV3. The pre-training process is completed in deep learning frameworks such as Pytorch, that is, the weight parameters of InceptionV3 are updated;

[0053] Step (2), feature extraction: load the weight parameters of the InceptionV3 model trained in the previous step into a new model, then add a custom fully connected layer or other type of layer to the new model and connect it to the last global average pooling layer of InceptionV3;

[0054] Step (3), model fine-tuning: After feature extraction is completed, all model layers before transfer learning can be frozen, and only the newly added custom layers can be trained.

[0055] Specific implementation method four: Combination Figures 1 to 8 To illustrate this embodiment, the method of data dimensionality reduction in step 3 of a multi-classification method of interconnected indium pillar defects based on transfer learning described in this embodiment is: a nonlinear data dimensionality reduction method is used to map the high-dimensional defect features extracted from the transfer learning network to a low-dimensional space, so as to facilitate visualization and improve the performance of the classification network.

[0056] The specific implementation steps of nonlinear data dimensionality reduction are:

[0057] Step (1), calculating the similarity between each data point and other data points in the high-dimensional space, usually using a Gaussian distribution function to calculate the similarity between data points;

[0058] Step (2), randomly initialize each data point in the low-dimensional space, and calculate the similarity between them and other data points, and also use the Gaussian distribution function to calculate the similarity between data points;

[0059] Step (3) calculating the similarity between each point in the high-dimensional space and each point in the low-dimensional space; using the t distribution function to calculate the similarity between each point in the high-dimensional space and each point in the low-dimensional space, and using the Gaussian distribution function to calculate the similarity between each point in the low-dimensional space and other points;

[0060] Step (4), using the gradient descent algorithm to minimize the KL divergence between the high-dimensional space and the low-dimensional space, and optimizing the error of similarity calculation by continuously adjusting the position of each data point in the low-dimensional space;

[0061] Step (5), repeat iterative steps (2) to (4) until the stopping condition is met.

[0062] Specific implementation method five: Combination Figures 1 to 8 To illustrate this embodiment, the defect classification method in step 4 of a multi-classification method for interconnected indium pillar defects based on transfer learning described in this embodiment is: using an improved support vector machine to classify the defect features after dimensionality reduction, thereby ensuring classification accuracy while reducing classification time.

[0063] The SVM classifier is used for training. The specific implementation process is as follows:

[0064] Step (1), ECOC algorithm first represents each category as a code consisting of 0 and 1, and then constructs a coding table. Each row in the coding table represents a binary coding method, as shown in Table 3-1;

[0065] Step (2), the binary classification problem corresponding to each row in the coding table is trained using SVM to obtain the best discriminant function;

[0066] Step (3): ECOC-SVM uses all trained binary classifiers to classify the test samples and obtain the output code result. The algorithm decodes the output code according to the pre-defined coding table to determine the category.

[0067] Table 3-1 Error correction output coding

[0068]

[0069] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A multi-classification method for interconnect indium pillar defects based on transfer learning, characterized by: The steps of the interconnect indium pillar defect multi-classification method based on transfer learning include: Step 1, generating defect data; using a method for generating defect data of unpaired interconnected indium pillars based on a convolutional neural network, using a model to simulate an interconnected indium pillar model, and then performing style conversion to achieve data generation; The VGG-19 network is selected as the backbone network for style transfer data generation, and the VGG-19 network is improved. First, the normalization layer is replaced by the IN layer to standardize each sample separately; the calculation process is: Step 1: For x∈R N*C*H*W , the IN layer only calculates the mean and variance of the data in the dimensions H and W of each sample, and retains the data in the dimensions N and C; Step 2: Normalize the mean and variance of the H and W dimensions; Step 3, adding scaling parameter γ and translation parameter β; Compared with the BN layer, the IN layer can not only accelerate the convergence of the model, but also perform normalization while maintaining the independence of each image instance; the IN layer processes the data in each channel separately, and its calculation results are not affected by the channel and batch size, making the model training process more stable; Secondly, the average pooling operation is used to improve the maximum pooling operation to more effectively improve the problem of gradient reconstruction. At the same time, the entire network uses a convolution kernel of 3×3 and a pooling size of 2×2, and uses a small filter convolution layer to improve a large filter to deepen the network structure and improve network performance. The calculation process of the interconnected indium pillar defect data generation method based on convolutional neural network is as follows: Define x and p as the original image and the generated image. x will pass through the convolutional neural network and calculate the activation value. Since each layer in the network can be understood as a nonlinear filter group, its response image will form a set of filter feature maps. Assume that there are N l There are N different convolution kernels. l Size is M l The feature maps are stored in the matrix is the activation value of the jth convolution kernel at position k on the lth layer, P l and F l For their respective feature representations in layer l, the loss function between two feature representations is expressed as: The method of generating interconnected indium pillars can be continuously optimized through iterations P until the same response as the original image is obtained in a certain layer of the network; On top of the responses of each layer of the network, a new representation is constructed to calculate the correlation between the responses of different filters. The feature correlation is given by the Gram matrix in is the inner product of feature maps i and j at layer l: A series of Gram matrices: G1, G2...G l It is the network's description of the background texture. In order to generate a new texture for an image, it is necessary to continuously optimize and iterate until another image that matches the Gram matrix of the original image is found. The optimization iteration is completed by the mean square error distance between the Gram matrix of the original image and the Gram matrix of the generated image. Define G l , A l Represents the Gram matrix corresponding to the original image and the generated image of the lth layer, then the loss function of the lth layer is: Define a as the network, the total loss function of background texture reconstruction is: The loss function of the entire network is: L total (p,a,x)=αL content +bL style (a,x), Where α is the weighting factor of content reconstruction, and b is the weighting factor of style reconstruction; Step 2: Establish a transfer learning model; use the existing knowledge and experience of the Google-Net network to improve the performance of the model, which is suitable for scenarios with small data volumes and requiring fast iterations; The feature extraction of interconnected indium pillar defect data is realized by using transfer learning based on InceptionV3, and the output of its convolutional layer is used as the feature extractor for the new task. On this basis, a new classifier or regressor is added for fine-tuning. The feature extraction capability of the pre-trained model and the starting point provided are used to improve the training effect of the target task, which can provide better performance in the target task, thereby improving the model accuracy, reducing the training time, and also reducing the data demand. The specific implementation steps are as follows: Step (1), model pre-training: Use the ImageNet dataset to complete pre-training on InceptionV3. The pre-training process is completed in the Pytorch deep learning framework, that is, the weight parameters of InceptionV3 are updated; Step (2), feature extraction: load the weight parameters of the InceptionV3 model trained in the previous step into a new model, then add a custom fully connected layer to the new model and connect it to the last global average pooling layer of InceptionV3; Step (3), model fine-tuning: After feature extraction is completed, all model layers before transfer learning can be frozen, and only the newly added custom layers can be trained; Step 3: Data dimensionality reduction: Use nonlinear data dimensionality reduction methods to map the high-dimensional defect features extracted from the transfer learning network to a low-dimensional space to facilitate visualization and improve the performance of the classification network; The specific implementation steps of nonlinear data dimensionality reduction are: Step (1), calculating the similarity between each data point and other data points in the high-dimensional space, usually using a Gaussian distribution function to calculate the similarity between data points; Step (2), randomly initialize each data point in the low-dimensional space, and calculate the similarity between them and other data points, and also use the Gaussian distribution function to calculate the similarity between data points; Step (3) calculating the similarity between each point in the high-dimensional space and each point in the low-dimensional space; using the t distribution function to calculate the similarity between each point in the high-dimensional space and each point in the low-dimensional space, and using the Gaussian distribution function to calculate the similarity between each point in the low-dimensional space and other points; Step (4), using the gradient descent algorithm to minimize the KL divergence between the high-dimensional space and the low-dimensional space, and optimizing the error of similarity calculation by continuously adjusting the position of each data point in the low-dimensional space; Step (5), repeat iterative steps (2) to (4) until the stopping condition is met; Step 4: Defect classification: Use the improved support vector machine to classify the defect features after dimensionality reduction, and ensure the classification accuracy while reducing the classification time; The SVM classifier is used for training. The specific implementation process is as follows: Step (1), ECOC algorithm first represents each category as a code consisting of 0 and 1, and then constructs a coding table, in which each row represents a binary coding method; Step (2), the binary classification problem corresponding to each row in the coding table is trained using SVM to obtain the best discriminant function; Step (3): ECOC-SVM uses all trained binary classifiers to classify the test samples and obtain the output code result. The algorithm decodes the output code according to the pre-defined coding table to determine the category.

Images