Artificial intelligence identification method for optical printing characters on surface of plastic package electronic component

Abstract

The invention relates to an artificial intelligence identification method for optical printing characters on the surface of a plastic package electronic component, and the detection method comprises the steps: 1), obtaining an image, obtaining a clear graphic data set sample, carrying out the size transformation of a graphic data set, and dividing the graphic data set into a training set, a verification set and a test set according to a proportion for model training verification and test; step 2), constructing an improved DBNet character detection network based on an attention mechanism; 3) constructing an improved convolutional recurrent neural recognition network based on the spatial transformation network; 4) combining the detection network in the step 2) and the recognition network in the step 3) to obtain an optical character recognition network model; and 5) training and testing the optical character recognition model in the step 4) by using the training set. The device has the advantages of high detection precision, high stability, high speed and high efficiency, and can be used for non-contact nondestructive detection.

Description

Technical Field

[0001] The invention relates to the technical field of electronic component testing, and in particular to an artificial intelligence recognition method for optically printed characters on the surface of a plastic-sealed electronic component. Background Art

[0002] As an important product of the information age, electronic components are a necessity of modern civilized society and are widely used in computers, mobile phones, and intelligent digital products. Electronic components are of various types and sizes, so it is essential to detect all aspects of the production and manufacturing of electronic components. Compared with metal packaging and ceramic packaging, plastic packaging is the most commonly used packaging method in the microelectronics industry. After the chip is packaged, it needs to be screened and tested. The first step of the screening test is to read the printed logo on the chip surface. The key information of the component, such as manufacturer, model, batch, production date, LOGO, etc., will be printed on the surface of the chip package. Therefore, accurately and quickly identifying and recording these key information is a necessary prerequisite for improving the efficiency of screening and testing. The size of existing electronic components is getting smaller and smaller, and there are many types. The character printing methods are diverse and difficult to distinguish clearly with the naked eye. As a result, the manual recording method in the traditional testing industry is time-consuming and laborious, the accuracy is difficult to guarantee, and it is easy to damage the chip. In contrast, the method of using artificial intelligence to perform optical printed character recognition on the surface of electronic components has more prominent advantages. First, the detection accuracy is high and the stability is strong. Second, it is fast and efficient. Third, it can perform non-contact non-destructive testing. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an artificial intelligence recognition method for optically printed characters on the surface of plastic-packaged electronic components. Aiming at the problem that it is difficult and inefficient to manually recognize printed identification characters on the chip surface during the screening and detection of plastic-packaged electronic components, a single-chip optical character recognition algorithm based on a combination of an improved DBNet and an improved convolutional recurrent neural network CRNN is proposed. The algorithm has the advantages of high detection accuracy, strong stability, fast speed and high efficiency, and can perform non-contact non-destructive testing.

[0004] The present invention solves the technical problem by the following technical solutions:

[0005] An artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components, the steps of the detection method are:

[0006] Step 1) Image acquisition: Use an optical microscope to collect the original image of the printed identification characters on the chip surface to obtain a clear graphic data set sample, resize the graphic data set and divide it into a training set, a validation set and a test set in proportion, which are used for model training validation and testing respectively;

[0007] Step 2), build an improved DBNet character detection network based on attention mechanism;

[0008] 2.1 Image Binarization: Calculate the approximate binary image by generating a probability map and a threshold map;

[0009] 2.2 Construct labels and loss functions, and train to obtain text boxes;

[0010] 2.3 Based on the network formed in 2.1 and 2.2, SK Conv is used to improve the convolution kernel to form an improved DBNet character detection network based on the attention mechanism;

[0011] Step 3) constructing an improved convolutional recurrent neural recognition network based on a spatial transformation network;

[0012] Step 4) merging the detection network of step 2) with the recognition network of step 3) to obtain an optical character recognition network model;

[0013] Step 5) uses the training set to train the optical character recognition model of step 4); uses the verification set to verify the performance of the optical character recognition model in each round of training, takes the optimal model, and saves the optimal model parameters; and uses the test set to test the optimal model and save the test results.

[0014] Moreover, the specific process of step 1) image acquisition is:

[0015] 1.1 Use an optical microscope to collect the original image of the printed identification characters on the chip surface to obtain a clear graphic data set;

[0016] 1.2 Scale the sample of the graphic dataset and normalize it to 480×480 size;

[0017] 1.3 Divide the graph dataset into training set, validation set and test set in the ratio of 8:1:1.

[0018] Moreover, the specific process of the image binarization in step 2.1 is as follows:

[0019] A. Digitizing the grayscale of the image and inputting different proportions into the training set;

[0020] B. After feature extraction, upsampling, and Concat operations, a feature map is obtained, and then a probability map and a threshold map are generated based on this;

[0021] C. Obtain an approximate binary image through binarization operations to solve the problem of non-differentiable gradients.

[0022] Moreover, the specific process of the binarization operation in step C is shown in formula (1):

[0023]

[0024] In the formula, is the output binary image, i, j are the position coordinates of the image pixels, P is the predicted probability, T is the predicted threshold map, and k is the enhancement factor.

[0025] Moreover, the specific process of constructing labels and loss functions in step 2.2 and training the text box is as follows:

[0026] The probability map and the approximate binary map use a shrinking method to construct labels, and the shrinkage offset D is calculated based on the image perimeter L and area A:

[0027]

[0028] Where r is the shrinkage factor, which is set to 0.4 in this paper;

[0029] The constructed loss function is:

[0030] L=L b +α×L s +β×L t (3)

[0031] Where L b is the loss of the approximate binary image, L s is the probability map loss, L t is the threshold map loss, α and β are weight coefficients.

[0032] Moreover, the specific process of using SK Conv to improve the convolution kernel in step 2.3 is as follows: during the sampling process, each convolution block contains 3×3 regular convolution, BN, and ReLU operations, and SK convolution is used to replace the 3×3 regular convolution in the basic network, so that the model can adaptively adjust the convolution kernel size according to the size of ROI, so as to accurately extract the feature information of the characters on the chip surface and improve the overall detection accuracy of the model.

[0033] Moreover, the specific process of step 3) constructing an improved convolutional recurrent neural network based on a spatial transformation network is as follows: adding an STN module before CNN so that the network can adaptively learn how to perform spatial transformation so that the recognition model has higher robustness to text deformation, and replacing the VGG structure convolution layer used for feature extraction in the original CRNN with ResNet50 so that the network can learn more complex features while solving the problems of gradient disappearance and gradient explosion during training.

[0034] Moreover, the specific process of step 5) using the test set to test the optimal model and saving the test structure and the optimal network model parameters is as follows: setting the batch training number batch to 32, the learning rate to 0.001, using the model for training, the number of training epochs to 500 rounds, taking the optimal result, and verifying the detection ability of the model in this paper by comparing with the original network, and comparing the test results of the optimal network model with the original network; selecting the mIoU indicator to evaluate the text positioning ability of the model, and selecting the accuracy indicator to evaluate the detection ability of the model.

[0035] The beneficial effects of the present invention are:

[0036] 1. The present invention discloses an artificial intelligence recognition method for optically printed characters on the surface of plastic-sealed electronic components. The system considers solving the problem of low efficiency of manual recognition of surface identification of electronic components during chip screening and detection, adopts artificial intelligence means, and adopts a single-chip optical character recognition (OCR) model of plastic-sealed electronic components based on a convolutional neural network. The model is a single-chip optical character recognition algorithm based on a combination of an improved differentiable binarization network (DBNet) and an improved convolutional recurrent neural network (CRNN). A selective convolution kernel (SK) is introduced into the feature extraction network of DBNet to effectively improve its text positioning capability; a spatial transform network (STN) is introduced into CRNN to enhance its recognition capability of tilted and distorted text. The model provides a new artificial intelligence recognition technology solution for fast and accurate single-chip optical character recognition of plastic-sealed electronic components.

[0037] 2. The artificial intelligence recognition method for optically printed characters on the surface of plastic-packaged electronic components of the present invention aims to solve the problem of high difficulty and low efficiency in manual recognition of printed identification characters on the chip surface during the screening and detection of plastic-packaged electronic components. A single-chip optical character recognition algorithm based on a combination of an improved DBNet and an improved convolutional recurrent neural network CRNN is proposed. SK is used to improve DBNet so that it has an attention mechanism, so that the model can pay attention to more important information during feature extraction and improve the feature representation ability of the model in the spatial dimension. The STN module is used to improve the CRNN network so that the model can adaptively learn how to perform spatial transformation and improve the recognition ability of distorted and tilted symbols. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1It is a schematic diagram of the overall process of the present invention;

[0039] Figure 2 This is a schematic diagram of the DBNet network model;

[0040] Figure 3 It is a schematic diagram of the SK Conv structure;

[0041] Figure 4 Schematic diagram of the convolution block structure before and after improvement;

[0042] Figure 5 Schematic diagram of the improved CRNN structure. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0044] An artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components, the overall flow chart of which is as follows: Figure 1 As shown, the specific steps are as follows:

[0045] Step 1) Image acquisition:

[0046] 1.1 Use an optical microscope to collect the original image of the printed identification characters on the chip surface to obtain a clear graphic data set;

[0047] 1.2 Scale the sample of the graphic dataset and normalize it to 480×480 size;

[0048] 1.3 Divide the graph dataset into training set, validation set and test set in the ratio of 8:1:1.

[0049] The camera used to obtain images in the present invention adopts a Leica S9i stereo microscope, uses an integrated CMOS camera, has a 10 million pixel resolution, a 37.6 mm field of view, and an optical resolution of 250 Ip / mm. The images used in the present invention are all collected from the detection and screening test, a total of 1,800 images, and the size is 1920×1080 pixels.

[0050] Step 2), build an improved DBNet character detection network based on attention mechanism;

[0051] 2.1 Image Binarization: Calculate the approximate binary image by generating a probability map and a threshold map;

[0052] A. Digitizing the grayscale of the image and inputting different proportions into the training set;

[0053] B. After feature extraction, upsampling, and Concat operations, a feature map is obtained, and then a probability map and a threshold map are generated based on this;

[0054] C. Obtain an approximate binary image through binarization operations to solve the problem of non-differentiable gradients.

[0055] DBNet is a network that implements text detection based on a segmentation method. DBNet first performs classification at the pixel level, first determining whether each pixel belongs to a text target in turn, so as to obtain a probability map of the text area, and then obtains the surrounding of the text segmentation area through post-processing. The image grayscale is digitized and input into the training set at different proportions. After feature extraction, upsampling, Concat and other operations, the feature map is obtained, and then the probability map and threshold map are generated. After that, the approximate binary map is obtained through binarization operation, and finally the approximate binary map is post-processed to obtain the text prediction area frame map. Therefore, DBNet can introduce the binarization operation into the segmentation network and train it at the same time, so that the threshold of each individual pixel is adaptively predicted, the problem of non-differentiable gradients is solved, the front end and background of the image are easily distinguished, and the detection performance of the entire model is finally improved. The DBNet network model is as follows: Figure 2 The binarization process in DBNet is differentiable, and the function is shown in formula (1).

[0056] The specific process of the binarization operation in step C is shown in formula (1):

[0057]

[0058] In the formula, is the output binary image, i, j are the position coordinates of the image pixels, P is the predicted probability, T is the predicted threshold map, and k is the enhancement factor.

[0059] This allows the model to generate clearer prediction results during the optimization process. At the same time, when a sample is identified incorrectly, the loss value will increase, making the gradient descent clearer. Generally, k=50 is set.

[0060] 2.2 Construct labels and loss functions, and train to obtain text boxes;

[0061] Step 2.2 builds labels and loss functions. The specific process of training to obtain text boxes is as follows:

[0062] The probability map and the approximate binary map use a shrinking method to construct labels, and the shrinkage offset D is calculated based on the image perimeter L and area A:

[0063]

[0064] Where r is the shrinkage factor, which is set to 0.4 in this paper.

[0065] The loss function of this model is:

[0066] L=L b +α×L s +β×L t (3)

[0067] Where L b is the loss of the approximate binary image, L s is the probability map loss, L t is the threshold map loss, α and β are weight coefficients.

[0068] 2.3 Based on the network formed in 2.1 and 2.2, SK Conv is used to improve the convolution kernel to form an improved DBNet character detection network based on the attention mechanism;

[0069] The specific process of using SKConv to improve the convolution kernel in step 2.3 is as follows: during the sampling process, each convolution block contains 3×3 regular convolution, BN, and ReLU operations. SK convolution is used to replace the 3×3 regular convolution in the basic network, so that the model can adaptively adjust the convolution kernel size according to the size of ROI to accurately extract the feature information of the characters on the chip surface and improve the overall detection accuracy of the model.

[0070] In the process of model feature extraction, in order to make DBNet focus on more useful information, the present invention introduces a selective convolution kernel optimization model in the feature extraction process, and uses SK Conv to improve the 3×3 conventional convolution kernel in DBNet to enhance the generalization ability of the model. SK Conv is essentially an attention mechanism based on the convolution kernel, which can adaptively adjust the receptive field according to multiple scales of the input features, while taking into account the dynamic convolution kernels of targets of different scales. The basic principle of SK Conv is as follows: Figure 3 shown.

[0071] First, define the input feature map X dimension as H×W×C, and expand it into two branches through convolution, BN, and ReLU respectively. and The convolution kernel sizes are 3×3 and 5×5. The 5×5 regular convolution can be replaced by a 1×1 dilated convolution with a dilation size of 2, which can reduce the amount of calculation while ensuring the receptive field. Finally, two feature maps with different receptive fields are obtained. and

[0072] The feature maps of the two branches are fused by element-by-element summation, as shown in formula (4). Global Average Pooling (GAP) is performed to obtain the global information of each channel, as shown in formula (5). The S obtained by the above operation is C ∈R C is a one-dimensional feature vector, extracting U CThe whole information of C channels has a global receptive field. The next step is to use the nonlinear transformation (BN+ReLU) of the fully connected operation to further reduce the dimension, as shown in formula (6). The BN and ReLU operations are represented by β and δ respectively, W∈R d×C , d = max(C / r,L) is the drop rate, r is called the reduction factor, the smaller r is, the larger the number of parameters is. In this paper, r = 16 is taken to balance the accuracy and complexity of the model. Finally, Z is the fusion compressed feature vector.

[0073]

[0074] Z=F fc (S) = δ(β(WS)) (6)

[0075] Based on the compressed feature Z, cross-channel adaptive feature map selection can be achieved and The weights of the two branches are extracted from Z after the softmax operation, as shown in formula (7). c +b c =1, where A,B∈R C×d , A C ∈R 1×d is the Cth channel feature map of A, a c is the Cth element of a. According to formula (8), the final feature map V is obtained by weighting the attention of each branch, where V = [V 1 ,V 2 ,…V C ],V C ∈R H×W .

[0076]

[0077] In summary, for the same input, SK convolution assigns different feature map receptive field weights, forming an attention mechanism that adaptively adjusts the receptive field. This paper uses SK convolution to replace the 3×3 regular convolution in the basic network, such as Figure 4 As shown, the model can adaptively adjust the convolution kernel size according to the size of the ROI to accurately extract the feature information of the characters on the chip surface and improve the overall detection accuracy of the model.

[0078] Step 3) Construct an improved convolutional recurrent neural recognition network based on the spatial transformer network; the specific process is: add the STN module before the CNN, so that the network can adaptively learn how to perform spatial transformation so that the recognition model has higher robustness to text deformation, and replace the VGG structure convolution layer used for feature extraction in the original CRNN with ResNet50, so that the network can learn more complex features while solving the problems of gradient disappearance and gradient explosion during training.

[0079] The CRNN algorithm is a text recognition model that combines a convolutional neural network with a long short-term memory neural network, in which CNN is responsible for learning the spatial morphological features of the image, and LSTM is responsible for learning the contextual time series features. Finally, the results are translated and output as the final prediction results. In order to optimize the shortcomings of the original CRNN algorithm in its ability to recognize tilted and distorted characters in images, the present invention adds an STN module before CNN. STN is a neural network module that enables the network to adaptively learn how to perform spatial transformations, so that the recognition model has higher robustness to text deformation. The convolution layer used for feature extraction in the original CRNN uses the VGG structure. The present invention replaces it with ResNet50, so that the network can learn more complex features. In addition, the residual connection can solve the problems of gradient disappearance and gradient explosion during training. The improved CRNN structure of the present invention is as follows: Figure 5 shown.

[0080] Step 4) merging the detection network of step 2) with the recognition network of step 3) to obtain an optical character recognition model;

[0081] Step 5) Use the training set to train the optical character recognition model of step 4); use the validation set to verify the performance of the optical character recognition model in each round of training, take the optimal model, and save the optimal model parameters; and use the test set to test the optimal model, and save the test structure and the optimal network model parameters. Set the batch training number batch to 32, the learning rate to 0.001, use the model training, the training epoch number is 500 rounds, take the optimal result, verify the detection ability of the model in this paper by comparing with the original network, and compare the test results of the optimal network model with the original network; select the mIoU indicator to evaluate the text positioning ability of the model, and select the accuracy indicator to evaluate the detection ability of the model.

[0082] The model training environment of the present invention is CPU Core i9-9900X 3.5GHz, GPU NvidiaRTX2080Ti(11GB)×4, Ubuntu 16.04 operating system, Pytorch deep learning framework. A total of 1800 original images were collected by the camera, which were divided into training set, validation set, and test set according to 8:1:1. The samples were normalized to 480×480 size as model input, the batch training number was set to 32, the learning rate was 0.001, and the model training mentioned above was used. The number of epochs was 500, and the best result was taken. The detection ability of the model in this paper was verified by comparing it with the original network.

[0083] The test results of the best network model of the present invention are compared with the original network, as shown in Table 1 and Table 2. Through the comparison, it can be seen that the present method has achieved a more superior detection accuracy both in the positioning stage and in the detection stage, which is better than the existing algorithms.

[0084] Table 1 Test results of the positioning model of the present invention

[0085]

[0086] Table 2 Test results of the detection model of the present invention

[0087]

[0088] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. An artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components, characterized in that: The steps of this detection method are: Step 1) Image acquisition: Use an optical microscope to collect the original image of the printed identification characters on the chip surface to obtain a clear graphic data set sample, resize the graphic data set and divide it into a training set, a validation set and a test set in proportion, which are used for model training validation and testing respectively; Step 2), build an improved DBNet character detection network based on attention mechanism; 2.1 Image Binarization: Calculate the approximate binary image by generating a probability map and a threshold map; 2.2 Construct labels and loss functions, and train to obtain text boxes; 2.3 Based on the network formed in 2.1 and 2.2, SK Conv is used to improve the convolution kernel to form an improved DBNet character detection network based on the attention mechanism; Step 3) constructing an improved convolutional recurrent neural recognition network based on a spatial transformation network; Step 4) merging the detection network of step 2) with the recognition network of step 3) to obtain an optical character recognition network model; Step 5) uses the training set to train the optical character recognition model of step 4); uses the verification set to verify the performance of the optical character recognition model in each round of training, takes the optimal model, and saves the optimal model parameters; and uses the test set to test the optimal model and save the test results.

2. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 1, characterized in that: The specific process of step 1) image acquisition is as follows: 1.1 Use an optical microscope to collect the original image of the printed identification characters on the chip surface to obtain a clear graphic data set; 1.2 Scale the sample of the graphic dataset and normalize it to 480×480 size; 1.3 Divide the graph dataset into training set, validation set and test set in the ratio of 8:1:

1.

3. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 1, characterized in that: The specific process of image binarization in step 2.1 is as follows: A. Digitizing the grayscale of the image and inputting different proportions into the training set; B. After feature extraction, upsampling, and Concat operations, a feature map is obtained, and then a probability map and a threshold map are generated based on this; C. Obtain an approximate binary image through binarization operations to solve the problem of non-differentiable gradients.

4. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 3, characterized in that: The specific process of the binarization operation in step C is shown in formula (1): In the formula, is the output binary image, i, j are the position coordinates of the image pixels, P is the predicted probability, T is the predicted threshold map, and k is the enhancement factor.

5. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 4, characterized in that: The specific process of constructing labels and loss functions and training text boxes in step 2.2 is as follows: The probability map and the approximate binary map use a shrinking method to construct labels, and the shrinkage offset D is calculated based on the image perimeter L and area A: Where r is the shrinkage factor, which is set to 0.4 in this paper; The constructed loss function is: L=L b +α×L s +β×L t (3) Where L b is the loss of the approximate binary image, L s is the probability map loss, L t is the threshold map loss, α and β are weight coefficients.

6. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 1, characterized in that: The specific process of using SK Conv to improve the convolution kernel in step 2.3 is as follows: during the sampling process, each convolution block contains 3×3 regular convolution, BN, and ReLU operations, and SK convolution is used to replace the 3×3 regular convolution in the basic network, so that the model can adaptively adjust the convolution kernel size according to the size of ROI, so as to accurately extract the feature information of the characters on the chip surface and improve the overall detection accuracy of the model.

7. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 1, characterized in that: The specific process of step 3) constructing an improved convolutional recurrent neural network based on a spatial transformation network is as follows: adding an STN module before CNN so that the network can adaptively learn how to perform spatial transformation so that the recognition model has higher robustness to text deformation, and replacing the VGG structure convolution layer used for feature extraction in the original CRNN with ResNet50 so that the network can learn more complex features while solving the problems of gradient vanishing and gradient exploding during training.

8. The artificial intelligence recognition method for optically printed characters on the surface of plastic-encapsulated electronic components according to claim 1, characterized in that: The specific process of step 5) using the test set to test the optimal model and save the test structure and the optimal network model parameters is as follows: setting the batch number of training batch to 32, the learning rate to 0.001, using the model for training, the number of training epochs to 500 rounds, taking the optimal result, and verifying the detection ability of the model in this paper by comparing it with the original network, and comparing the test results of the optimal network model with the original network; The mIoU indicator is selected to evaluate the text positioning ability of the model, and the accuracy indicator is selected to evaluate the detection ability of the model.

Images