Chip pin rusty spot identification method and device and storage medium

Abstract

The invention provides a chip pin rusty spot identification method and device and a storage medium, and the method comprises the steps: carrying out the multi-angle shooting of a chip pin through machine vision, and obtaining a chip pin image; inputting the gray value of the pixel of the chip pin image into a chip pin rusty spot identification and classification model, and determining the category to which the chip pin image belongs; wherein the chip pin rust recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; and determining whether the chip pin image has rusty spots or not according to the category to which the chip pin image belongs. Compared with the prior art, the chip pin rust spot recognition and classification model is constructed based on the convolutional neural network model and the hidden Markov model, the automation degree is high, detection of a large number of chip pins can be completed in a short time, and the chip pin rust spot recognition and classification method is suitable for rapid quality detection in large-scale production; and the accuracy of judging the rust spots of the chip pins is obviously improved.

Description

Technical Field

[0001] The present disclosure relates to the field of semiconductor technology, and in particular, to a method, apparatus, and storage medium for identifying rust spots on chip pins. Background Art

[0002] Batch purchasing is a common way to reduce the chip procurement cost. Under vacuum drying and airtight conditions, chips can usually be stored for a long time. However, if stored improperly, especially when the packaging bag is damaged, storing chips for a long time will cause rust spots on the chip pins. In the printed circuit board assembly process, due to the chip bonding and soldering process, it is difficult to meet the requirements of the production line by manually selecting chips with damaged packaging bags one by one. Therefore, using machine vision to identify rust spots on chip pins is a necessary screening measure. Machine vision technology obtains the image information of chip pins through an optical imaging device and analyzes and processes the image with the help of computer algorithms to achieve the identification of rust spots. However, traditional machine vision methods have obvious limitations in identifying rust spots on chip pins, and the recognition accuracy is relatively low. This is mainly because the rust spot shapes and sizes of chip pins are different, and under different lighting conditions, the image features change complexly. Traditional machine vision algorithms often have difficulty in comprehensively and accurately extracting the feature information of rust spots, resulting in misjudgment and missed judgment cases from time to time. Therefore, there is an urgent need to provide a method for identifying rust spots on chip pins to improve the accuracy of identifying rust spots on chip pins. Summary of the Invention

[0003] Embodiments of the present disclosure provide a method, apparatus, and storage medium for identifying rust spots on chip pins to solve the problem of low accuracy of existing machine vision algorithms in identifying rust spots on chip pins.

[0004] Based on the above problems, in a first aspect, a method for identifying rust spots on chip pins provided by an embodiment of the present disclosure includes: Taking multi-angle photos of chip pins using machine vision to obtain chip pin images; Inputting the gray values of the pixels of the chip pin images into a chip pin rust spot recognition and classification model to determine the category to which the chip pin images belong; wherein, the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; Determining whether there are rust spots on the chip pin images according to the category to which the chip pin images belong.

[0005] In combination with the first aspect, in a possible implementation manner, the inputting the gray values of the pixels of the chip pin images into a chip pin rust spot recognition and classification model to determine the category to which the chip pin images belong includes: Input the gray values of the pixels of the chip pin image into a convolutional neural network model for feature extraction to obtain a first number of two-dimensional rust feature matrices; Convert the first number of two-dimensional rust feature matrices into a first number of hidden Markov chains by using a Peano scan; Input the first number of hidden Markov chains into a hidden Markov model to determine the category to which the chip pin image belongs.

[0006] Combined with the first aspect, in a possible implementation manner, the inputting the gray values of the pixels of the chip pin image into a convolutional neural network model for feature extraction to obtain a first number of two-dimensional rust feature matrices includes: Input the gray values of the pixels of the chip pin image into the input layer of the convolutional neural network model; Perform feature extraction on the gray values of the pixels of the chip pin image through a preset number of convolutional layers, activation functions, and pooling layers in the convolutional neural network model, and output a first number of two-dimensional rust feature matrices through the output layer; Among them, the convolutional neural network model uses the ReLU function as the activation function and the Softmax function as the output layer.

[0007] Combined with the first aspect, in a possible implementation manner, the inputting the first number of hidden Markov chains into a hidden Markov model to determine the category to which the chip pin image belongs includes: Determine the first number of hidden Markov chains as the observables of the first number of rust features of the hidden Markov model; According to the observables of the first number of rust features, determine the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust features; Use the expectation-maximization algorithm to iteratively determine the parameters of the hidden Markov model until the iteration stop condition is met; in each iteration process, according to the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust features and the parameters of the current-generation hidden Markov model, use the Bayesian marginal posterior mode to determine the probability that the hidden state is the category to which the chip pin image belongs.

[0008] Combined with the first aspect, in a possible implementation manner, the parameters of the hidden Markov model include: a state probability vector, a state transition probability matrix, and an observation probability matrix; The using the Bayesian marginal posterior mode to determine the probability that the hidden state is the category to which the chip pin image belongs according to the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust features and the parameters of the current-generation hidden Markov model includes: Determine the forward probability and backward probability of the Bayesian marginal posterior mode according to the product of conditional probabilities of the first number of rust spot features under the hidden state transition probability matrix, the state probability vector, the state transition probability matrix, and the observation probability matrix in the parameters of the current-generation hidden Markov model; Determine the probability that the hidden state is the category to which the chip pin image belongs according to the product of the forward probability and the backward probability.

[0009] Combined with the first aspect, in a possible implementation manner, the step of using the expectation-maximization algorithm to iteratively determine the parameters of the hidden Markov model until the iteration stop condition is met includes: According to the observables and hidden variables of the first number of rust spot features, use the expectation-maximization algorithm to iteratively determine the covariance value of the expected first number of rust spot feature observables and the parameters of the current-generation hidden Markov model until the iteration stops when the covariance value of the expected first number of rust spot feature observables is less than or equal to a first preset value; Wherein, the hidden variables include: the state transition probability matrix, and the covariance value of the expected first number of rust spot feature observables.

[0010] Combined with the first aspect, in a possible implementation manner, the step of determining whether there is a rust spot on the chip pin image according to the category to which the chip pin image belongs includes: Determine the linear weighted value of the observation probability matrix in the parameters of the corresponding hidden Markov model according to the maximum value of the probability that the hidden state is the category to which the chip pin image belongs; When the linear weighted value is greater than a second preset value, determine whether there is a rust spot on the chip pin image according to the category to which the chip pin image belongs corresponding to the maximum value of the probability that the hidden state is the category to which the chip pin image belongs; The categories to which the chip pin image belongs include: there is a rust spot on the chip pin image, there is no rust spot on the chip pin image; and / or, the level of the rust spot on the chip pin image.

[0011] Combined with the first aspect, in a possible implementation manner, the chip pin rust spot recognition and classification model is trained in the following manner: Screen the chip pin images to determine the training data of the chip pin rust spot recognition and classification model; Input the gray values of the pixels of the training data into the chip pin rust spot recognition and classification model to determine the recall rate, precision, F1 score, and accuracy of the chip pin rust spot recognition and classification model; When the recall rate is greater than or equal to a third preset value, the precision is greater than or equal to a fourth preset value, the F1 score is greater than or equal to a fifth preset value, and the accuracy is greater than or equal to a sixth preset value, the training of the chip pin rust spot recognition and classification model is completed.

[0012] In a second aspect, a chip pin rust spot recognition device is provided, including: An image acquisition module, configured to take multi-angle pictures of chip pins by using machine vision to obtain chip pin images; A chip pin rust spot recognition module, configured to input the gray values of the pixels of the chip pin image into a chip pin rust spot recognition and classification model to determine the category to which the chip pin image belongs; wherein, the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; according to the category to which the chip pin image belongs, it is determined whether there is a rust spot on the chip pin image.

[0013] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the chip pin rust spot recognition method described in the first aspect or any possible implementation manner combined with the first aspect.

[0014] The beneficial effects of the embodiments of the present disclosure include: The chip pin rust spot recognition method, device and storage medium provided by the present disclosure include: taking multi-angle pictures of chip pins by using machine vision to obtain chip pin images; inputting the gray values of the pixels of the chip pin images into a chip pin rust spot recognition and classification model to determine the category to which the chip pin images belong; wherein, the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; according to the category to which the chip pin images belong, it is determined whether there is a rust spot on the chip pin images. The chip pin rust spot recognition method provided by the embodiments of the present disclosure can image chip pins with high resolution and high precision by using machine vision technology. Compared with the prior art, the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model, with a high degree of automation, can complete the detection of a large number of chip pins in a short time, greatly improves the detection efficiency, is suitable for rapid quality inspection in mass production, and the accuracy of chip pin rust spot judgment is significantly improved. Description of the Drawings

[0015] Figure 1 It is a flowchart of the chip pin rust spot recognition method provided by the embodiments of the present disclosure; Figure 2 It is a schematic diagram of recall rate comparison provided by the embodiments of the present disclosure; Figure 3 It is a schematic diagram of precision comparison provided by the embodiments of the present disclosure; Figure 4 Schematic diagram for comparing F1 scores provided by embodiments of the present disclosure; Figure 5 Schematic diagram for comparing accuracies provided by embodiments of the present disclosure; Figure 6 Structural diagram of an identification device for rust spots on chip pins provided by embodiments of the present disclosure. Detailed implementation manners

[0016] Embodiments of the present disclosure provide a method, device, and storage medium for identifying rust spots on chip pins. The preferred embodiments of the present disclosure are described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0017] Embodiments of the present disclosure provide a method for identifying rust spots on chip pins, as Figure 1 shown, including: S101. Take multi-angle photos of the chip pins using machine vision to obtain chip pin images; S102. Input the gray values of the pixels of the chip pin images into a chip pin rust spot identification and classification model to determine the category to which the chip pin images belong; wherein, the chip pin rust spot identification and classification model is constructed based on a convolutional neural network model and a hidden Markov model; S103. Determine whether there are rust spots on the chip pin images according to the category to which the chip pin images belong.

[0018] Embodiments of the present disclosure are applied to the field of semiconductor technology, especially in the printed circuit board assembly process, for the recognition and classification of chip pin rust spots. In the modern electronics industry, as the core component, the quality and performance of chips directly affect the quality of electronic products. Batch procurement, as an effective means to reduce the procurement cost of chips, is widely adopted by many enterprises. In a vacuum-dried and airtight environment, chips can be stored for a long time with stable performance. However, during the actual storage process, once improper storage occurs, especially when the chip packaging bag is damaged, rust spots are likely to form on the chip pins after long-term storage. As the key part connecting the chip to the external circuit, in the printed circuit board assembly engineering (PCBA), due to the chip bonding and soldering process, the method of manually selecting chips with damaged packaging bags one by one is inefficient and difficult to meet the production requirements of large scale and high speed. Therefore, introducing machine vision technology to identify chip pin rust spots has become a necessary screening measure to improve production efficiency and product quality. Machine vision technology obtains the image information of chip pins through optical imaging devices and analyzes and processes the images with machine vision algorithms to achieve the recognition of rust spots. However, traditional machine vision methods have obvious limitations in the recognition of chip pin rust spots, with relatively low recognition accuracy. This is mainly because the rust spot shapes and sizes of chip pins are diverse, and under different lighting conditions, the image features change complexly. Traditional machine vision algorithms often struggle to comprehensively and accurately extract the feature information of rust spots, resulting in frequent misjudgments and missed judgments.

[0019] In the embodiments of the present disclosure, machine vision is used to capture multi-angle images of chip pins, and clear images of chip pins can be obtained. Machine vision captures images of chip pins from different angles through a high-resolution industrial camera, a multi-axis motion control system, and an optical lighting system. Exemplarily, machine vision adjusts the position of the chip through a rotating platform or a robotic arm, such as 0°, 45°, 90°, etc., to eliminate the blind spots of a single perspective and perform multi-angle imaging of the chip pins. For the reflective surface of the chip, clear images are synthesized through multiple exposures to enhance the contrast between the chip pins and the background. For the chip pin image, a pixel is the smallest unit of the image, representing a point in the image. In the chip pin image, each pixel has one or more numerical values to describe its color and brightness information. The chip pin image can include grayscale images and color images. For a grayscale image, each pixel has only one numerical value, i.e., the grayscale value. The grayscale value is the brightness information of the pixel. For example, it is usually represented by an integer between 0 and 255. In a grayscale image, 0 represents pure black, 255 represents pure white, and intermediate values represent different shades of gray. The range of grayscale values depends on the bit depth of the image. For example, the grayscale value range of an 8-bit grayscale image is 0 to 255, while the grayscale value range of a 16-bit grayscale image is 0 to 65535. The higher the grayscale value, the brighter the pixel; the lower the grayscale value, the darker the pixel. A grayscale image is a single-channel image, and each pixel has only one grayscale value. For a color image, such as an RGB image, each pixel has three channels, namely the red, green, and blue channels, and each channel has a corresponding grayscale value. The color image can also be converted into a grayscale image. For example, the grayscale values of the three channels are combined into a single grayscale value Gray through a formula. The formula is as follows: Gray = 0.299×R + 0.587×G + 0.114×B where R, G, and B respectively represent the grayscale values of the red, green, and blue channels. The grayscale value can be used to identify and process different regions in the image. In chip pin detection, the grayscale value can be used to extract features such as the shape and texture of the pins. For example, the edge of the pin can be detected by calculating the change in the grayscale value, thereby determining the position and shape of the pin. For the detection of chip pin rust spots, the change in the grayscale value can reflect the rust situation on the pin surface. The grayscale value of the rust spot area usually differs from that of the normal pin area, and the rust spots of the chip pins can be identified by analyzing the distribution of the grayscale values. Before inputting the grayscale value of the chip pin image pixels into the chip pin rust spot recognition and classification model, preprocessing can also be performed on the chip pin image, including grayscale conversion, denoising, contrast enhancement, etc. Grayscale conversion is to convert a color image into a grayscale image. Denoising can reduce the noise interference in the image, and contrast enhancement can make the difference in grayscale values between the pin area and the background area more obvious.

[0020] Further, input the gray value of the chip pin image pixels into the chip pin rust recognition and classification model. The chip pin rust recognition and classification model is constructed based on the convolutional neural network model and the hidden Markov model. The convolutional neural network model (CNN, Convolutional Neural Networks) has strong image feature extraction capabilities and can automatically learn complex features in images. The hidden Markov model (HMM, Hidden Markov Model) can model the temporal or spatial correlations in the image sequence. The combination of the two can more accurately identify the rust on the chip pin image. After being processed by the chip pin rust recognition and classification model, the chip pin image can be classified into different categories to obtain the category to which the chip pin image belongs. The categories to which the pin image belongs can include: the chip pin image has rust; the chip pin image has no rust; or the level of rust on the chip pin image, such as severe rust, mild rust, no rust, etc. Based on the category to which the chip pin image belongs, it is thus determined whether there is rust on the chip pin.

[0021] In the embodiment of the present application, machine vision technology can be used to achieve high-resolution and high-precision imaging of chip pins. On this basis, the constructed chip pin rust recognition and classification model integrates the convolutional neural network model and the hidden Markov model and has highly automated characteristics. The chip pin rust recognition and classification model can complete the detection of a large number of chip pins in an extremely short time, significantly improving the detection efficiency and particularly meeting the rapid quality detection requirements in large-scale production scenarios. Moreover, the chip pin rust recognition and classification model has achieved a significant improvement in the accuracy of chip pin rust judgment, providing a strong guarantee for chip quality control.

[0022] In another embodiment of the present disclosure, in the above step S102, inputting the gray value of the chip pin image pixels into the chip pin rust recognition and classification model to determine the category to which the chip pin image belongs includes the following steps: Step 1: Input the gray value of the chip pin image pixels into the convolutional neural network model for feature extraction to obtain the first number of two-dimensional rust feature matrices; Step 2: Use Peano scanning to convert the first number of two-dimensional rust feature matrices into the first number of hidden Markov chains; Step 3: Input the first number of hidden Markov chains into the hidden Markov model to determine the category to which the chip pin image belongs.

[0023] In the embodiments of the present disclosure, the gray values of the chip pin images are input into a convolutional neural network model to extract a two-dimensional rust feature matrix, which is converted into a hidden Markov chain through Peano scanning and then input into a hidden Markov model for recognition and classification. For the above step 1, first, the gray values of the pixels of the chip pin images captured by machine vision are input into the convolutional neural network model. Exemplarily, the convolutional neural network model can construct the number of input layer nodes corresponding to the number of pixel gray values. The input layer sends the gray values to the convolutional layer, and the convolutional layer extracts the gray features to obtain a feature matrix. After being activated by the activation function, the feature matrix is sent to the pooling layer. The pooling layer condenses the data of the feature matrix and generates the first number of two-dimensional rust feature matrices according to the preset number of features, and the output layer outputs this matrix. For the above step 2, Peano scanning is used to convert the first number of two-dimensional rust feature matrices into the first number of hidden Markov chains. Peano scanning traverses each point in the two-dimensional space in an orderly manner, and can arrange the elements in the two-dimensional feature matrix into a one-dimensional sequence in a certain order, thereby retaining the spatial structure information in the matrix. For example, a two-dimensional rust feature matrix M has a size of 3×3.

[0024]

[0025] Peano scanning can extract the elements in the matrix M in sequence according to the path of the Peano curve to form a one-dimensional sequence S, and this sequence retains the spatial relationship of the elements in the matrix. For example, the path of Peano scanning is: Path = [1, 2, 3, 6, 5, 4, 7, 8, 9], then the obtained one-dimensional sequence S is S = [1, 2, 3, 6, 5, 4, 7, 8, 9]. The obtained one-dimensional sequence S is used as the hidden Markov chain. Each two-dimensional rust feature matrix corresponds to a hidden Markov chain, and the number of hidden Markov chains is the same as the number of two-dimensional rust feature matrices output by the convolutional neural network model, both being the first number. For example, if the convolutional neural network model outputs N two-dimensional rust feature matrices, where N represents the first number, then Peano scanning is performed on these N two-dimensional rust feature matrices to obtain N hidden Markov chains. For the above step 3, all the hidden Markov chains obtained by Peano scanning have the following characteristics: each hidden Markov chain can be used as an observable quantity of the rust feature, that is, the observable quantity of the hidden Markov model. The first number of hidden Markov chains are input into the hidden Markov model, so that the category to which the chip pin image belongs can be determined. The convolutional neural network model can automatically learn the complex features in the chip pin image, and the hidden Markov model can model the feature sequence, considering the spatio-temporal correlation between the features, so as to accurately classify the chip pin image and determine whether there is rust on the chip pins.

[0026] In another embodiment of the present disclosure, in the above step 1, the gray values of the chip pin image pixels are input into a convolutional neural network model for feature extraction to obtain the first number of two-dimensional rust feature matrices, including the following steps: Step 1: Input the gray values of the chip pin image pixels into the input layer of the convolutional neural network model; Step 2: Perform feature extraction on the gray values of the chip pin image pixels through the convolutional layers, activation functions, and pooling layers with a preset number of layers in the convolutional neural network model, and output the first number of two-dimensional rust feature matrices through the output layer; Among them, the convolutional neural network model uses the ReLU function as the activation function and the Softmax function as the output layer.

[0027] In the embodiment of the present disclosure, a convolutional neural network model is used to extract features from the chip pin image. For the above step 1, the gray values of the chip pin image pixels are input into the input layer of the convolutional neural network model. The input layer is the first layer of the convolutional neural network model. It is used to receive the gray values of the chip pin image pixels and transfer them to the convolutional layer. The number of nodes in the input layer is related to the number of gray values of the chip pin image pixels. Exemplarily, the number of nodes in the input layer = image width × height × number of channels. For a grayscale image, the number of channels is 1, and for a color image, such as an RGB image, the number of channels is 3. For example, for a grayscale image with a resolution of 640×480 pixels, the number of nodes in the input layer = 640×480×1 = 307,200. For a color image with a resolution of 640×480 pixels, the number of nodes in the input layer = 640×480×3 = 921,600. For the above step 2, the convolutional layer is used to find the pin rust feature data from the gray values received from the input layer. The ReLU function is used as the activation function, which is used to convert the linear convolution operation into a non-linear transformation, can fit the complex features of the pin rust, set the low response values in the normal area of the pin to zero, highlight the high response features related to rust, and retain the intensity information of the rust features. The extracted feature data is sent to the pooling layer, and the pooling layer is used to reduce the size of the rust feature matrix while retaining the most critical features. The output layer is used to output the feature matrix of the pooling layer to obtain the first number of two-dimensional rust feature matrices. The Softmax function is used as the output layer, which can include the probability distribution in the output information, making the output of the convolutional neural network model more intuitive and easy to interpret. Each two-dimensional rust feature matrix can represent a feature of the chip pin image, such as a texture feature, a total of the first number of features. Through the above steps, the convolutional neural network model can automatically learn the features in the chip pin image, and these features can be used to accurately judge whether there is rust on the pins.

[0028] In another embodiment of the present disclosure, in the above step 3, inputting the first number of hidden Markov chains into the hidden Markov model to determine the category to which the chip pin image belongs includes the following steps: Step 1: Determine the first number of hidden Markov chains as the observables of the first number of rust spot features of the hidden Markov model; Step 2: Determine the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust spot features; Step 3: Use the expectation-maximization algorithm to iteratively determine the parameters of the hidden Markov model until the iteration stop condition is met; in each iteration process, according to the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust spot features and the parameters of the current generation of the hidden Markov model, use the Bayesian marginal posterior mode to determine the probability that the hidden state is the category to which the chip pin image belongs.

[0029] In the embodiment of the present disclosure, the hidden Markov model is used to process the first number of hidden Markov chains to determine the category to which the chip pin image belongs. The hidden Markov model is a statistical model. The process of determining the category to which the chip pin image belongs can be regarded as the process of solving the hidden state according to the observables of the hidden Markov model. The hidden Markov model includes two main parts: the hidden state sequence, which represents the internal state changes and these states are unobservable. The observable sequence: the observable data generated by the hidden state. In the chip pin image classification, the hidden state can be used to represent the category to which the chip pin image belongs, such as normal or rust spot, and the observable sequence can be derived from the two-dimensional rust spot feature matrix extracted from the chip pin image. For the above step 1, for the hidden Markov model, the first number of hidden Markov chains is a known quantity and can be objective observables. Determine the first number of hidden Markov chains as the observables of the first number of rust spot features of the hidden Markov model, denoted as Y, then Y i represents the i-th observable. The hidden Markov chain is a state variable characterizing the rust spot feature or the description value of each rust spot feature. For the above step 2, X is the hidden state transition probability matrix of the hidden Markov model, and X i is the hidden state transition probability, where i = 1, 2... M, and M is the number of hidden state transitions X i corresponding to each rust spot feature matrix in the hidden Markov model. Then the probability distribution P(X) of the hidden state transition probability X i can be expressed as:

[0030] The hidden state transition probability X i satisfies the Gaussian distribution, then the hidden state transition probability X iThe conditional probability under the observable Y i can be expressed as:

[0031] where represents the conditional probability of the hidden state transition probability matrix X given the observable Y. represents the product of the conditional probabilities of the observable Y and the hidden state transition probability matrix X in the hidden Markov model, that is, the product of the conditional probabilities of the hidden state transition probability matrix under the observables of the first number of rust features. For the above step three, the expectation-maximization algorithm is an iterative optimization algorithm commonly used for the maximum likelihood estimation of the parameters of a probability model with hidden variables. The EM algorithm alternately executes two steps: the E-step (expectation step) and the M-step (maximization step) to gradually optimize the parameters of the hidden Markov model until the iterative stop condition is met. In each iteration, according to the product of the conditional probabilities of the hidden state transition probability matrix under the observables of the first number of rust features and the parameters of the current-generation hidden Markov model, the probability that the hidden state belongs to the category of the chip pin image is determined using the Bayesian marginal posterior mode. The Bayesian marginal posterior mode (MPM) can make the classification method an unsupervised classification method by maximizing the marginal probability of the posterior distribution to determine the most likely value of the parameter. The hidden Markov model combined with the expectation-maximization algorithm and the Bayesian marginal posterior mode can be used to accurately determine the category to which the chip pin image belongs.

[0032] In another embodiment of the present disclosure, the parameters of the hidden Markov model include: a state probability vector, a state transition probability matrix, and an observation probability matrix; In the above step three, according to the product of the conditional probabilities of the hidden state transition probability matrix under the observables of the first number of rust features and the parameters of the current-generation hidden Markov model, using the Bayesian marginal posterior mode to determine the probability that the hidden state belongs to the category of the chip pin image includes: Step (1), according to the product of the conditional probabilities of the hidden state transition probability matrix under the observables of the first number of rust features, the state probability vector, the state transition probability matrix, and the observation probability matrix in the parameters of the current-generation hidden Markov model, determine the forward probability and the backward probability of the Bayesian marginal posterior mode; Step (2), according to the product of the forward probability and the backward probability, determine the probability that the hidden state belongs to the category of the chip pin image.

[0033] In the embodiment of the present disclosure, the Bayesian marginal posterior mode (MPM) can make the classification method an unsupervised classification method and estimate the value of the hidden state transition probability matrix X in the hidden Markov model. is the set of estimated values of the hidden state transition probability matrix X, . is the estimated value. For a hidden Markov chain with the first quantity being N, , all satisfy: , all satisfy:

[0034] where is the set to be classified, dimensionless; is the forward probability; is the backward probability. For the above step (i), according to the product of the conditional probabilities of the hidden state transition probability matrix under the observables of the first quantity of rust spot features, the state probability vector, the state transition probability matrix, and the observation probability matrix in the parameters of the current generation's hidden Markov model, determine the forward probability and the backward probability of the Bayesian marginal posterior mode. The forward probability , the backward probability . The forward probability and the backward probability are recursively calculated according to the following formulas:

[0035]

[0036]

[0037]

[0038] where and are the recursive formulas for the forward probability . represents the joint probability of the initial estimate of the hidden state transition probability and the initial value of the observable . is the recursive formula for the forward probability . is the final value of the backward probability. is the recursive formula for the backward probability to be recursively calculated forward.

[0039] For the above step (ii), , can represent that the estimated value of the hidden state transition probability is the maximum value obtained when the product of the forward probability and the backward probability falls into class. class can represent the probability of the chip pin image belonging to the class. The state probability vector is expressed as , that is, the initial state probability, the state transition probability matrix can be expressed as , the observation probability matrix can be expressed as For any one-dimensional hidden Markov chain expanded from a two-dimensional rust spot feature matrix, the observable Y can be written as: .

[0040]

[0041] where, for the i-th hidden Markov chain, is the forward probability, and t is the t-th step of iteration; is the (t + 1)-th iteration of the backward probability; is the initial state transition probability value of the hidden Markov chain, = , that is, the state transition probability matrix; is the observed probability value of iteration, that is, the observed probability matrix. Under the condition of , the probability value of observing the maximum value of the hidden state transition probability can be calculated using the following formula: =

[0042]

[0043] represents the iterative state probability value of the i-th hidden Markov chain; the iterative calculation formula of the state transition probability of iteration is as follows:

[0044] According to the product of the forward probability and the backward probability, the estimated value of the hidden state transition probability can be classified into the corresponding category in the way of the maximum probability. Thus, the maximum probability of the hidden state belonging to the category of the chip pin image is obtained.

[0045] In another embodiment of the present disclosure, in the above step three, the expectation-maximization algorithm is used to iteratively determine the parameters of the hidden Markov model until the iteration stop condition is satisfied, including: According to the observables and hidden variables of the first number of rust spot features, the expectation-maximization algorithm is used to iteratively determine the covariance value of the observables of the expected first number of rust spot features and the parameters of the current-generation hidden Markov model until the iteration stops when the covariance value of the observables of the expected first number of rust spot features is less than or equal to the first preset value; where, the hidden variables include: the state transition probability matrix, the covariance value of the observables of the expected first number of rust spot features.

[0046] In the embodiments of the present disclosure, iteration is stopped when the covariance value of the observables of the expected first number of rust spot features is less than or equal to a first preset value. The hidden variables include: the state transition probability matrix, and the covariance value of the observables of the expected first number of rust spot features. The state transition probability matrix can be expressed as . The calculation formula for the covariance value of the observables of the expected first number of rust spot features is:

[0047] where represents the mean value of the observables of the expected first number of rust spot features, and its calculation formula is expressed as:

[0048] Based on the observables of the first number of rust spot features and the hidden variables, the expectation-maximization algorithm is used for iteration to determine the covariance value of the observables of the expected first number of rust spot features and the parameters of the current-generation hidden Markov model. The parameters of the current-generation hidden Markov model include the state probability vector expressed as , that is, the initial state probability, the state transition probability matrix can be expressed as , and the observation probability matrix can be expressed as . When the covariance value of the observables of each expected rust spot feature is less than or equal to the first preset value, the number of iterations for the observables of each rust spot feature may be different. Iteration is stopped when the covariance value of the observables of the expected first number of rust spot features is less than or equal to the first preset value.

[0049] In another embodiment of the present disclosure, in step S103 above, determining whether there is rust on the chip pin image according to the category to which the chip pin image belongs includes: Step 1: Determine the linear weighted value of the observation probability matrix in the parameters of the corresponding hidden Markov model according to the maximum value of the probability that the hidden state is the category to which the chip pin image belongs; Step 2: When the linear weighted value is greater than a second preset value, determine whether there is rust on the chip pin image according to the category to which the chip pin image belongs corresponding to the maximum value of the probability that the hidden state is the category to which the chip pin image belongs; The categories to which the chip pin image belongs include: the chip pin image has rust, the chip pin image has no rust; and / or, the level of rust on the chip pin image.

[0050] In the embodiments of the present disclosure, a linear weighted calculation is performed on the observation probability matrix in the parameters of the corresponding hidden Markov model. When the obtained linear weighted value is greater than a second preset value, it is considered that the maximum value of the probability that the hidden state determined by the hidden Markov model belongs to the category of the chip pin image is a valid value. For the above step 1, in the iterative process, the corresponding observation probability matrix can be obtained from the maximum value of the probability that the hidden state belongs to the category of the chip pin image. Since the observation probability values corresponding to the two-dimensional rust feature matrix for the hidden Markov chain, that is, the expected number of iterations of the observable quantity of each rust feature may be different, and the number of iterations corresponding to the maximum value of the probability that the hidden state belongs to the category of the chip pin image may also be different, it is preset as the weight of the observation probability value obtained from the hidden Markov chain corresponding to the first two-dimensional rust feature matrix , where is the observation probability value finally calculated after t1 iterations; is the weight of the observation probability value obtained from the hidden Markov chain corresponding to the second two-dimensional rust feature matrix , where is the observation probability value calculated after t2 iterations; similarly, is the weight of the observation probability value obtained from the hidden Markov chain corresponding to the Nth two-dimensional rust feature matrix , and the first quantity can be represented by N; where

[0051] the linear weighted value of the observation probability matrix in the parameters of the corresponding hidden Markov model is: . For the above step 2, when the linear weighted value is greater than the second preset value, for example, Q>0.5, according to the category to which the chip pin image belongs corresponding to the maximum value of the probability that the hidden state belongs to the category of the chip pin image, it is determined whether there is rust on the chip pin image. The categories to which the pin image belongs may include: there is rust on the chip pin image, there is no rust on the chip pin image; or the level of rust on the chip pin image, such as severe rust, mild rust, no rust, etc.

[0052] In another embodiment of the present disclosure, the chip pin rust recognition and classification model is trained in the following manner: Step 1: Screen the chip pin images to determine the training data of the chip pin rust recognition and classification model; Step 2: Input the gray values of the pixels of the training data into the chip pin rust recognition and classification model to determine the recall rate, precision, F1 score, and accuracy of the chip pin rust recognition and classification model; Step 3: When the recall rate is greater than or equal to the third preset value, the precision is greater than or equal to the fourth preset value, the F1 score is greater than or equal to the fifth preset value, and the accuracy is greater than or equal to the sixth preset value, complete the training of the chip pin rust spot recognition and classification model.

[0053] In the embodiments of the present disclosure, when the recall rate, precision, and F1 score of the chip pin rust spot recognition and classification model meet the requirements, complete the training of the chip pin rust spot recognition and classification model. For the above-mentioned step 1, screen the chip pin images, and the screened images can have typical rust spot characteristics, and determine the screened chip pin images as the training data of the chip pin rust spot recognition and classification model. For the above-mentioned step 2, input the gray values of the pixels of the training data into the chip pin rust spot recognition and classification model, and determine the recall rate, precision, F1 score, and accuracy of the chip pin rust spot recognition and classification model. For the above-mentioned step 3, when the recall rate is greater than or equal to the third preset value, the precision is greater than or equal to the fourth preset value, the F1 score is greater than or equal to the fifth preset value, and the accuracy is greater than or equal to the sixth preset value, complete the training of the chip pin rust spot recognition and classification model. Introduce a comparison model, and the comparison model includes a convolutional neural network model and a convolutional neural network-support vector machine model. The formula for calculating the recall rate is: true positive / (true positive + false negative). As Figure 2 shown, the chip pin rust spot recognition and classification model is much better than the simple convolutional neural network classification model and the convolutional neural network-support vector machine classification model in terms of recall rate. As the number of samples increases, the growth trend of the recall rate of the chip pin rust spot recognition and classification model is not as good as that of the single convolutional neural network, but the recall rate data performs better in the small sample state. The convolutional neural network-support vector machine classification model shows a relatively stable recall rate growth value as the number of training samples increases, and the recall rate in the small sample state is also at a medium level among the three classification models. From the perspective of the recall rate, when the number of samples exceeds 500, the convolutional neural network performs better than the convolutional neural network-support vector machine classification model. The formula for calculating the precision is: true positive / (true positive + false positive). As Figure 3 shown, the performance of the three classification models is relatively in line with expectations, and as the number of samples increases, the precision shows an increasing trend. Among them, the chip pin rust spot recognition and classification model is superior to the simple convolutional neural network model and the convolutional neural network-support vector machine model in terms of precision. The formula for calculating the F1 score is: 2 * precision * recall rate / (precision + recall rate). As Figure 4As shown, as the number of samples increases, the F1 score shows an increasing trend. Among them, the chip pin rust spot recognition and classification model is superior to the simple convolutional neural network model and the convolutional neural network - support vector machine model in terms of the F1 score. As the number of samples increases, the growth rate of the F1 value of the convolutional neural network model relative to the convolutional neural network - support vector machine model becomes larger. The accuracy calculation formula is: (true positives + true negatives) / (true positives + false positives + true negatives + false negatives). As Figure 5 shown, the performance of the three classification models is relatively in line with expectations. The chip pin rust spot recognition and classification model has the highest accuracy; as the number of samples increases, the accuracy of the three classification models all shows an increasing trend. Among them, the convolutional neural network - support vector machine model is superior to the simple convolutional neural network model in terms of accuracy. When the recall rate is greater than or equal to the third preset value, the precision is greater than or equal to the fourth preset value, the F1 score is greater than or equal to the fifth preset value, and the accuracy is greater than or equal to the sixth preset value, the training of the chip pin rust spot recognition and classification model is completed, which can improve the accuracy of chip pin rust spot recognition.

[0054] Based on the same general inventive concept, the embodiments of the present disclosure also provide an apparatus for identifying chip pin rust spots. Since the principles of the problems solved by these apparatuses are similar to those of the foregoing method for determining the inclination angle of a photovoltaic array, the implementation of this apparatus can refer to the implementation of the foregoing method, and the repeated parts will not be elaborated.

[0055] The embodiments of the present disclosure provide an apparatus for identifying chip pin rust spots, as Figure 6 shown, including: An image acquisition module 601, configured to take multi - angle pictures of chip pins using machine vision to obtain chip pin images; A chip pin rust spot recognition module 602, configured to input the gray - scale values of the pixels of the chip pin image into a chip pin rust spot recognition and classification model to determine the category to which the chip pin image belongs; wherein, the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; according to the category to which the chip pin image belongs, determine whether there is a rust spot on the chip pin image.

[0056] In another embodiment of the present disclosure, the chip pin rust spot recognition module 602 is configured to input the gray - scale values of the pixels of the chip pin image into a convolutional neural network model for feature extraction to obtain a first number of two - dimensional rust spot feature matrices; Use Peano scanning to convert the first number of two - dimensional rust spot feature matrices into a first number of hidden Markov chains; Input the first number of hidden Markov chains into a hidden Markov model to determine the category to which the chip pin image belongs.

[0057] In another embodiment of the present disclosure, the recognition module 602 for chip pin rust spots is configured to input the gray values of the pixels of the chip pin image into the input layer of the convolutional neural network model; Feature extraction is performed on the gray values of the pixels of the chip pin image through convolutional layers, activation functions, and pooling layers of a preset number of layers in the convolutional neural network model, and a first number of two-dimensional rust spot feature matrices are output through the output layer; Among them, the convolutional neural network model uses the ReLU function as the activation function and the Softmax function as the output layer.

[0058] In another embodiment of the present disclosure, the recognition module 602 for chip pin rust spots is configured to determine the first number of hidden Markov chains as the observables of the first number of rust spot features of the hidden Markov model; According to the observables of the first number of rust spot features, determine the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust spot features; Use the expectation-maximization algorithm to iteratively determine the parameters of the hidden Markov model until the iteration stop condition is met; in each iteration process, according to the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust spot features and the parameters of the current-generation hidden Markov model, use the Bayesian marginal posterior mode to determine the probability that the hidden state is the category to which the chip pin image belongs.

[0059] In another embodiment of the present disclosure, the parameters of the hidden Markov model include: a state probability vector, a state transition probability matrix, and an observation probability matrix; The recognition module 602 for chip pin rust spots is configured to determine the forward probability and backward probability of the Bayesian marginal posterior mode according to the conditional probability product of the hidden state transition probability matrix under the observables of the first number of rust spot features, the state probability vector, the state transition probability matrix, and the observation probability matrix in the parameters of the current-generation hidden Markov model; According to the product of the forward probability and the backward probability, determine the probability that the hidden state is the category to which the chip pin image belongs.

[0060] In another embodiment of the present disclosure, the recognition module 602 for chip pin rust spots is configured to use the expectation-maximization algorithm to iteratively determine the covariance value of the expected first number of rust spot feature observables and the parameters of the current-generation hidden Markov model according to the observables of the first number of rust spot features and the hidden variables, until the iteration stops when the covariance value of the expected first number of rust spot feature observables is less than or equal to a first preset value; Among them, the hidden variables include: a state transition probability matrix, and covariance values of observable quantities of a first desired number of rust spot features.

[0061] In another embodiment of the present disclosure, the recognition module 602 for chip pin rust spots is configured to determine a linear weighted value of the observation probability matrix in the parameters of the corresponding hidden Markov model according to the maximum value of the probability that the hidden state belongs to the category of the chip pin image; When the linear weighted value is greater than a second preset value, determine whether there are rust spots on the chip pin image according to the category to which the chip pin image belongs corresponding to the maximum value of the probability that the hidden state belongs to the category of the chip pin image; The categories to which the chip pin image belongs include: the chip pin image has rust spots, the chip pin image has no rust spots; and / or, the level of rust spots on the chip pin image.

[0062] In another embodiment of the present disclosure, the recognition module 602 for chip pin rust spots is configured to train the chip pin rust spot recognition and classification model in the following manner: Screen the chip pin images to determine the training data of the chip pin rust spot recognition and classification model; Input the gray values of the pixels of the training data into the chip pin rust spot recognition and classification model to determine the recall rate, precision, F1 score, and accuracy of the chip pin rust spot recognition and classification model; When the recall rate is greater than or equal to a third preset value, the precision is greater than or equal to a fourth preset value, the F1 score is greater than or equal to a fifth preset value, and the accuracy is greater than or equal to a sixth preset value, complete the training of the chip pin rust spot recognition and classification model.

[0063] Based on the same inventive concept, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, it executes the steps of the method for recognizing chip pin rust spots as described in any of the above embodiments.

[0064] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by hardware, or can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present disclosure.

[0065] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing the present disclosure.

[0066] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device of the embodiment according to the description of the embodiment, or can be correspondingly changed to be located in one or more devices different from this embodiment. The modules of the above embodiments can be combined into one module, or further split into multiple sub-modules.

[0067] The serial numbers of the above embodiments of the present disclosure are only for description and do not represent the advantages and disadvantages of the embodiments.

[0068] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure also intends to include these changes and modifications.

Claims

1. A method for identifying rust spots on chip pins, characterized in that: include: Use machine vision to shoot chip pins at multiple angles to obtain chip pin images; Inputting the grayscale value of the chip pin image pixel into the chip pin rust spot recognition and classification model to determine the category to which the chip pin image belongs; wherein the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; According to the category to which the chip pin image belongs, it is determined whether the chip pin image has rust spots.

2. The method according to claim 1, characterized in that The step of inputting the grayscale value of the chip pin image pixel into the chip pin rust spot recognition classification model to determine the category to which the chip pin image belongs includes: Inputting the grayscale values ​​of the chip pin image pixels into a convolutional neural network model for feature extraction to obtain a first number of two-dimensional rust spot feature matrices; Using peano scanning to convert the first number of two-dimensional rust feature matrices into a first number of hidden Markov chains; The first number of hidden Markov chains are input into a hidden Markov model to determine the category to which the chip pin image belongs.

3. The method according to claim 2, characterized in that The step of inputting the grayscale values ​​of the chip pin image pixels into a convolutional neural network model for feature extraction to obtain a first number of two-dimensional rust feature matrices includes: Inputting the grayscale values ​​of the chip pin image pixels into the input layer of the convolutional neural network model; Performing feature extraction on the grayscale values ​​of the pixels of the chip pin image through a preset number of convolutional layers, activation functions, and pooling layers in the convolutional neural network model, and outputting a first number of two-dimensional rust feature matrices through an output layer; The convolutional neural network model uses the ReLU function as the activation function and the Softmax function as the output layer.

4. The method according to claim 2, characterized in that Inputting the first number of hidden Markov chains into a hidden Markov model to determine the category to which the chip pin image belongs includes: Determining the first number of hidden Markov chains as observables of the first number of rust spot features of the hidden Markov model; Determine, according to the observable quantities of the first number of rust spot features, the conditional probability product of the hidden state transition probability matrix under the observable quantities of the first number of rust spot features; The parameters of the hidden Markov model are iteratively determined using an expectation-maximization algorithm until an iteration stop condition is met; in each iteration, the probability that the hidden state is the category to which the chip pin image belongs is determined using a Bayesian marginal posterior model based on the conditional probability product of the hidden state transition probability matrix under the observable measurements of the first number of rust spot features and the parameters of the current generation of the hidden Markov model.

5. The method according to claim 4, characterized in that The parameters of the hidden Markov model include: state probability vector, state transition probability matrix and observation probability matrix; The method of determining the probability that the hidden state is the category to which the chip pin image belongs by using a Bayesian marginal posterior model based on the conditional probability product of the hidden state transition probability matrix under the observable quantity of the first number of rust spot features and the parameters of the current generation hidden Markov model comprises: Determine the forward probability and backward probability of the Bayesian marginal posterior mode according to the conditional probability product of the hidden state transition probability matrix under the observable quantity of the first number of rust spot features, the state probability vector in the parameters of the current generation of the hidden Markov model, the state transition probability matrix and the observation probability matrix; According to the product of the forward probability and the backward probability, the probability that the hidden state is the category to which the chip pin image belongs is determined.

6. The method according to claim 4, characterized in that The method of iteratively determining the parameters of the hidden Markov model using the expectation maximization algorithm until the iteration stop condition is met includes: According to the observable quantities and hidden variables of the first number of rust spot features, an expectation maximization algorithm is used to iteratively determine the covariance value of the observable quantities of the first number of rust spot features and the parameters of the current generation hidden Markov model, until the iteration is stopped when the covariance value of the observable quantities of the first number of rust spot features is less than or equal to a first preset value; The hidden variables include: a state transition probability matrix and covariance values ​​of observable quantities of the expected first number of rust spot features.

7. The method according to claim 5, characterized in that The determining whether the chip pin image has rust spots according to the category to which the chip pin image belongs includes: Determine the linear weighted value of the observation probability matrix in the parameters of the corresponding hidden Markov model according to the maximum value of the probability that the hidden state is the category to which the chip pin image belongs; In the case where the linear weighted value is greater than the second preset value, determining whether the chip pin image has rust spots according to the category to which the chip pin image belongs corresponding to the maximum value of the probability that the hidden state is the category to which the chip pin image belongs; The categories to which the chip pin image belongs include: the chip pin image having rust spots, the chip pin image not having rust spots; and / or the level of the chip pin image having rust spots.

8. The method according to claim 1, characterized in that The chip pin rust spot recognition and classification model is trained in the following manner: Screening the chip pin image to determine the training data of the chip pin rust spot recognition classification model; Inputting the grayscale values ​​of the pixels of the training data into the chip pin rust spot recognition and classification model, and determining the recall rate, precision, F1 score and accuracy of the chip pin rust spot recognition and classification model; When the recall rate is greater than or equal to the third preset value, the precision is greater than or equal to the fourth preset value, the F1 score is greater than or equal to the fifth preset value, and the accuracy is greater than or equal to the sixth preset value, the training of the chip pin rust spot recognition and classification model is completed.

9. A device for identifying rust spots on chip pins, characterized in that: include: An image acquisition module is used to use machine vision to shoot chip pins at multiple angles to obtain chip pin images; The chip pin rust spot recognition module is used to input the grayscale value of the chip pin image pixel into the chip pin rust spot recognition and classification model to determine the category to which the chip pin image belongs; wherein the chip pin rust spot recognition and classification model is constructed based on a convolutional neural network model and a hidden Markov model; and determine whether the chip pin image has rust spots according to the category to which the chip pin image belongs.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying rust spots on chip pins as claimed in any one of claims 1 to 8 is executed.

Images