Electronic component packaging defect detection method and detection system based on image acquisition

Abstract

The invention relates to the field of image processing, in particular to an electronic component packaging defect detection method and system based on image acquisition. The feature extraction sub-operator and the information reduction sub-operator are adopted to form a defect detection algorithm, a standardization module in each sub-operator is in front of an operator composition module, and in addition, short-circuit paths of the input end and the output end of the standardization module in the sub-operator form a short-circuit path of adding post-standardization on a pre-standardization structure; the short-circuit path can establish a stable gradient transmission path, when the number of operator layers of the defect detection algorithm is increased progressively, the defect detection algorithm converges accordingly, and training of the algorithm is more robust. Furthermore, a post-standardized structure in the defect detection algorithm can improve the defect detection quality of the defect detection algorithm, and meanwhile, the universality of the algorithm is enhanced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an electronic component packaging defect detection method and detection system based on image acquisition. Background Art

[0002] As the basic component of modern electronic devices, the quality and reliability of electronic components directly affect the performance of the entire electronic device. With the development of machine vision technology, it has been widely used in the field of electronic component packaging inspection. Machine vision technology obtains images of electronic component packaging through image acquisition equipment, and then uses computer vision algorithms to analyze these images to detect whether there are defects in the packaging. Among them, the establishment and training of defect detection algorithms are the core of achieving accurate detection. When existing machine vision technology is used for electronic component packaging inspection, deep neural networks are a commonly used tool for building defect detection algorithms. Deep neural networks are composed of multiple layers, which can be roughly divided into feature extraction parts and information restoration parts, which are similar to the functions of encoders and decoders. The feature extraction part is used to extract various features from the input electronic component packaging image, while the information restoration part attempts to determine whether there are defects and the type and location of the defects based on the extracted features.

[0003] When constructing a deep neural network, each layer contains a standardized structure. The different positions of these standardized structures will have different effects on the network performance. Among them, although the network with a post-standardized structure performs well in terms of performance and generalization, it has defects in stability. When the number of network layers is large, this post-standardized structure network is prone to collapse during the training process. This is because in the training process of the deep neural network, as the number of network layers increases, the gradient may disappear or explode during the back propagation process, and the post-standardized structure is difficult to effectively alleviate this situation, resulting in unstable network training and affecting the effect of the defect detection algorithm obtained by the final training. Due to the above-mentioned problems of the post-standardized structure network, the defect detection algorithm constructed based on this network has insufficient detection accuracy when detecting defects in electronic component packaging. For example, some minor defects may be missed or some normal packages may be misjudged as defective. At the same time, the universality of the algorithm is also limited, that is, for different types of electronic component packaging or in different production environments, the performance of the algorithm may fluctuate greatly and cannot stably provide accurate detection results. Summary of the invention

[0004] In view of this, the present application provides an electronic component packaging defect detection method and detection system based on image acquisition. The technical solution of the present application is implemented as follows: On the one hand, the present application provides an electronic component packaging defect detection method based on image acquisition, which is applied to a detection system, wherein the detection system is provided with a defect detection algorithm, wherein the defect detection algorithm includes u feature extraction sub-operators arranged in sequence and v information restoration sub-operators arranged in sequence, each of the feature extraction sub-operators and each of the information restoration sub-operators includes a standardization module and an operator composition module arranged in sequence, u is a natural number not less than 2, and v is a natural number not less than 3, and the method includes: acquiring an electronic component packaging image; transmitting the electronic component packaging image to the u feature extraction sub-operators, using each of the u feature extraction sub-operators to perform descriptor mining in sequence to obtain an image descriptor output by the u feature extraction sub-operators; wherein each of the feature extraction sub-operators is a natural number not less than 2, and each of the feature extraction sub-operators is a natural number not less than 3. The sub-operator is used to sequentially use the standardization module and the operator composition module to perform descriptor mining, and use the short-circuited path between the input and output ends of the standardization module to process the descriptor mining results of the feature extraction sub-operator; the image descriptor is transmitted to the v information restoration sub-operators, and each of the v information restoration sub-operators is used to sequentially perform descriptor mining to obtain the defect annotation descriptors output by the v information restoration sub-operators; wherein each of the information restoration sub-operators is used to sequentially use the standardization module and the operator composition module to perform descriptor mining, and use the short-circuited path to process the descriptor mining results of the information restoration sub-operator; the defect annotation results of the electronic component packaging image are estimated based on the defect annotation descriptors; On the other hand, the present application provides a detection system, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the steps in the above method when executing the program.

[0005] Beneficial effects of the present application: The present application adopts feature extraction sub-operators and information restoration sub-operators to form a defect detection algorithm, because the standardization module in each sub-operator is in front of the operator composition module, and the input and output ends of the standardization module in the sub-operator are short-circuited, forming a short-circuited path of adding post-standardization to the pre-standardization structure. This short-circuited path can establish a stable gradient transfer path. When the number of operator layers of the defect detection algorithm increases, the defect detection algorithm converges accordingly, and the algorithm training is more robust. Furthermore, the post-standardization structure in the defect detection algorithm can improve the defect detection quality of the defect detection algorithm and enhance the universality of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0007] Figure 1 A schematic diagram of an implementation flow of an electronic component packaging defect detection method based on image acquisition provided in an embodiment of the present application; Figure 2 A hardware entity schematic diagram of a detection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0008] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0009] The embodiment of the present application provides an electronic component packaging defect detection method based on image acquisition, which can be executed by a processor of a detection system. The detection system can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, a desktop computer, a mobile device, etc. The detection system is provided with a defect detection algorithm, which includes u feature extraction sub-operators arranged in sequence and v information restoration sub-operators arranged in sequence, each feature extraction sub-operator and each information restoration sub-operator including a standardized module and an operator composition module arranged in sequence, u is a natural number not less than 2, and v is a natural number not less than 3.

[0010] Figure 1 A schematic diagram of the implementation process of an electronic component packaging defect detection method based on image acquisition provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: Step S100: Acquire an electronic component packaging image.

[0011] Electronic components are the basic units that make up electronic circuits, such as resistors, capacitors, chips, etc. In the actual production process, these electronic components need to be packaged, which not only protects the internal components but also provides an interface for the connection between the components and the external circuit. The packaging process can produce some defects, such as damaged appearance of the package, bent pins, poor sealing, etc. In order to detect these defects, the inspection system must first obtain the electronic component packaging image.

[0012] The inspection system can obtain the package image of electronic components through image acquisition equipment. There are many types of image acquisition equipment, such as industrial cameras. Industrial cameras have the characteristics of high resolution and high frame rate, which can meet the high-precision requirements for the acquisition of electronic component packaging images. Its working principle is based on the principle of optical imaging, and the package area of ​​the electronic components to be inspected is imaged onto the camera's image sensor through the lens. The image sensor can be a CCD (charge coupled device) or a CMOS (complementary metal-oxide-semiconductor) sensor.

[0013] Step S200: The electronic component packaging image is transmitted to u feature extraction sub-operators, and each of the u feature extraction sub-operators is used to perform descriptor mining in sequence to obtain the image descriptors output by the u feature extraction sub-operators; each feature extraction sub-operator is used to perform descriptor mining in sequence using a standardized module and an operator composition module, and to process the descriptor mining results of the feature extraction sub-operator using a short-circuited path between the input and output ends of the standardized module.

[0014] In step S200, the electronic component package image is obtained by the detection system after step S100, and this image contains information about the appearance, structure, etc. of the electronic component package. In the electronic component package, such as chip package, it may include the chip body, pins, and package shell, etc. These parts are expressed in different forms in the image, and their shape, color, texture and other characteristics are the key basis for subsequent detection.

[0015] The u feature extraction sub-operators are important components of the defect detection algorithm. Feature extraction is a process of converting raw data into more representative and discriminative features. Each feature extraction sub-operator is designed to mine a specific type of feature descriptor from an image. The descriptor here is a quantitative representation of the local or global features of an image. For example, in some descriptors based on texture features, the texture feature descriptor can be obtained by calculating the co-occurrence matrix of the pixel grayscale values ​​in a certain area of ​​the image. Let the image I(x,y), where x and y are pixel coordinates, and the grayscale co-occurrence matrix P(d,θ) represents the frequency of occurrence of pixel grayscale value pairs at a distance of d and a direction of θ. By performing some statistical calculations on P(d,θ), such as the calculation of parameters such as contrast, correlation, energy, and entropy, a descriptor reflecting the texture characteristics of the image can be obtained.

[0016] When the inspection system transmits the electronic component package image to u feature extraction sub-operators, it processes them sequentially. This sequential processing helps to gradually extract features of different levels and types. Different feature extraction sub-operator combinations and sequences may be required for different electronic component package types. For example, for surface mount electronic component packages, it may be necessary to first extract the sub-operator of its surface texture features, and then perform edge feature extraction sub-operator processing.

[0017] In each feature extraction sub-operator, the normalization module and the operator composition module are used in sequence to perform descriptor mining. The function of the normalization module is to normalize the input data to improve the stability and convergence speed of the algorithm. For example, a feasible batch normalization method, for a batch of input data x={x1, x2,…,xn}, its normalization formula is: (calculate the mean), (compute the variance), and then normalize each data point xi to ,in is a very small number used to prevent the denominator from being 0. Through this standardization operation, the distribution of data can be made more stable, the problem of gradient disappearance or explosion can be reduced, and the effect of feature extraction can be improved.

[0018] The operator component module is built based on a specific algorithm or model structure, and is used to perform further feature mining operations on the standardized data. For example, it can be based on the structure of a convolutional neural network (CNN). In CNN, the convolution layer slides the convolution kernel on the image to perform convolution operations to extract local features in the image. Let the convolution kernel be K and the image be I. The convolution operation can be expressed as , where m and n are the size ranges of the convolution kernel, and O(x,y) is the value of the convolution result at the coordinate (x,y). This convolution operation can automatically learn the characteristic patterns in the image, such as edges, textures, etc., and is a very effective feature extraction method.

[0019] In addition, each feature extraction sub-operator also uses a short-circuit path (such as residual connection) between the input and output of the standardization module to process the descriptor mining results of the feature extraction sub-operator. Residual connection is an effective structure that allows the network to directly pass input information to the subsequent layers. For example, in a simple two-layer neural network, let the output of the first layer be F(x) and the input be x. The output of the second layer is F(x) in the traditional connection method, while when the residual connection is used, the output of the second layer is F(x)+x. This short-circuit path helps to effectively transfer the gradient, especially when the number of network layers is large, it can alleviate the problem of gradient vanishing or gradient explosion, making the network easier to train. Using this structure in the feature extraction sub-operator can further optimize the effect of descriptor mining, making the mined image descriptors more accurate and representative.

[0020] Through this series of operations, u feature extraction sub-operators process the electronic component packaging images respectively and finally output image descriptors. These image descriptors contain feature information extracted and optimized by each sub-operator, which will provide a basis for further processing by the subsequent information restoration sub-operator, thereby gradually realizing the detection of electronic component packaging defects. The descriptors mined by different feature extraction sub-operators may complement each other and jointly construct a comprehensive and effective feature representation of the electronic component packaging image, so as to accurately determine whether there are defects and the type and location of defects in the subsequent steps.

[0021] Step S300: The image descriptor is transmitted to v information restoration sub-operators, and each of the v information restoration sub-operators is used to perform descriptor mining in sequence to obtain defect annotation descriptors output by the v information restoration sub-operators; wherein each information restoration sub-operator is used to perform descriptor mining in sequence using a standardized module and an operator composition module, and to process the descriptor mining results of the information restoration sub-operator using a short-circuited path.

[0022] Step S300 is a step in which the detection system transmits the image descriptor obtained in step S200 to v information restoration sub-operators, and uses these sub-operators to mine descriptors in sequence, thereby obtaining the defect annotation descriptors output by the v information restoration sub-operators. This step is a key intermediate link in the electronic component packaging defect detection method based on image acquisition. It further mines information related to defect annotation on the basis of feature extraction to prepare for the final defect annotation result estimation. The image descriptor is the result obtained after the electronic component packaging image is processed by u feature extraction sub-operators in step S200. These image descriptors contain various feature information extracted from the original image. For example, for an integrated circuit package with a complex structure, the image descriptor may cover the texture features of the package shell, the geometric shape features of the pins, and the spatial relationship features of the connection part between the chip and the package body. These features exist in the image descriptor in a specific quantized form, providing basic data for subsequent information restoration operations. The v information restoration sub-operators are also an important part of the defect detection algorithm. Information restoration here can be understood as restoring more direct information related to the electronic component packaging defects based on the extracted feature information, thereby obtaining the defect annotation descriptor. Similar to the feature extraction sub-operator, each information restoration sub-operator has its specific function and structure.

[0023] The detection system transmits the image descriptor to v information restoration sub-operators in sequence. This sequential processing is to gradually build a complete defect annotation descriptor. For example, when processing electronic component packaging with multiple possible defect types, the feature information related to the surface integrity of the package may be processed by an information restoration sub-operator first, and then the information related to the reliability of the pin connection may be processed by the next sub-operator. Within each information restoration sub-operator, the descriptor mining is carried out in sequence using the standardization module and the operator composition module. The principle of the standardization module is similar to that in step S200. Its purpose is to normalize the input data to improve the stability and performance of the algorithm. Taking Z-Score standardization as an example, for a set of data x={x1, x2,…, xn}, its standardization formula is: ,in is the mean of the data, is the standard deviation of the data. Through such a standardization operation, data of different scales can be converted into a relatively uniform scale range, so that the operator component module can process data more effectively.

[0024] The operator composition module operates on the standardized data according to a specific algorithm structure to mine descriptors. For example, in some information reduction operators based on deep learning, the structure of recurrent neural network (RNN) may be used. For a sequence data (the image descriptor here can be regarded as a sequence data, although it is different from the traditional time series, but there is a similarity in the processing order), let the input sequence be , the hidden state update formula in RNN is: ,in is the hidden state at the current moment, is the hidden state of the previous moment, U and W are weight matrices, and f is the activation function (such as tanh function or ReLU function, etc.). Through such calculations, RNN can process long-term dependencies in the data and thus mine deep information related to defect annotation.

[0025] At the same time, each information restoration sub-operator uses a short-circuited path between the input and output ends of the standardization module to process the descriptor mining results. This short-circuited path is similar to that in step S200, for example, a similar residual connection method is used. Assuming that the input of a certain layer of the information restoration sub-operator is x, and the output after processing by the operator composition module is y, when a residual connection is used, the final output is y+x. This short-circuited path helps to solve the problem of gradient disappearance or gradient explosion that is prone to occur in deep network structures, and ensures that the information restoration sub-operator can stably mine effective descriptor information.

[0026] Through this series of operations, v information restoration operators process the image descriptors respectively, and finally obtain defect annotation descriptors. These defect annotation descriptors contain information that is more directly related to the defects of electronic component packaging, such as whether there are scratches on the packaging surface, whether the pins are broken or short-circuited, and other related information such as the possible locations of these defects. These defect annotation descriptors will serve as an important basis for estimating the defect annotation results of electronic component packaging images in subsequent steps.

[0027] In the actual application of electronic component packaging detection, different types of electronic component packaging may require different combinations of information restoration sub-operators and parameter settings. For example, for power-type electronic component packaging, due to its high requirements for heat dissipation, it may be necessary to pay more attention to the relevant information of the connection between the packaging shell and the heat sink. This feature needs to be optimized in the design of the information restoration sub-operator. For high-frequency electronic component packaging, it may be more necessary to pay attention to information related to factors such as capacitive coupling between pins. This requires that the algorithm structure and parameters that can effectively extract such features be considered in the design of the operator composition module.

[0028] Step S400: estimating defect annotation results of the electronic component packaging image according to the defect annotation descriptor.

[0029] The defect annotation descriptor is the result obtained after processing by v information restoration operators in step S300. These descriptors contain information closely related to the packaging defects of electronic components. For example, for the packaging of an electronic chip, the defect annotation descriptor may contain information representation about whether the chip pins are bent, whether the packaging shell has cracks, or whether there is poor contact between the package and the chip. This information is a quantitative or characterization expression obtained after the previous complex operations such as feature extraction and information restoration.

[0030] When the detection system makes an estimate based on the defect annotation descriptor, it needs to use a specific decision algorithm or classification algorithm. A feasible technical approach is to use a classifier, such as a support vector machine (SVM). The basic principle of SVM is to find an optimal hyperplane in the feature space to separate data of different categories. Suppose we have a defect annotation descriptor vector ,in Represents different eigenvalues. For a binary classification problem (e.g., classifying electronic component packages into defective and non-defective categories), the decision function of SVM can be expressed as ,in is the weight vector, and b is the bias term. Train the SVM using the training data to determine and the value of b, when > 0, it is judged as a class (such as a class without defects). < 0, it is judged as another category (such as defective category).

[0031] For multi-classification problems (for example, distinguishing different types of defects, such as pin defects, package shell defects, etc.), an extended SVM method can be used, such as One-vs-All or One-vs-One strategy. In the one-vs-all strategy, assuming there are k categories, k binary classifiers need to be built, each of which distinguishes one category from all other categories; in the one-vs-one strategy, k(k-1) / 2 binary classifiers need to be built, each of which distinguishes a pair of categories. In addition to SVM, neural networks can also be used to make estimates based on defect annotation descriptors. For example, a multilayer perceptron (MLP), which consists of an input layer, a hidden layer, and an output layer. Assume that the number of nodes in the input layer is the number of features n of the input defect annotation descriptor, the hidden layer has m nodes, and the number of nodes in the output layer is the number of categories to be predicted p. For the input defect annotation descriptor vector , the calculation in the hidden layer is ,in is the weight matrix from the input layer to the hidden layer, is the bias vector, f1 is the activation function (such as ReLU function: f(x)=max(0,x)); the calculation at the output layer is ,in is the weight matrix from the hidden layer to the output layer, is the bias vector, is an activation function (such as the softmax function: , which is used to convert the output into the probability distribution of each category). According to the results of the output layer, the defect annotation results of the electronic component packaging image can be estimated. For example, the category with the largest output probability is the estimated defect type.

[0032] In actual electronic component packaging inspection, different electronic component types and packaging forms may require different adjustments to the algorithm. For example, for small surface mount components, their defects may be more concentrated on the flatness of the pins and the welding conditions with the circuit board. This requires that when constructing defect annotation descriptors, more attention should be paid to features related to the pins, and in the algorithm for estimating defect annotation results, these features should be given appropriate weights or the parameters of the classification algorithm should be adjusted. For large power electronic component packaging, more attention may be paid to defects related to the heat dissipation inside the package. When processing defect annotation descriptors, feature information related to heat dissipation must be taken into account, and this information should be accurately used in the estimation algorithm to determine whether there are heat dissipation-related defects and the degree of defects.

[0033] In one embodiment, the training process of the defect detection algorithm includes the following steps: Step S10: Acquire electronic component packaging training images; Step S20: transmitting the electronic component packaging training image to u feature extraction sub-operators, using each of the u feature extraction sub-operators to perform descriptor mining in sequence, and obtaining the image descriptors output by the u feature extraction sub-operators; each feature extraction sub-operator is used to perform descriptor mining using a standardized module and an operator composition module in sequence, and to process the descriptor mining results of the feature extraction sub-operator using a short-circuited path between the input and output ends of the standardized module; Step S30: the image descriptor is transmitted to v information restoration sub-operators, each of the v information restoration sub-operators is used to perform descriptor mining in sequence, and the defect labeling descriptors output by the v information restoration sub-operators are obtained, each information restoration sub-operator is used to perform descriptor mining in sequence using a standardized module and an operator composition module, and a short-circuited path is used to process the descriptor mining results of the information restoration sub-operator; Step S40: estimating training defect annotation results of the electronic component packaging training images according to the defect annotation descriptor; Step S50: iterating algorithm parameters of the defect detection algorithm according to the difference between the actual defect annotation result of the electronic component packaging training image and the training defect annotation result.

[0034] When the inspection system executes step S10, its purpose is to collect data for the training of the defect detection algorithm. The electronic component package training images are a set of images containing the appearance information of the electronic component package in different states (defective and non-defective). These images are the basis for the algorithm to learn to identify defect patterns.

[0035] For example, when producing integrated circuit (IC) packages, there are various package types, such as dual in-line package (DIP), ball grid array package (BGA), etc. In order to train the defect detection algorithm for these package types, a large number of normal and defective DIP and BGA package images under different angles and different lighting conditions need to be collected. For DIP packages, possible defects include bent pins, cracks in the package body, etc.; for BGA packages, solder joint defects, abnormal ball spacing, etc. may occur. The inspection system can obtain these training images from the image acquisition device on the production line, or use an image database specially created for training.

[0036] In step S20, the detection system inputs the acquired electronic component packaging training image into a module composed of u feature extraction sub-operators. Each feature extraction sub-operator processes the image in sequence and mines out descriptors that can represent the image features. These descriptors are a quantitative representation of the original image features, which is helpful for subsequent analysis and defect judgment.

[0037] The normalization module first processes the input data in each feature extraction sub-operator. Taking batch normalization as an example, suppose the input batch data of a feature extraction sub-operator is , its mean ,variance The normalized output (in is a small positive number to prevent the denominator from being 0). This standardization operation makes the distribution of input data more stable, which helps to improve the convergence speed and stability of the algorithm.

[0038] The operator composition module performs feature mining on the standardized data based on different algorithm structures. For example, the convolution layer in the convolutional neural network (CNN) structure is used. Through the convolution operation, local features in the image, such as edges and textures, can be automatically extracted. These local features are critical for identifying defects on electronic component packaging. For example, for detecting scratches on the package surface, the convolution layer can extract the difference in texture and edges between the scratch and the surrounding normal surface.

[0039] The short-circuit path (such as residual connection) between the input and output of the standardization module in the feature extraction sub-operator further optimizes the results of descriptor mining. Assume that the input of a certain layer of a feature extraction sub-operator is x, and the output after processing by the operator component module is y. When there is a residual connection, the final output is y+x. This short-circuit path helps to effectively transfer the gradient, especially when the number of network layers is large, which can avoid the problem of gradient disappearance or gradient explosion, so that the network can better mine effective image descriptors.

[0040] Through this series of operations, the u feature extraction sub-operators process the input electronic component packaging training images respectively, and finally obtain their own output image descriptors. These image descriptors contain various feature information extracted from the original training images, laying the foundation for subsequent information restoration operations.

[0041] In step S30, the detection system transmits the image descriptor obtained in step S20 to v information restoration sub-operators. These information restoration sub-operators further process the image descriptors in sequence to mine out descriptors that are more directly related to defect annotation, namely, defect annotation descriptors.

[0042] Similar to the normalization module in the feature extraction sub-operator, the normalization module in the information restoration sub-operator also normalizes the input data. For example, Z-Score normalization is used to calculate the mean and standard deviation of a set of input image descriptor data, so that data of different scales are more comparable in subsequent processing, thereby improving the accuracy of the algorithm.

[0043] The operator component module operates on the standardized data according to a specific algorithm structure. Taking the recurrent neural network (RNN) as an example, for the input image description subsequence , the hidden state update formula in RNN is (where h t is the hidden state at the current moment, is the hidden state of the previous moment, U and W are weight matrices, and f is an activation function, such as tanh function or ReLU function. RNN can handle long-term dependencies in data, which is very useful for mining deep information related to defect annotation from image descriptors. For example, when judging defects related to the long-term stability of electronic component packaging, RNN can analyze based on previously extracted feature information (such as the trend of packaging materials over time).

[0044] The short-circuit paths (such as residual connections) in the information restoration operator also play a role in optimizing descriptor mining. Assuming that the input of a certain layer of the information restoration operator is x, the output after processing by the operator component module is y, and the final output is y+x when the residual connection is used. This helps to solve the gradient vanishing or gradient exploding problems that may occur in the deep network structure, and ensures that the information restoration operator can stably mine effective defect labeling descriptors.

[0045] Through the processing of v information restoration operators, the obtained defect annotation descriptor contains information directly related to the packaging defects of electronic components, such as the type and location of the defect and other preliminary judgment information.

[0046] Step S40 uses a specific algorithm to estimate the training defect annotation result of the electronic component packaging training image using the defect annotation descriptor obtained in step S30. The estimated result is the algorithm's judgment on whether there are defects and the defect type in the training image based on the current parameters and model structure.

[0047] Decision tree is a feasible classification algorithm. For example, for the classification of defects in electronic component packaging, each internal node of the decision tree is an attribute test (such as the color of the package, the number of pins, and other features), the branch is the test output, and the leaf node is the category (such as no defect, pin defect, package defect, etc.). Assuming that the defect annotation descriptor contains the color feature C of the package, the number feature P of the pins, etc., the decision tree is classified according to the different values ​​of these features. If C = "black" and P>10, it may be judged as no defect; if C = "white" and P<5, it may be judged as a pin defect, etc. Alternatively, the naive Bayes algorithm is used, based on the Bayesian theorem, assuming that each feature is independent of each other. For the detection of defects in electronic component packaging, suppose the features in the defect annotation descriptor are , the category is y (such as non-defective or defective category). According to Bayes' theorem In practical applications, since the features are assumed to be independent, By calculating the probability values ​​from the training data, the training defect annotation results can be estimated based on the new defect annotation descriptor. For example, if in the training data, most white packages with small pin spacing are defective, when the new defect annotation descriptor meets this feature, it may be judged as defective.

[0048] In step S50, the detection system compares the difference between the actual defect annotation results (known real situation) of the electronic component packaging training image and the estimated training defect annotation results in step S40, and then adjusts the parameters in the defect detection algorithm according to this difference to improve the accuracy of the algorithm. Taking the weight parameter update in the neural network as an example, if the mean square error (MSE) is used as the loss function, let y be the actual defect annotation result, is the training defect labeling result, n is the number of samples, and the loss function For the weight parameter w in the neural network, according to the gradient descent algorithm, the weight update formula is (in is the learning rate). For example, if in one iteration, it is found that the estimated result for a certain type of electronic component packaging defect (such as package crack) is significantly different from the actual result, by calculating the gradient of the loss function with respect to the weight, and then updating the weight parameter according to the above formula, the algorithm can more accurately identify such defects in the next iteration. Alternatively, an adaptive learning rate algorithm, such as the Adagrad algorithm, can be used, which adjusts the learning rate according to the historical gradient information of each parameter. Assume is the gradient of the i-th parameter in the t-th iteration, the cumulative sum of squared gradients , then the update formula of the i-th parameter in the t-th iteration is (in is a very small positive number to prevent the denominator from being 0). This algorithm can automatically adjust the learning rate according to the learning conditions of different parameters, so that the algorithm can converge faster during training.

[0049] By continuously iterating the parameters of the defect detection algorithm, the algorithm can gradually reduce the difference between the actual defect annotation results and the training defect annotation results, thereby improving the accuracy and generalization ability of the defect detection algorithm for electronic component packaging defect detection.

[0050] In one embodiment, the defect detection algorithm includes p feature extraction operators arranged in sequence, each feature extraction operator includes a number of feature extraction sub-operators arranged in sequence in u feature extraction sub-operators, and p is a natural number not less than 2. Then, step S20, transmitting the electronic component packaging training image to the u feature extraction sub-operators, using each feature extraction sub-operator in the u feature extraction sub-operators to perform descriptor mining in sequence, and obtaining the image descriptors output by the u feature extraction sub-operators, including: Step S21: transmitting the electronic component packaging training image to the first feature extraction operator, using each feature extraction sub-operator in the first feature extraction operator to perform descriptor mining in sequence, and obtaining the feature extraction result of the first feature extraction operator; Step S22: The feature extraction result of the rth feature extraction operator is transmitted to the r+1th feature extraction operator, and each feature extraction sub-operator in the r+1th feature extraction operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the r+1th feature extraction operator, where r is a natural number greater than 0, and r+1<p.

[0051] In step S21, the detection system transmits the electronic component package training image as input to the first feature extraction operator. This feature extraction operator is composed of multiple feature extraction sub-operators arranged in order. Each feature extraction sub-operator has its specific function and works together to perform preliminary feature extraction operations on the input image. For example, in the electronic component package detection, for a training image of a surface mounted chip package, the first feature extraction operator may be responsible for extracting some basic features of the package surface, such as texture, color distribution, etc.

[0052] Each feature extraction sub-operator operates in sequence. Assume that there are three feature extraction sub-operators in the first feature extraction operator, namely sub-operator A, sub-operator B and sub-operator C. Sub-operator A may first perform some standardization operation on the input image, and then perform preliminary feature mining through its operator component module. For example, if the operator component module of sub-operator A is based on the convolutional neural network (CNN) structure, for a local area of ​​the input image, the convolution operation between the convolution kernel and the image pixels is used to extract features. The result output by sub-operator A is then used as the input of sub-operator B. Sub-operator B may further refine or combine these features and then pass the result to sub-operator C. After similar operations, the output result of sub-operator C is the feature extraction result of the first feature extraction operator. This result is a quantitative representation of the preliminary feature extraction of the training image of the electronic component package, which contains feature information such as the texture and color of the package surface.

[0053] In step S22, the feature extraction result obtained by the rth feature extraction operator is passed to the r+1th feature extraction operator. Here, r represents the sequential number of the feature extraction operator, which gradually increases from 1. For example, when r=1, the result of the first feature extraction operator is passed to the second feature extraction operator. Each feature extraction operator is composed of multiple feature extraction sub-operators, and these sub-operators in the r+1th feature extraction operator also process the input feature extraction results in sequence. This sequential processing is a process of gradually and deeply mining the features of the electronic component packaging training images.

[0054] Assume that when detecting defects in multilayer ceramic capacitor (MLCC) packages, the first feature extraction operator may mainly extract the basic texture and color features of the MLCC package surface. When this result is passed to the second feature extraction operator, the feature extraction sub-operator in the second feature extraction operator can further mine features from different angles. For example, a sub-operator in the second feature extraction operator can analyze the periodicity and directionality of the texture based on the texture features provided by the first feature extraction operator through some complex algorithms (such as texture analysis algorithms based on wavelet transform) to detect whether there are texture anomalies caused by manufacturing process problems. Let the wavelet function be , perform wavelet transform on the input texture feature function f(x): (where a is the scale parameter and b is the translation parameter) and deeper texture features can be mined by analyzing the characteristics of W(a,b).

[0055] As the order of feature extraction operators increases, the level of mined features continues to deepen. For example, for the third feature extraction operator, the results of the first two feature extraction operators can be combined to analyze the characteristics of the electronic component package from the perspective of spatial layout. If it is a chip package inspection, it can be analyzed whether the position of the chip in the package is accurate, whether the spatial layout of the connection part between the pin and the chip meets the standard, etc. This requires the integration of the output results of multiple feature extraction sub-operators and is achieved through some spatial analysis algorithms (such as detecting the positional relationship of geometric shapes such as straight lines or circles in the image based on Hough transform). Suppose there is a point (x, y) in the image. In the Hough transform, for detecting a straight line, the point (x, y) is mapped to the parameter space ,satisfy , by counting the peak values ​​in the parameter space, the straight line features in the image are determined, so as to analyze the layout features of straight-line components such as chip pins in the package image.

[0056] In addition, the defect detection algorithm includes p information restoration operators arranged in sequence, and each information restoration operator includes at least two information restoration sub-operators arranged in sequence in v information restoration sub-operators. Then, step S30 transmits the image descriptor to the v information restoration sub-operators, and uses each of the v information restoration sub-operators to perform descriptor mining in sequence to obtain the defect annotation descriptors output by the v information restoration sub-operators, including: Step S31: transmitting the feature extraction result of the pth feature extraction operator to the first information restoration operator, using each information restoration sub-operator in the first information restoration operator to perform descriptor mining in sequence, and obtaining the feature extraction result of the first information restoration operator; Step S32: The feature extraction result of the pth feature extraction operator and the feature extraction result of the sth information restoration operator are transmitted to the s+1th information restoration operator, and each information restoration sub-operator in the s+1th information restoration operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the s+1th information restoration operator, where s is a natural number greater than 0, and s+1<p.

[0057] In step S31, after completing all feature extraction operator operations, the detection system transmits the feature extraction result of the pth feature extraction operator to the first information restoration operator. At this time, the feature extraction result has been mined by multiple feature extraction operators and contains rich feature information about the electronic component packaging. The first information restoration operator begins to process this feature information and converts it into information that is closer to defect annotation. For example, when detecting the electronic component packaging on a printed circuit board (PCB), the result after processing by the feature extraction operator may contain various feature information such as the shape, position, color and texture of each component package. The first information restoration operator will start to analyze the factors related to the packaging defects based on this information, such as whether there is information related to the defect such as package color change and shape distortion caused by welding heat.

[0058] Each information restoration sub-operator in the first information restoration operator performs descriptor mining in sequence. Assume that there are three information restoration sub-operators in the first information restoration operator, namely, sub-operator D, sub-operator E, and sub-operator F. Sub-operator D first processes the input feature extraction results. It can perform standardization operations on the input data (such as Z-Score standardization: for the input data, calculate the mean and standard deviation, and then perform information mining through its operator composition module. If the operator composition module of sub-operator D adopts a recursive neural network (RNN) structure, for the input feature sequence (here the feature extraction results are regarded as sequence data) , the hidden state update formula in RNN is (where h t is the hidden state at the current moment, is the hidden state of the previous moment, U and W are weight matrices, and f is an activation function, such as tanh function or ReLU function). The output of sub-operator D is then used as the input of sub-operator E, which is further processed by sub-operator E and passed to sub-operator F. The result of sub-operator F after processing is the feature extraction result of the first information restoration operator. Compared with the feature extraction result initially input to the first information restoration operator, this result is closer to the information that can be directly used to judge defects.

[0059] In step S32, the detection system transmits the feature extraction results of the pth feature extraction operator and the feature extraction results of the sth information restoration operator to the s+1th information restoration operator at the same time. This operation realizes the fusion of multi-source information, so that the information restoration process can integrate more information to mine descriptors related to defect annotation. For example, when detecting integrated circuit (IC) packaging defects, the feature extraction results of the pth feature extraction operator may contain macroscopic features of the IC package, such as the overall shape and pin layout, etc.; while the feature extraction results of the sth information restoration operator may already contain some preliminary judgment information related to microscopic defects, such as tiny scratches on the package surface or local deformation of the pins. Inputting these two results into the s+1th information restoration operator at the same time can analyze the defect situation more comprehensively.

[0060] Each information restoration sub-operator in the s+1th information restoration operator performs descriptor mining in sequence. Assume that there are three information restoration sub-operators in the s+1th information restoration operator, sub-operator G, sub-operator H, and sub-operator I. Sub-operator G first processes the input fusion information. It can use an algorithm similar to the attention mechanism to focus on the key parts of the input information. For example, when processing IC package images, if there is important information about the pin part in the input information (such as the degree of bending of the pin, the firmness of the connection with the package body, etc.), the attention mechanism can highlight this key information by calculating the weight of each feature. Let the input feature vector be , calculate the attention weight vector , then the output after attention weighting is The output of sub-operator G is then processed by sub-operator H and sub-operator I in turn. Sub-operator H and sub-operator I can use different algorithms to further mine information related to defect annotation. For example, sub-operator H may use a method based on principal component analysis (PCA) to reduce the dimension of features and extract the main components, and sub-operator I may use a classification algorithm based on support vector machine (SVM) to classify features, and finally obtain the feature extraction result of the s+1th information restoration operator. This result contains more comprehensive and accurate information related to the defects of electronic component packaging, which provides an important basis for the subsequent defect annotation result estimation.

[0061] In one embodiment, each feature extraction operator includes a first feature extraction sub-operator and a second feature extraction sub-operator arranged in sequence, the operator composition module of the first feature extraction sub-operator adopts a joint attention mining mechanism, and the operator composition module of the second feature extraction sub-operator is a convolutional neural network. Then, step S22, the feature extraction result of the rth feature extraction operator is transmitted to the r+1th feature extraction operator, and each feature extraction sub-operator in the r+1th feature extraction operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the r+1th feature extraction operator, including: Step S221: transmitting the feature extraction result of the rth feature extraction operator to the first feature extraction sub-operator of the r+1th feature extraction operator; Step S222: sequentially adopting the standardization module of the first feature extraction sub-operator and the joint attention mining mechanism to perform descriptor mining; Step S223: adding the feature extraction result of the rth feature extraction operator and the feature extraction result of the normalization module of the first feature extraction sub-operator using the short-circuited path and the feature extraction result of the joint attention mining mechanism of the first feature extraction sub-operator to obtain the feature extraction result of the first feature extraction sub-operator; Step S224: transmitting the feature extraction result of the first feature extraction sub-operator to the second feature extraction sub-operator of the r+1th feature extraction operator; Step S225: sequentially using the standardization module and the convolutional neural network of the second feature extraction sub-operator to perform descriptor mining; Step S226: The feature extraction result of the first feature extraction sub-operator and the feature extraction result of the standardization module of the second feature extraction sub-operator are added together with the feature extraction result of the convolutional neural network of the second feature extraction sub-operator using a short-circuited path to obtain the feature extraction result of the r+1th feature extraction operator.

[0062] Step S221 is a key link in the feature extraction process. The detection system passes the feature extraction result of the rth feature extraction operator to the first feature extraction sub-operator in the next feature extraction operator (r+1th). After a series of operations, the rth feature extraction operator has performed a certain level of feature mining on the electronic component package image. These mined features contain a variety of information about the electronic component package, such as the package's shape contour, texture features, color distribution, etc. The purpose of passing these results is to further mine deeper and more discriminative features in subsequent feature extraction sub-operators.

[0063] Taking the detection of a surface mount electronic component package as an example, it is assumed that the rth feature extraction operator has identified the approximate shape of the package (such as a rectangle) and the basic texture of the surface (smooth or with a certain graininess). When this feature extraction result is passed to the first feature extraction sub-operator of the r+1th feature extraction operator, it provides a basis for further analyzing the more detailed shape features of this rectangular package (such as the flatness of the edge, the symmetry of the four corners, etc.) and the microstructure of the texture (such as the periodicity of the texture, the consistency of the texture direction, etc.). This data transfer is similar to a step-by-step in-depth analysis process, gradually moving from a preliminary understanding of the whole to local and detailed feature mining. In step S222, the detection system first uses a standardization module in the first feature extraction sub-operator to process the input data (i.e., the feature extraction result passed by the rth feature extraction operator). The role of the standardization module is to make the distribution of the input data more regular, which helps to improve the stability and convergence speed of the algorithm. For example, the batch normalization technology is used. Through this standardization operation, the scale differences between different features are adjusted, so that the subsequent joint attention mining mechanism can process data more effectively.

[0064] The joint attention mining mechanism, also known as the multi-head attention mechanism, is a technology that can focus on key parts of the data and mine relevant feature descriptors. In the detection of electronic component packaging, it can simultaneously consider multiple factors to determine the areas and features that need to be focused on. For example, for a chip package with multiple pins, the joint attention mining mechanism can simultaneously focus on the location of the pins and specific areas on the surface of the package. Assume that in the image, there is a certain correlation between the pin area and certain defect-prone areas on the surface of the package (such as areas near the heat sink connection point) in the feature space. The joint attention mining mechanism focuses on these key areas for feature mining by calculating the relationship weights between these areas. It may be based on some algorithms to calculate the correlation between different areas, such as calculating the cosine similarity between feature vectors.

[0065] In step S223, the detection system uses a short-circuited path to add the feature extraction result of the r-th feature extraction operator and the feature extraction result of the normalization module of the first feature extraction sub-operator to the feature extraction result of the joint attention mining mechanism. The advantage of this short-circuited path is that it can effectively solve the problem of gradient disappearance or gradient explosion that is prone to occur in deep neural networks. For example, when the number of network layers is large, if there is no such short-circuited path, the gradient can become very small (gradient disappearance) or very large (gradient explosion) during the back propagation process, making the network difficult to train. The short-circuited path allows the original input information (the feature extraction result of the r-th feature extraction operator and the feature extraction result of the normalization module) to be directly added to the result after complex processing (joint attention mining mechanism), so that the information can be transmitted more directly in the network, ensuring the effective propagation of the gradient, thereby improving the training effect of the network.

[0066] Through this addition calculation, the result is a new feature representation that combines the original feature information and the mined feature information. For electronic component packaging detection, this fused feature representation is more comprehensive. For example, the feature extraction result of the rth feature extraction operator may contain the overall macroscopic features of the package, while the result after processing by the joint attention mining mechanism focuses on the microscopic features of certain key areas. After adding them together, the feature extraction result of the first feature extraction operator contains both the overall macroscopic information and the microscopic information of the key areas, which is very important for the subsequent accurate judgment of whether there are defects in the electronic component packaging and the type and location of the defects.

[0067] In step S224, the detection system passes the feature extraction result obtained by the first feature extraction sub-operator that integrates multiple information to the second feature extraction sub-operator in the r+1th feature extraction operator. This transfer is based on the logical order of the entire feature extraction process. The purpose is to further mine other types of features or further optimize and transform the mined features through the second feature extraction sub-operator on the basis of the first feature extraction sub-operator. For example, the first feature extraction sub-operator may have extracted features related to the shape and some key areas of the package, and the second feature extraction sub-operator can further process these features from different angles (such as from the perspective of optical properties or spatial layout).

[0068] Taking the inspection of an electronic component package with complex optical properties (such as a package with a special coating) as an example, the first feature extraction sub-operator may identify features such as the approximate shape and position of the coating coverage area. When this result is passed to the second feature extraction sub-operator, the second feature extraction sub-operator can analyze the relationship between the optical reflection characteristics of the coating and the internal structure of the package. It can process these features through some optical models or image processing algorithms. For example, it can use the light scattering model to analyze the effect of the microstructure of the coating surface on the scattering of light, thereby mining features related to the integrity of the internal structure of the package (because the optical properties of the coating can be affected by defects inside the package).

[0069] In step S225, similar to the first feature extraction sub-operator, the detection system first applies the standardization module in the second feature extraction sub-operator. This is because even after being processed by the first feature extraction sub-operator, the data input to the second feature extraction sub-operator may still have scale differences or uneven distribution problems. Performing the standardization operation again can ensure that the subsequent convolutional neural network (CNN) performs feature mining based on more stable input data. For example, if the value range of some features in the output data of the first feature extraction sub-operator is large, while others are small, the standardization operation can adjust these data to a suitable range, so that the convolutional neural network can better learn the feature patterns in the data.

[0070] Convolutional neural network (CNN) is a feature mining tool. In the second feature extraction sub-operator, CNN extracts features by sliding the convolution kernel in the convolution layer on the input data for convolution operation. In the electronic component packaging inspection, CNN can automatically learn various feature patterns in the package image. For example, for detecting tiny scratches on the package surface, the convolution kernel can capture the difference in pixel values ​​between the scratch area and the surrounding normal area through convolution operations. At the same time, the pooling layer of CNN can downsample the features, reducing the amount of data while retaining key features. For example, the maximum pooling layer selects the maximum value in each local area as the output, which helps to improve computational efficiency and enhance the robustness of features.

[0071] Step S226 is similar to the short-circuited path in step S223. In this step, the detection system uses the short-circuited path to add the feature extraction results of the first feature extraction sub-operator and the feature extraction results of the standardization module of the second feature extraction sub-operator to the feature extraction results of the convolutional neural network. This short-circuited path once again plays a role in stabilizing network training and fusing feature information at different levels. In the second feature extraction sub-operator, the features mined by the convolutional neural network may be more abstract and advanced, such as some complex feature patterns about the overall structure of the package. The feature extraction results of the first feature extraction sub-operator contain the feature information related to the key areas previously mined, and the feature extraction results of the standardization module are the information related to the convolutional neural network input after standardization. By adding them by the short-circuited path, feature information of different levels and types can be fused together to obtain a more comprehensive and representative feature extraction result of the r+1th feature extraction operator. For example, for an electronic component package with both tiny scratches on the surface and slight deformation of the internal structure, the first feature extraction sub-operator has mined the feature information related to the scratches, and the convolutional neural network has mined the abstract feature pattern related to the internal structure deformation in the second feature extraction sub-operator. The feature extraction result of the r+1th feature extraction operator obtained after fusing them can reflect these two different types of defect information at the same time, providing a richer feature basis for subsequent information restoration and defect labeling, which helps to improve the accuracy and robustness of the entire defect detection algorithm, enabling it to better cope with various types of electronic component packaging defects.

[0072] Through the detailed explanation of sub-steps S221-S226 of step S22, we can have a deeper understanding of the operating logic and technical means of the entire electronic component packaging defect detection algorithm in the feature extraction process. These steps cooperate with each other to gradually extract comprehensive and effective feature information from the electronic component packaging image, laying a solid foundation for subsequent defect detection.

[0073] In one implementation, step S223, the feature extraction result of the rth feature extraction operator and the feature extraction result of the normalization module of the first feature extraction sub-operator are added by using a short-circuited path and the feature extraction result of the joint attention mining mechanism of the first feature extraction sub-operator to obtain the feature extraction result of the first feature extraction sub-operator, including: Step S2231: determining a first product result of a feature extraction result of the rth feature extraction operator and a first influencing factor; and determining a second product result of a feature extraction result of a standardization module of the first feature extraction sub-operator and a second influencing factor; Step S2232: perform addition calculation on the first product result and the second product result using the feature extraction result of the joint attention mining mechanism of the short-circuited path and the first feature extraction sub-operator.

[0074] In step S2231, the impact factor is a parameter used to adjust the weight of feature information from different sources. The detection system determines two product results in this step, the purpose of which is to weight different feature extraction results according to the pre-set or learned impact factors. The first impact factor and the second impact factor play a role in balancing the importance of different features in the entire algorithm.

[0075] For example, when detecting chip packaging defects, the feature extraction results of the rth feature extraction operator may include the overall shape features of the chip, while the feature extraction results of the standardization module of the first feature extraction operator may focus on the chip surface texture features. If the first impact factor is large, it means that the overall shape features of the chip are relatively more important in the current detection task; if the second impact factor is large, it means that the chip surface texture features are more critical. This weighting method can be adjusted according to different electronic component packaging types and specific detection requirements.

[0076] Let the feature extraction result of the rth feature extraction operator be the vector The first impact factor is , then the first product result is the vector Similarly, let the feature extraction result of the standardization module of the first feature extraction sub-operator be the vector The second impact factor is , then the second product result is the vector The vector representation here is a general way to represent different feature information. In practical applications, these features can be pixel values ​​of an image, feature values ​​processed by a specific algorithm, etc.

[0077] In step S2232, the detection system performs this operation in order to fuse the feature information after different weightings with the feature information mined by the joint attention mining mechanism. Through addition calculation, the feature information obtained from different angles can be integrated together to form a more comprehensive and representative feature representation. For example, continuing to take chip packaging detection as an example, the first product result may contain information weighted according to the overall shape characteristics of the chip, and the second product result contains information weighted according to the surface texture characteristics of the chip, and the feature extraction result of the joint attention mining mechanism may be features mined for specific key areas of the chip (such as pin connection areas). Adding these three parts of information together can make the final feature contain both the comprehensive information of the overall shape and surface texture, and the feature information of the key areas, which is more conducive to the accurate detection of chip packaging defects.

[0078] The short-circuited path ensures that feature information from different levels and sources can be effectively fused, avoiding the loss or distortion of information during the transmission and fusion process. In the deep neural network structure, the short-circuited path can alleviate the problem of gradient vanishing or gradient exploding, allowing the network to be trained more stably. In this specific feature fusion scenario, it allows the weighted feature information to be directly added to the results of the joint attention mining mechanism, ensuring the integrity and effectiveness of the feature information. For example, if there is no short-circuited path, when the number of network layers is large or the feature information is complex, some feature information may be overly weakened or enhanced during the fusion process, thereby affecting the final defect detection effect.

[0079] Optionally, step S226, adding the feature extraction result of the first feature extraction sub-operator and the feature extraction result of the standardization module of the second feature extraction sub-operator to the feature extraction result of the convolutional neural network of the second feature extraction sub-operator using a short-circuited path to obtain the feature extraction result of the r+1th feature extraction operator, including: Step S2261: determining a third product result of a feature extraction result of the first feature extraction sub-operator and a third influencing factor; at the same time, determining a fourth product result of a feature extraction result of the standardization module of the second feature extraction sub-operator and a fourth influencing factor; Step S2262: The third product result and the fourth product result are added to the feature extraction result of the convolutional neural network of the second feature extraction sub-operator using a short-circuited path.

[0080] Step S2261 is similar to step S2231. In this step, the detection system uses the influence factors again to weight different feature extraction results. The third influence factor and the fourth influence factor are used to adjust the weights of the feature extraction results of the first feature extraction sub-operator and the feature extraction results of the standardization module of the second feature extraction sub-operator, respectively. This is because the feature information from these two sources has different importance under different electronic component packaging structures and defect types.

[0081] For example, when detecting an electronic component package with a complex internal structure (such as a multilayer ceramic capacitor package), the feature extraction result of the first feature extraction sub-operator may include the external geometry and preliminary texture features of the package, while the feature extraction result of the standardization module of the second feature extraction sub-operator may be the features related to the internal structure after standardization (mined by means such as convolutional neural networks). If the third impact factor is large, it means that the external geometry and preliminary texture features are relatively more important in the current detection; if the fourth impact factor is large, it means that the features related to the internal structure are more critical.

[0082] Let the feature extraction result of the first feature extraction operator be the vector The third impact factor is , then the third product result is the vector . Let the feature extraction result of the standardization module of the second feature extraction sub-operator be vector The fourth impact factor is , then the fourth product result is the vector Here, feature information is also represented in vector form to facilitate weighting operations and subsequent calculations.

[0083] In step S2262, the detection system performs this addition calculation in order to fuse the weighted different feature information with the feature information mined by the convolutional neural network. This fusion can comprehensively consider the features of multiple aspects to form a more complete feature representation, thereby improving the accuracy of the detection of defects in the packaging of electronic components. For example, in the detection of multilayer ceramic capacitor packaging, the third product result contains the information weighted according to the external geometric shape and preliminary texture features, the fourth product result contains the information weighted according to the internal structure related features, and the feature extraction result of the convolutional neural network of the second feature extraction sub-operator may be information about the relationship between the internal structure and the external features or more abstract overall structural features. Adding these three parts of information can make the final feature contain multi-faceted information from the outside to the inside and from the local to the whole, which is more conducive to accurately judging whether there are defects in the package, such as whether there is damage to the internal structure that causes deformation of the external shape or abnormal texture. The short-circuit path ensures the effectiveness and stability when fusing different feature information. In this complex feature fusion process, the short-circuit path allows the weighted features to be directly added to the results of the convolutional neural network, avoiding the problem of feature information loss or distortion that may occur in the deep network structure. For example, if the network structure is complex, in the absence of a short-circuited path, gradient propagation difficulties may occur, resulting in the inability to correctly transmit and integrate certain feature information during the fusion process. The short-circuited path can effectively solve this problem and ensure that the feature extraction results of the final r+1th feature extraction operator can accurately reflect the various feature information of the electronic component package, providing a reliable basis for subsequent operations such as information restoration and defect marking.

[0084] The detailed explanation of these sub-steps provides a deeper understanding of the fine-grained operations of the electronic component packaging defect detection algorithm in terms of feature fusion and adjustment of feature weights, which help improve the accuracy and robustness of the algorithm for different types of electronic component packaging defect detection.

[0085] In some embodiments, the method may further include: Step S227: iterating one or more of the first influencing factor, the second influencing factor, the third influencing factor and the fourth influencing factor according to the difference between the actual defect annotation result of the electronic component packaging training image and the training defect annotation result.

[0086] In step S227, the detection system iterates one or more of the first influencing factor, the second influencing factor, the third influencing factor and the fourth influencing factor according to the difference between the actual defect annotation result of the electronic component packaging training image and the training defect annotation result.

[0087] In the electronic component packaging defect detection algorithm, the influencing factor plays a key role in adjusting the weight of feature information from different sources. The first influencing factor, the second influencing factor, the third influencing factor and the fourth influencing factor play a role in different feature fusion steps to balance the contribution of different features to the final result. For example, when detecting a complex integrated circuit (IC) package, the first influencing factor may be related to the overall shape characteristics of the package, the second influencing factor may be related to the texture characteristics of the package surface, the third influencing factor may involve the characteristics of a specific area (such as near the pin), and the fourth influencing factor may be related to the weight of the internal structure-related features. The detection system iterates the influencing factors by comparing the difference between the actual defect annotation results and the training defect annotation results. The actual defect annotation results are the real defects of the electronic component packaging, while the training defect annotation results are the judgments made by the algorithm based on the current influencing factors and model structures. For example, assuming that when detecting the pin bending defect of the IC package, the actual situation is that the pin is obviously bent (the actual defect annotation result), but the training defect annotation results obtained by the algorithm based on the current influencing factors and models show that the pin is normal. This shows that the current influencing factor settings may be unreasonable, resulting in a deviation in the algorithm's emphasis on different features.

[0088] In order to correct this deviation, the detection system can use an optimization algorithm based on gradient descent to iterate the impact factor. Let the impact factor be (here It can represent the first, second, third or fourth influencing factor), and the loss function is L. The loss function represents the difference between the actual defect annotation result and the training defect annotation result. For example, the commonly used mean square error (MSE) is used as the loss function. For a set of training samples, let the actual result be The training result is , n is the number of samples, then .

[0089] According to the gradient descent algorithm, the update formula of the impact factor is ,in is the learning rate, which determines the step size of the update of the influence factor at each iteration. For example, if the learning rate is set to 0.01, when calculating the partial derivative of the loss function with respect to a certain influence factor (such as the first influence factor) After that, the value of the first influencing factor can be updated according to the formula. If the partial derivative is positive, it means that the current value of the first influencing factor increases the loss function, and the value of the influencing factor needs to be reduced; conversely, if the partial derivative is negative, the value of the influencing factor needs to be increased.

[0090] Through this iterative process, the detection system can continuously adjust the influencing factors according to the actual detection results, making the weight of feature information from different sources in the algorithm more reasonable. This helps to improve the accuracy of the algorithm in detecting defects in electronic component packaging.

[0091] In some embodiments, the method may further include: Step S60: standardize the defect annotation descriptors output by the v information restoration sub-operators to obtain standardized defect annotation descriptors.

[0092] In step S60, the information restoration operator, i.e., the decoder, outputs a defect annotation descriptor containing various information related to the electronic component packaging defects, but these information may differ in terms of numerical range, distribution, etc. The purpose of standardization is to convert these descriptors into a form with a unified standard for subsequent processing and analysis.

[0093] For example, when detecting defects in electronic chip packaging, the information reduction operator may output annotation descriptors for different types of defects such as chip pin defects, package cracks, and poor contact between the chip and the package. These descriptors may be expressed in different numerical forms, such as some based on probability values ​​(indicating the possibility of a certain defect), and some may be based on the quantified values ​​of certain features (such as the relative value of the width of the package crack). When the detection system is standardized, a variety of technical means can be used. One feasible method is Z-Score standardization.

[0094] Standardized defect annotation descriptors help improve the performance of subsequent algorithms. When using classification algorithms (such as support vector machines, neural networks, etc.) to perform final classification or evaluation of defects, standardized data can make the algorithm converge faster and improve the accuracy and stability of the algorithm. For example, in a neural network, if the input data is not standardized, it can lead to problems such as gradient vanishing or gradient explosion during the gradient descent process, affecting the training effect of the network. The standardized defect annotation descriptor can avoid these problems, allowing the neural network to more effectively learn the relationship between the defect annotation descriptor and the actual defect, thereby improving the accuracy of electronic component packaging defect detection.

[0095] In some embodiments, each information restoration operator includes a first information restoration sub-operator, a second information restoration sub-operator, and a third information restoration sub-operator arranged in sequence, the operator composition module of the first information restoration sub-operator is a first joint attention mining mechanism, the operator composition module of the second information restoration sub-operator is a second joint attention mining mechanism, and the operator composition module of the third information restoration sub-operator is a convolutional neural network. Then, step S32, the feature extraction result of the pth feature extraction operator and the feature extraction result of the sth information restoration operator are transmitted to the s+1th information restoration operator, and each information restoration sub-operator in the s+1th information restoration operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the s+1th information restoration operator, including: Step S321: transmitting the feature extraction result of the s-th information restoration operator to the first information restoration sub-operator of the s+1-th information restoration operator; Step S322: sequentially adopting the standardization module of the first information restoration sub-operator and the first joint attention mining mechanism to perform descriptor mining; Step S323: adding the feature extraction result of the sth information restoration operator and the feature extraction result of the standardization module of the first information restoration sub-operator to the feature extraction result of the first joint attention mining mechanism of the first information restoration sub-operator using a short-circuited path to obtain the feature extraction result of the first information restoration sub-operator; Step S324: transmitting the feature extraction result of the p-th feature extraction operator and the feature extraction result of the first information restoration sub-operator to the second information restoration sub-operator of the s+1-th information restoration operator; Step S325: using the standardization module of the second information restoration sub-operator and the second joint attention mining mechanism in sequence to perform descriptor mining; Step S326: adding the feature extraction result of the first information restoration sub-operator and the feature extraction result of the standardization module of the second information restoration sub-operator to the feature extraction result of the second joint attention mining mechanism of the second information restoration sub-operator using a short-circuited path to obtain the feature extraction result of the second information restoration sub-operator; Step S327: transmitting the feature extraction result of the second information restoration sub-operator to the third information restoration sub-operator of the s+1th information restoration operator; Step S328: sequentially using the standardization module of the third information restoration sub-operator and the convolutional neural network to perform descriptor mining; Step S329: The feature extraction result of the second information restoration sub-operator and the feature extraction result of the standardization module of the third information restoration sub-operator are added together with the feature extraction result of the convolutional neural network of the third information restoration sub-operator using a short-circuited path to obtain the feature extraction result of the s+1th information restoration operator.

[0096] In step S321, when the detection system executes this step, the feature extraction result of the sth information restoration operator is passed as input to the first information restoration sub-operator of the s+1th information restoration operator. In the entire electronic component packaging defect detection process, this data transfer is an important link in the gradual restoration and refinement of information. Each information restoration operator processes the previous feature information to get closer to the final defect labeling result. The feature extraction result of the sth information restoration operator already contains information related to the defect to a certain extent. Passing it to the first information restoration sub-operator in the next information restoration operator is to further mine and convert this information in this sub-operator, and move towards a more accurate defect labeling descriptor.

[0097] For example, when inspecting an electronic component package with a complex internal structure (such as a multilayer ceramic capacitor package), the sth information restoration operator may have identified some features of the internal structure of the package at a certain level, such as the relative position relationship between different layers, possible small deformation areas, etc. After these feature extraction results are passed to the first information restoration operator of the s+1th information restoration operator, they can provide a basis for further analysis of whether these deformation areas are related to specific types of defects (such as interlayer short circuits, dielectric damage, etc.). This is like passing existing clues to the next link in a step-by-step puzzle-solving process in order to explore the essence of the problem in more depth.

[0098] In step S322, the detection system processes the input data (i.e., the feature extraction result delivered by the sth information restoration operator) using the standardization module of the first information restoration operator. The standardization module is designed to make the distribution of the input data more consistent with the algorithm requirements and improve the stability and consistency of the data. Through the standardization operation, the scale of the data is unified, avoiding problems caused by large differences in data scale in subsequent operations, such as some features may be ignored or their effects may be over-amplified in subsequent calculations due to excessively large or small values.

[0099] The first joint attention mining mechanism is a technical means for focusing on key information and mining relevant feature descriptors. In the scenario of electronic component packaging inspection, it can determine which parts are more relevant to defect annotation based on the relationship between different parts of the input data. For example, for the multilayer ceramic capacitor package mentioned earlier, in the standardized data, the first joint attention mining mechanism can focus on those features related to areas where defects may exist (such as interlayer contact areas). It may determine the allocation of attention by calculating the correlation coefficient between different features. In this way, the first joint attention mining mechanism can focus more on defect-related features and mine more discriminative descriptors.

[0100] In step S323, the detection system uses a short-circuited path (such as a residual connection) to perform addition calculations. The short-circuited path plays an important role in this process. It can effectively solve the problem of gradient vanishing or gradient exploding that may occur in the deep network structure. In this specific operation, the feature extraction result of the sth information restoration operator and the feature extraction result of the standardization module of the first information restoration operator are added to the feature extraction result of the first joint attention mining mechanism. For example, when processing complex electronic component packaging images, as the network depth increases, if there is no short-circuited path, the gradient can become very small (gradient vanishes) or very large (gradient explodes) during the back propagation process, making the network difficult to train. The short-circuited path allows the original input information (the feature extraction result of the sth information restoration operator and the feature extraction result of the standardization module) to be directly added to the result after complex processing (the first joint attention mining mechanism), so that the information can be transmitted more directly in the network, ensuring the effective propagation of the gradient, thereby improving the training effect of the network. This addition calculation integrates feature information from different sources. The feature extraction result of the sth information restoration operator contains the overall information in the previous information restoration step, the feature extraction result of the standardization module of the first information restoration sub-operator is the information related to the current sub-operator after standardization, and the feature extraction result of the first joint attention mining mechanism is the feature information after the attention mechanism focuses and mines. Adding these three parts of information together can obtain a more comprehensive and integrated feature extraction result of the first information restoration sub-operator. For example, when detecting pin defects in electronic component packages, the feature extraction result of the sth information restoration operator may contain the overall layout information of the pin in the package, the feature extraction result of the standardization module may contain standardized information related to the appearance characteristics of the pin, and the feature extraction result of the first joint attention mining mechanism may be the features mined for the key parts of the pin (such as the bending point or near the solder joint). The result obtained by adding them can simultaneously contain the information of the overall layout, appearance characteristics and key parts, which is more conducive to accurately judging whether the pin has defects.

[0101] In step S324, the detection system transmits the feature extraction result of the pth feature extraction operator and the feature extraction result of the first information restoration sub-operator to the second information restoration sub-operator of the s+1th information restoration operator at the same time. This step realizes the fusion of multi-source information. The feature extraction result of the pth feature extraction operator contains rich information obtained from the original image through feature extraction. This information describes the characteristics of the electronic component packaging from different angles, such as shape, texture, color and other aspects. The feature extraction result of the first information restoration sub-operator is the defect-related information obtained after processing by the previous sub-operator. Passing these two results to the second information restoration sub-operator at the same time allows this sub-operator to integrate more information to further explore features related to defect annotation.

[0102] For example, when detecting chip package defects, the feature extraction results of the pth feature extraction operator may include information such as the shape features of the chip (such as square or round), the texture features of the chip surface (such as smooth or slightly granular), and the color contrast between the chip and the package. The feature extraction results of the first information restoration operator may have identified the tiny scratches that may exist on the chip surface or the local poor contact area between the chip and the package. After passing these two results to the second information restoration operator, the relationship between these features can be analyzed more comprehensively, for example, whether the shape of the chip has changed slightly due to internal defects, whether the texture changes on the chip surface are related to local poor contact, etc., so as to more accurately judge whether there are defects in the chip package and the type and severity of the defects.

[0103] In step S325, the detection system first uses the standardization module in the second information restoration sub-operator to process the input data (i.e., the fused feature results transmitted by the p-th feature extraction operator and the first information restoration sub-operator). Although the data may have been standardized before, it is necessary to standardize it again when new fused data is input. This is because data from different sources can produce new scale differences or uneven distributions after fusion. For example, the feature extraction results of the p-th feature extraction operator may differ from the feature extraction results of the first information restoration sub-operator in terms of numerical range and distribution. The standardization module can adjust these fused data to a state that is more suitable for subsequent operations, so that the second joint attention mining mechanism can process data more effectively.

[0104] The second joint attention mining mechanism is similar to the first joint attention mining mechanism, but it mines key information in the data from different angles based on the new standardized data. For example, when detecting defects in electronic component packaging, the second joint attention mining mechanism can pay more attention to those feature relationships that stand out after the fusion of multi-source information. If a certain correlation is found between the chip surface texture features and the contact between the chip and the package in the previous step, the second joint attention mining mechanism can further explore the relationship between this correlation and possible defects. It can use more complex algorithms to calculate the relationship weights between different features, such as a neural network-based attention mechanism, which determines the importance weights of different features in judging defects by learning feature patterns in the data, thereby mining more targeted descriptors.

[0105] Step S326 is similar to the short-circuited path in step S323. In this step, the detection system uses the short-circuited path to add the feature extraction results of the first information restoration sub-operator, the feature extraction results of the standardization module of the second information restoration sub-operator, and the feature extraction results of the second joint attention mining mechanism. The short-circuited path continues to play a role in stabilizing network training and fusing feature information at different levels. In this operation, the feature extraction results of the first information restoration sub-operator contain part of the defect-related information mined by the previous sub-operator, the feature extraction results of the standardization module of the second information restoration sub-operator are the information related to the current sub-operator after standardization, and the feature extraction results of the second joint attention mining mechanism are more targeted feature information mined by this mechanism. By adding them by the short-circuited path, feature information from different sources and different levels can be effectively integrated, avoiding the loss or distortion of information that may occur in the deep network structure. The result of this addition calculation is a feature representation that more comprehensively and accurately reflects the information related to the packaging defects of electronic components. For example, when detecting heat dissipation-related defects in electronic component packages, the feature extraction results of the first information restoration sub-operator may contain preliminary information on the temperature distribution on the surface of the package, and the feature extraction results of the standardization module of the second information restoration sub-operator may contain standardized information related to the heat dissipation structure (such as the shape and size of the heat sink after standardization). The feature extraction results of the second joint attention mining mechanism may be the features mined for the heat dissipation structure and the heat conduction path inside the package. The feature extraction results of the second information restoration sub-operator obtained by adding these three parts of information can comprehensively consider multiple aspects of heat dissipation, which is more conducive to accurately judging whether there are heat dissipation-related defects, such as heat dissipation channel blockage, uneven heat dissipation, etc.

[0106] In step S327, the detection system passes the feature extraction result of the second information restoration sub-operator to the third information restoration sub-operator of the s+1th information restoration operator. This transfer is based on the logical order within the entire information restoration operator. The purpose is to further process and transform the feature information through the third information restoration sub-operator on the basis of the second information restoration sub-operator to obtain a result that is closer to the final defect annotation descriptor. For example, the second information restoration sub-operator may have mined some intermediate-level features related to electronic component packaging defects. After passing these results to the third information restoration sub-operator, the deeper relationship between these features can be further analyzed, or these features can be combined with other information that has not yet been fully utilized to obtain more accurate defect-related information.

[0107] Taking the detection of an electronic component package with multiple pins as an example, the feature extraction result of the second information reduction operator may include information such as the relative position relationship between the pins, some features on the pin surface (such as whether there are traces of oxidation, etc.), and the connection between the pin and the package. After passing this result to the third information reduction operator, the third information reduction operator can further analyze the relationship between these pin features and the electrical performance of the entire electronic component. For example, through some electrical models or empirical rules, it is determined whether these pin features will affect the normal operation of the electronic component, whether they may cause defects such as short circuits, open circuits, or abnormal signal transmission, so as to further mine information that is more directly related to defect annotation.

[0108] In step S328, the detection system first uses the standardization module in the third information restoration sub-operator to process the input data (i.e., the feature extraction result transmitted by the second information restoration sub-operator). This is because even after the previous information processing steps, the data input to the third information restoration sub-operator may still have inconsistent numerical scales or uneven distribution. The standardization module can adjust the distribution of the data again to make it more suitable for the processing of the convolutional neural network (CNN). For example, in the previous information restoration sub-operator, due to the different sources and processing methods of different features, the value range of some features may be larger, while others are smaller. Through the standardization operation, the data can be converted to a more uniform numerical range, so that the convolutional neural network can learn the feature patterns in the data more effectively. The convolutional neural network (CNN) is used to mine descriptors in this sub-operator. In the detection of electronic component packaging, CNN can automatically learn various feature patterns in the package image. For example, for detecting tiny scratches on the surface of the package, the convolution kernel can capture the difference in pixel values ​​between the scratch area and the surrounding normal area through the convolution operation. At the same time, the pooling layer of CNN can downsample features, reducing the amount of data while retaining key features. For example, the maximum pooling layer selects the maximum value in each local area as the output, which helps improve computational efficiency and enhance the robustness of features. Through these operations of the convolutional neural network, the third information reduction operator can mine descriptors that are more directly related to the packaging defects of electronic components.

[0109] In step S329, the detection system uses the short-circuited path to add the feature extraction results of the second information restoration sub-operator and the feature extraction results of the standardization module of the third information restoration sub-operator to the feature extraction results of the convolutional neural network of the third information restoration sub-operator. The short-circuited path plays a key role again in this step. In a deep network structure, as information processing deepens, problems such as gradient disappearance or gradient explosion are prone to occur, and the short-circuited path can effectively alleviate these problems. It allows feature information from different sources (the feature extraction results of the second information restoration sub-operator and the feature extraction results of the standardization module) to be directly added to the feature results mined by the convolutional neural network, ensuring the integrity and effectiveness of the information. For example, if there is no short-circuited path, when processing complex electronic component packaging defect detection, some feature information may be overly weakened or enhanced during the fusion process, thereby affecting the final defect labeling accuracy. The feature extraction result of the s+1th information restoration operator obtained by this addition calculation is a feature representation that integrates multiple aspects of information. It combines the intermediate-level features related to defects mined by the second information reduction operator, the features related to the input of the convolutional neural network processed by the standardization module of the third information reduction operator, and the more abstract and directly defect-related features mined by the convolutional neural network. For example, when detecting the sealing defects of electronic component packaging, the feature extraction results of the second information reduction operator may contain some feature information of the sealing edge of the package, the feature extraction results of the standardization module of the third information reduction operator may contain standardized information related to the characteristics of the sealing material, and the feature extraction results of the convolutional neural network may be features mined for the overall structural integrity of the sealing area. The result obtained by adding these three parts of information can more comprehensively reflect the various aspects of the sealing defects, and provide a more reliable basis for the final accurate estimation of the defect annotation results of the electronic component packaging image. This comprehensive feature representation can capture various features related to the sealing defects in a more detailed manner, such as the tiny damage of the sealing edge, the unevenness of the sealing material, and the integrity of the overall sealing structure. Through such multi-step information fusion and processing, the entire defect detection algorithm can more accurately deal with electronic component packaging defects of different types and complexities, and improve the accuracy and reliability of detection.

[0110] In the actual production process of electronic component packaging, different types of components (such as resistors, capacitors, chips, etc.) have their own unique packaging structures and possible defect types. For example, for capacitor packaging, more attention may be paid to the sealing of its internal electrolyte and the connection between the electrode and the external pin; while for chip packaging, in addition to the pin connection, it is also necessary to focus on the fit between the chip and the package body, the integrity of the chip surface, etc. These sub-steps in step S32 can adapt to the detection needs of various electronic component packaging by continuously integrating and mining feature information at different levels and from different sources.

[0111] Taking the pin welding defect detection of chip package as an example, the feature extraction result of the sth information restoration operator may include the overall arrangement of the pins observed from a macroscopic perspective, and the first information restoration operator may focus on the preliminary defect characteristics on the pin surface (such as oxidation that may be implied by color changes). In the subsequent steps, after being fused with the feature extraction results of the pth feature extraction operator (which may include basic features such as the overall shape and size of the chip), the second information restoration operator can further analyze the relationship between poor pin welding and the overall structure of the chip, such as whether the chip is slightly deformed due to thermal stress during welding. The third information restoration operator can mine more microscopic features through convolutional neural networks, such as changes in the metal microstructure at the pin welding point. These sub-steps are progressive, and the feature extraction results of the s+1th information restoration operator can fully and accurately reflect the various possible situations of chip package pin welding defects, whether it is a macroscopic arrangement anomaly or a microscopic welding quality problem.

[0112] For example, when detecting whether there is a shell damage defect in the resistor package, the entire process also follows the logic of these sub-steps. Starting from the initial information restoration operator, different feature information is gradually integrated. After processing in each sub-step, tiny cracks or damage points on the resistor package shell can be accurately identified. Feature information from different sources, such as the color characteristics of the resistor (which may be related to the consistency of the shell material), the relative position relationship between the resistor body and the pin (which may change due to shell damage), etc., are effectively integrated into the final feature extraction results after a series of standardization, attention mining mechanism and convolutional neural network operations, thereby improving the accuracy of detecting damage defects in the resistor package shell.

[0113] In one embodiment, the method may further include: Step S70: Under the premise that the iteration number of the defect detection algorithm is not greater than a first preset iteration number, constraining the algorithm parameter amount of each iteration of the defect detection algorithm to be less than a set parameter amount; Step S80: On the premise that the iteration number of the defect detection algorithm is not less than the second preset iteration number, the parameter quantity is no longer constrained, and the second preset iteration number is greater than the first preset iteration number.

[0114] In step S70, the iteration number refers to the number of times the algorithm is updated when training the defect detection algorithm. In each iteration, the algorithm adjusts the algorithm parameters to improve the accuracy of the algorithm based on the input electronic component packaging training image and the difference between the corresponding actual defect annotation results and the training defect annotation results. The algorithm parameter quantity refers to the number of parameters that need to be optimized and adjusted in the algorithm, including but not limited to weights and biases in the neural network. For example, in a simple linear regression model middle, and These are the algorithm parameters, and in a complex deep neural network, there may be tens of thousands of these parameters.

[0115] It is of great significance that the detection system constrains the algorithm parameter amount of each iteration to be less than the set parameter amount when the iteration number of the defect detection algorithm is not greater than the first preset iteration number. In the early stages of algorithm training, limiting the update of the algorithm parameter amount helps to prevent the algorithm from fitting the noise in the training data too quickly, thereby improving the generalization ability of the algorithm. For example, when detecting tiny defects in electronic component packaging (such as fine scratches on the surface of chip packaging), if the algorithm parameters are updated too much in the early iterations, the algorithm may pay too much attention to some random noise or abnormal samples in the training data, resulting in the inability to accurately detect similar defects when faced with new, unseen electronic component packaging images.

[0116] One technical approach is to use a variant of the stochastic gradient descent (SGD) algorithm, such as Adagrad or Adadelta. Taking Adagrad as an example, when updating algorithm parameters, it adjusts the learning rate based on the historical gradient information of each parameter. In this way, the amplitude of parameter updates at each iteration can be controlled, thereby indirectly limiting the amount of algorithm parameters for each generation of iterations. In the early stages of algorithm iteration, the initial learning rate can be set to a smaller value, and the attenuation strategy of the learning rate can be adjusted according to the first preset iteration number, so that the update of the algorithm parameter amount is kept within a smaller range under the condition that the iteration number is not greater than the first preset iteration number.

[0117] In step S80, when the iterative number of the defect detection algorithm reaches no less than the second preset iterative number, the detection system no longer constrains the set parameter amount. This transition is based on the stage characteristics of the algorithm training process. In the early stage of algorithm training, as described in step S70, it is necessary to limit the update of the algorithm parameter amount to avoid overfitting noise, but as the iterative number increases, the algorithm gradually converges to a relatively stable state. At this time, releasing the constraints on the algorithm parameter amount allows the algorithm to further fine-tune the parameters to improve the accuracy of electronic component packaging defect detection. For example, when detecting multiple defect types (such as internal dielectric defects, external pin connection defects, etc.) in complex electronic component packaging structures (such as multilayer ceramic capacitor packaging), after a certain number of iterative numbers, the algorithm has a certain understanding of the data. At this time, releasing the constraints allows the algorithm to more deeply explore the feature relationships in the data and further optimize the detection capabilities of different types of defects.

[0118] In contrast to the early iteration stage in step S70, the early constraints are to prevent overfitting, while the relaxation of constraints at this time is to further improve the performance of the algorithm on the basis of avoiding overfitting. The two work together so that the algorithm can maintain good generalization ability throughout the training process and improve the fitting accuracy of specific data in the later stage. For example, when detecting pin defects in electronic component packaging, early constraints may enable the algorithm to roughly identify the location and basic structure of the pins, and after the constraints are relaxed in the later stage, the algorithm can more accurately detect the bending degree of the pins, the quality of the solder joints and other detailed features.

[0119] When the second preset iteration number is reached, the detection system can switch to a more aggressive parameter update strategy. For example, the learning rate can be adjusted to a relatively large value (within a reasonable range), or a more complex optimization algorithm can be used, such as the Adam (Adaptive Moment Estimation) algorithm. The Adam algorithm combines the ideas of the momentum method and the adaptive learning rate. It calculates the first-order moment estimate (similar to momentum) and the second-order moment estimate (used to adaptively adjust the learning rate) of each parameter. is the algorithm parameter, t is the number of iterations, is the first-order moment estimate, is the second-order moment estimate, and is the decay rate (usually close to 1), is a small positive number, and the initial learning rate is . Then at the tth iteration, ,Then In this way, the algorithm can adjust parameters more quickly to accommodate the update requirements of the algorithm parameter volume that is no longer constrained, thereby improving the accuracy of electronic component packaging defect detection.

[0120] To sum up, steps S70-S80 take measures to constrain and release the algorithm parameters at different iteration stages, so that the algorithm can avoid overfitting and improve the accuracy and generalization ability of electronic component packaging defect detection, thereby better meeting the needs of electronic component packaging quality inspection in actual production.

[0121] An embodiment of the present application provides a detection system, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0122] Figure 2 A hardware entity diagram of a detection system provided in an embodiment of the present application, such as Figure 2 As shown, the hardware entity of the detection system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0123] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for detecting defects in electronic component packaging based on image acquisition, characterized in that: The invention is applied to a detection system, wherein the detection system is provided with a defect detection algorithm, wherein the defect detection algorithm comprises u feature extraction sub-operators arranged in sequence and v information restoration sub-operators arranged in sequence, wherein each of the feature extraction sub-operators and each of the information restoration sub-operators comprises a standardization module and an operator composition module arranged in sequence, wherein u is a natural number not less than 2, and v is a natural number not less than 3, and wherein the method comprises: obtaining an electronic component packaging image; transmitting the electronic component packaging image to the u feature extraction sub-operators, and using each of the u feature extraction sub-operators to perform descriptor mining in sequence to obtain an image descriptor output by the u feature extraction sub-operators; wherein each of the feature extraction sub-operators is used to use the standardization module in sequence to extract the image descriptor; wherein The block and the operator composition module are used to perform descriptor mining, and the short-circuited path between the input and output ends of the standardization module is used to process the descriptor mining results of the feature extraction sub-operator; the image descriptor is transmitted to the v information restoration sub-operators, and each of the v information restoration sub-operators is used to perform descriptor mining in sequence to obtain the defect annotation descriptors output by the v information restoration sub-operators; wherein each of the information restoration sub-operators is used to perform descriptor mining using the standardization module and the operator composition module in sequence, and the short-circuited path is used to process the descriptor mining results of the information restoration sub-operator; and the defect annotation results of the electronic component packaging image are estimated according to the defect annotation descriptors.

2. The method according to claim 1, characterized in that The training process of the defect detection algorithm includes: acquiring an electronic component packaging training image; transmitting the electronic component packaging training image to the u feature extraction sub-operators, using each of the u feature extraction sub-operators to perform descriptor mining in sequence to obtain the image descriptor output by the u feature extraction sub-operators; each of the feature extraction sub-operators is used to perform descriptor mining in sequence using the standardization module and the operator composition module, and using the short-circuited path between the input and output ends of the standardization module to process the descriptor mining result of the feature extraction sub-operator; transmitting the image descriptor to the v information restoration sub-operators Sub-operators, using each of the v information restoration sub-operators to perform descriptor mining in sequence to obtain defect annotation descriptors output by the v information restoration sub-operators, each of the information restoration sub-operators is used to use the standardization module and the operator composition module in sequence to perform descriptor mining, and use the short-circuited path to process the descriptor mining results of the information restoration sub-operators; based on the defect annotation descriptors, the training defect annotation results of the electronic component packaging training images are estimated; based on the difference between the actual defect annotation results of the electronic component packaging training images and the training defect annotation results, the algorithm parameters of the defect detection algorithm are iterated.

3. The method according to claim 2, characterized in that The defect detection algorithm includes p feature extraction operators arranged in sequence, each of the feature extraction operators includes a number of the feature extraction sub-operators arranged in sequence in the u feature extraction sub-operators, and p is a natural number not less than 2; the electronic component packaging training image is transmitted to the u feature extraction sub-operators, and each of the feature extraction sub-operators in the u feature extraction sub-operators is used to perform descriptor mining in sequence to obtain the image descriptor output by the u feature extraction sub-operators, including: transmitting the electronic component packaging training image to the first feature extraction operator, and each of the feature extraction sub-operators in the first feature extraction operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the first feature extraction operator; transmitting the feature extraction result of the rth feature extraction operator to the r+1th feature extraction operator, and each of the feature extraction sub-operators in the r+1th feature extraction operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the r+1th feature extraction operator, where r is a natural number greater than 0, and r+1<p; the defect detection algorithm includes p feature extraction operators arranged in sequence, each of the feature extraction sub-operators in the r+1th feature extraction operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the r+1th feature extraction operator, and r is a natural number greater than 0, and r+1<p; The measurement algorithm includes p information restoration operators arranged in sequence, each of the information restoration operators includes at least two of the v information restoration operators arranged in sequence; the image descriptor is transmitted to the v information restoration operators, and each of the v information restoration operators is used to perform descriptor mining in sequence to obtain the defect annotation descriptor output by the v information restoration operators, including: transmitting the feature extraction result of the pth feature extraction operator to the first information restoration operator, using Each of the information restoration sub-operators in the first information restoration operator performs descriptor mining in sequence to obtain a feature extraction result of the first information restoration operator; the feature extraction result of the p-th feature extraction operator and the feature extraction result of the s-th information restoration operator are transmitted to the s+1-th information restoration operator, and each of the information restoration sub-operators in the s+1-th information restoration operator performs descriptor mining in sequence to obtain a feature extraction result of the s+1-th information restoration operator, where s is a natural number greater than 0, and s+1<p.

4. The method according to claim 3, characterized in that Each of the feature extraction operators includes a first feature extraction sub-operator and a second feature extraction sub-operator arranged in sequence, the operator composition module of the first feature extraction sub-operator adopts a joint attention mining mechanism, and the operator composition module of the second feature extraction sub-operator is a convolutional neural network; the feature extraction result of the rth feature extraction operator is transmitted to the r+1th feature extraction operator, and each of the feature extraction sub-operators in the r+1th feature extraction operator is used to perform descriptor mining in sequence to obtain the feature extraction result of the r+1th feature extraction operator, including: transmitting the feature extraction result of the rth feature extraction operator to the first feature extraction sub-operator of the r+1th feature extraction operator; using the standardization module and the joint attention mining mechanism of the first feature extraction sub-operator in sequence to perform descriptor mining; the feature extraction result of the rth feature extraction operator is transmitted to the first feature extraction sub-operator of the r+1th feature extraction operator; The feature extraction result of the first feature extraction sub-operator and the feature extraction result of the standardization module of the first feature extraction sub-operator are added by using the short-circuited path and the feature extraction result of the joint attention mining mechanism of the first feature extraction sub-operator to obtain the feature extraction result of the first feature extraction sub-operator; the feature extraction result of the first feature extraction sub-operator is transmitted to the second feature extraction sub-operator of the r+1th feature extraction operator; the standardization module of the second feature extraction sub-operator and the convolutional neural network are used in sequence to perform descriptor mining; the feature extraction result of the first feature extraction sub-operator and the feature extraction result of the standardization module of the second feature extraction sub-operator are added by using the short-circuited path and the feature extraction result of the convolutional neural network of the second feature extraction sub-operator to obtain the feature extraction result of the r+1th feature extraction operator.

5. The method according to claim 4, characterized in that The feature extraction result of the r-th feature extraction operator and the feature extraction result of the standardization module of the first feature extraction sub-operator are added by using the short-circuited path and the feature extraction result of the joint attention mining mechanism of the first feature extraction sub-operator to obtain the feature extraction result of the first feature extraction sub-operator, including: determining a first product result of the feature extraction result of the r-th feature extraction operator and a first influencing factor; and determining a second product result of the feature extraction result of the standardization module of the first feature extraction sub-operator and a second influencing factor; adding the first product result and the second product result by using the short-circuited path and the feature extraction result of the joint attention mining mechanism of the first feature extraction sub-operator. Calculation; the feature extraction result of the first feature extraction sub-operator and the feature extraction result of the standardization module of the second feature extraction sub-operator are added together with the feature extraction result of the convolutional neural network of the second feature extraction sub-operator using the short-circuited path to obtain the feature extraction result of the r+1th feature extraction operator, including: determining a third product result of the feature extraction result of the first feature extraction sub-operator and a third influencing factor; at the same time, determining a fourth product result of the feature extraction result of the standardization module of the second feature extraction sub-operator and a fourth influencing factor; the third product result and the fourth product result are added together with the feature extraction result of the convolutional neural network of the second feature extraction sub-operator using the short-circuited path.

6. The method according to claim 5, characterized in that The method further includes iterating one or more of the first influencing factor, the second influencing factor, the third influencing factor and the fourth influencing factor according to the difference between the actual defect annotation result of the electronic component packaging training image and the training defect annotation result.

7. The method according to claim 2, characterized in that The method further includes: performing standardization processing on the defect annotation descriptors output by the v information restoration sub-operators to obtain standardized defect annotation descriptors.

8. The method according to any one of claims 3 to 6, characterized in that: Each of the information restoration operators includes a first information restoration sub-operator, a second information restoration sub-operator and a third information restoration sub-operator arranged in sequence, the operator composition module of the first information restoration sub-operator is a first joint attention mining mechanism, the operator composition module of the second information restoration sub-operator is a second joint attention mining mechanism, and the operator composition module of the third information restoration sub-operator is a convolutional neural network; the feature extraction result of the pth feature extraction operator and the feature extraction result of the sth information restoration operator are transmitted to the s+1th information restoration operator, and each of the information restoration sub-operators in the s+1th information restoration operator is used in sequence Descriptor mining is performed to obtain the feature extraction result of the s+1th information restoration operator, including: transmitting the feature extraction result of the sth information restoration operator to the first information restoration sub-operator of the s+1th information restoration operator; sequentially using the standardization module of the first information restoration sub-operator and the first joint attention mining mechanism to perform descriptor mining; adding the feature extraction result of the sth information restoration operator and the feature extraction result of the standardization module of the first information restoration sub-operator, using the short-circuited path and the feature extraction result of the first joint attention mining mechanism of the first information restoration sub-operator, to obtain the feature extraction result of the sth information restoration operator. The feature extraction result of the first information restoration sub-operator is transmitted; the feature extraction result of the p-th feature extraction operator and the feature extraction result of the first information restoration sub-operator are transmitted to the second information restoration sub-operator of the s+1-th information restoration operator; the standardization module of the second information restoration sub-operator and the second joint attention mining mechanism are sequentially used to perform descriptor mining; the feature extraction result of the first information restoration sub-operator and the feature extraction result of the standardization module of the second information restoration sub-operator are added by the short-circuited path with the feature extraction result of the second joint attention mining mechanism of the second information restoration sub-operator. , obtain the feature extraction result of the second information restoration sub-operator; transmit the feature extraction result of the second information restoration sub-operator to the third information restoration sub-operator of the s+1th information restoration operator; sequentially use the standardization module and the convolutional neural network of the third information restoration sub-operator to perform descriptor mining; add the feature extraction result of the second information restoration sub-operator and the feature extraction result of the standardization module of the third information restoration sub-operator, using the short-circuited path and the feature extraction result of the convolutional neural network of the third information restoration sub-operator, to obtain the feature extraction result of the s+1th information restoration operator.

9. The method according to any one of claims 2 to 6, characterized in that: The method also includes: on the premise that the iterative number of the defect detection algorithm is not greater than a first preset iterative number, constraining the algorithm parameter amount of each iteration of the defect detection algorithm to be less than a set parameter amount; on the premise that the iterative number of the defect detection algorithm is not less than a second preset iterative number, no longer constraining the set parameter amount, and the second preset iterative number is greater than the first preset iterative number.

10. A detection system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.

Images