Chip packaging defect identification method and system based on machine vision

Abstract

The invention relates to the field of image processing, and particularly provides a chip packaging defect identification method and system based on machine vision, and the method comprises the steps: determining a to-be-identified part in a chip packaging image, and extracting a corresponding related image region; detecting to obtain interested image features in each extracted related image region, and determining interested image implicit representation under each channel; for each defect evaluation channel, determining a reference defect implicit representation of the to-be-identified part under the corresponding defect evaluation channel; according to the reference defect implicit representation of the to-be-identified part under each defect evaluation channel in the plurality of defect evaluation channels, determining defect evaluation parameter implicit representations respectively corresponding to the plurality of defect evaluation parameters; and obtaining a to-be-identified part defect identification result corresponding to the to-be-identified part according to the defect evaluation parameter implicit representation. According to the invention, each interested image feature in the chip packaging image can be integrated and defect identification can be carried out so that identification precision can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a chip packaging defect recognition method and system based on machine vision. Background Art

[0002] In the modern semiconductor industry, chip packaging is a key link in the chip manufacturing process. Chip packaging not only provides physical protection for the chip, but also plays an important role in electrical connection, heat dissipation, etc. However, in the chip packaging process, due to the influence of various complex factors, a variety of defects may occur, which will seriously affect the performance, reliability and service life of the chip.

[0003] With the continuous development of chip technology, the integration of chips is getting higher and higher, and the packaging structure is becoming more and more complex. Traditional methods of manually detecting chip packaging defects, such as visual inspection, have many limitations. In order to improve the efficiency and accuracy of chip packaging defect detection, machine vision technology has gradually been introduced into the field of chip packaging defect recognition. The machine vision system can quickly acquire images of chip packages and analyze the images through image processing algorithms. However, the existing chip packaging defect recognition technology based on machine vision is relatively rough, that is, the accuracy can be improved. Summary of the invention

[0004] The object of the present invention is to provide a chip packaging defect recognition method and system based on machine vision. The present invention is implemented as follows:

[0005] In a first aspect, the present invention provides a chip packaging defect recognition method based on machine vision, the method comprising: determining a component to be recognized in a chip packaging image, and extracting a relevant image area corresponding to the component to be recognized in the chip packaging image; detecting an image feature of interest in each extracted relevant image area, and determining an implicit image representation of interest for each detected image feature of interest in each of a plurality of preset defect assessment channels; for each of the plurality of defect assessment channels, based on the implicit image representation of interest for each image feature of interest in the targeted defect assessment channel, determining an implicit reference defect representation of the component to be recognized in the targeted defect assessment channel; based on the implicit reference defect representation of the component to be recognized in each of the plurality of defect assessment channels, determining implicit representations of defect assessment parameters corresponding to the component to be recognized under a plurality of defect assessment parameters deployed in advance; based on the implicit representations of defect assessment parameters corresponding to the component to be recognized under the plurality of defect assessment parameters, obtaining a defect recognition result of the component to be recognized corresponding to the component to be recognized.

[0006] In a second aspect, the present invention provides a computer system comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the above-mentioned method is implemented.

[0007] Beneficial effects of the present invention: The chip packaging defect recognition method and system based on machine vision provided by the present invention, for the components to be recognized in the chip packaging image, based on extracting the relevant image area corresponding to the components to be recognized in the chip packaging image, and detecting the image features of interest in each extracted relevant image area, preliminarily determine which defects the components to be recognized may belong to, based on determining the implicit representation of the image of interest of each detected image feature of interest in each defect assessment channel in a plurality of preset defect assessment channels, and for each defect assessment channel in a plurality of defect assessment channels, based on the implicit representation of the image of interest of each image feature of interest in the targeted defect assessment channel, determine the implicit representation of the reference defect of the components to be recognized in the targeted defect assessment channel, so as to determine the reference mode of the defect of the components to be recognized in the chip packaging image, and determine the implicit representation of the defect assessment parameters corresponding to the components to be recognized under the plurality of defect assessment parameters deployed in advance, and obtain the defect recognition result of the components to be recognized corresponding to the components to be recognized, so that the various image features of interest in the chip packaging image can be integrated and the defect recognition can be performed, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of a chip packaging defect recognition method based on machine vision provided by an embodiment of the present invention.

[0009] Figure 2 It is a schematic diagram of the composition of a computer system provided by an embodiment of the present invention.

[0010] Summary of reference numerals: 100 - computer system; 101 - processor; 102 - bus; 103 - memory; 104 - transceiver. DETAILED DESCRIPTION

[0011] The execution subject of the chip packaging defect recognition method based on machine vision in the embodiment of the present invention is a computer system, including but not limited to a server, a personal computer, a notebook computer, a tablet computer, a smart phone, etc. Figure 1 As shown, the method includes:

[0012] Step S100: determining a component to be identified in a chip package image, and extracting a relevant image region corresponding to the component to be identified in the chip package image.

[0013] In an embodiment of the present invention, when determining the components to be identified in the chip packaging image, the computer system obtains the chip packaging image. The chip packaging image is an image obtained by photographing the appearance of the chip after packaging, which contains the image information of various parts of the chip after packaging, such as the chip body, pins, packaging shell and other components. The computer system determines the components to be identified by the target detection algorithm in the image recognition technology. There are many target detection algorithms, such as the Convolutional Neural Network (CNN) algorithm based on deep learning. CNN extracts and classifies images through structures such as convolutional layers, pooling layers and fully connected layers. The computer system uses a pre-trained CNN model, which has learned the characteristic patterns of various components in the chip packaging image. The chip packaging image is input into the model, and the model outputs the category probability of each component in the image. The computer system determines the components to be identified in the chip packaging image according to the set threshold (for example, when the category probability of a component exceeds 0.8, it is considered that the component is detected). For example, when detecting pins in a chip package image, if the CNN model outputs a class probability of 0.9 for the pins, which exceeds the set threshold of 0.8, the computer system determines that the pins are components to be identified.

[0014] Next, the relevant image area corresponding to the component to be identified is extracted from the chip package image. After the computer system determines the component to be identified, the image area corresponding to the component is accurately found in the chip package image. Taking the chip body in the chip package image as an example, the computer system can extract the relevant image area by determining the position coordinates of the chip body in the image. Assuming that the chip package image is a two-dimensional plane image, its coordinate system is a Cartesian coordinate system. The computer system obtains the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) of the circumscribed rectangular frame of the chip body through the target detection algorithm. Then, the relevant image area is the rectangular area defined by these two coordinates, that is, the pixels in this area constitute the relevant image area corresponding to the chip body. The computer system can extract this area through the pixel matrix of the image. For example, using an image processing library (such as OpenCV) in Python language, the slicing operation can be used to obtain the pixel matrix of this area, that is, image[y1:y2, x1:x2], where image is the pixel matrix of the chip package image. This operation of extracting relevant image areas helps in the subsequent defect identification of specific components because it narrows the scope of analysis to the image portion directly related to the component to be identified, avoiding interference from other irrelevant parts in defect identification. It also reduces the amount of calculation and improves the efficiency and accuracy of defect identification.

[0015] In addition, when determining the components to be identified in the chip package image, the computer system can also use some preprocessing techniques to improve the accuracy of target detection. For example, the chip package image is grayed. Graying is the process of converting a color image into a grayscale image. The formula is: Gray=0.299*R+0.587*G+0.114*B, where Gray is the grayscale value, and R, G, and B are the values ​​of the red, green, and blue channels in the color image. The grayed image removes the color information and only retains the brightness information, which helps to reduce the amount of data and highlight the structure and texture information in the image, making it easier for the target detection algorithm to identify the components to be identified. At the same time, the computer system may also filter the image, such as Gaussian filtering. Gaussian filtering is a linear smoothing filter that reduces the noise in the image by weighted averaging each pixel in the image and its neighboring pixels. Through these preprocessing techniques, the quality of the chip package image can be improved, so that the components to be identified can be more accurately determined and the corresponding related image areas can be extracted.

[0016] Step S200: detecting an image feature of interest in each extracted relevant image region, and determining an implicit image representation of interest for each detected image feature of interest in each defect assessment channel of a plurality of preset defect assessment channels.

[0017] In an embodiment of the present invention, the computer system detects and obtains the image features of interest in each relevant image region extracted. The image features of interest refer to those image features that are of great significance for determining whether the chip packaging component has defects. These features may include edge features, texture features, shape features, etc. Taking edge features as an example, the edge is where the pixel value in the image changes sharply, which may correspond to the boundary of the chip packaging component or the connection between different parts. The computer system can obtain edge features through an edge detection algorithm. Common edge detection algorithms include Sobel operator, Canny operator, etc.

[0018] The Sobel operator is a discrete difference operator that detects edges by calculating the horizontal and vertical gradients of each pixel in the image. For a two-dimensional image I(x,y), the Sobel operator is used in the horizontal direction G x and vertical direction G y The convolution kernels are: , ; By convolving the two convolution kernels with the image respectively, the horizontal and vertical gradient components G are obtained x (x,y) and G y (x,y), then according to the formula Calculate the gradient magnitude G(x,y) of each pixel point. The place with a larger gradient magnitude is the location of the edge, thus obtaining the edge feature, which is an image feature of interest.

[0019] Texture can reflect information such as the microstructure and roughness of the chip package surface. The computer system can use the gray level co-occurrence matrix (GLCM) to describe texture features. GLCM is a method to describe texture by counting the frequency of pixel pairs with different gray values ​​in an image in a specific direction and distance. For example, when calculating the GLCM of a pixel pair with a horizontal distance of d, for each pixel point (x, y) in the image, the frequency of the pixel pair consisting of its gray value I (x, y) and I (x + d, y) in the image is counted, and a matrix P (i, j) is constructed, where i and j represent the gray values ​​of two pixels respectively. Texture feature parameters such as contrast, correlation, energy and entropy can be extracted from this matrix. For the detection of shape features, the computer system can use a contour-based method or a region-based method. The contour-based method is to first detect the contour of the object, and then obtain the shape features such as the perimeter, area, circularity, etc. of the contour by analyzing the contour. For example, for a closed contour C, its perimeter L can be obtained by accumulating the distances between adjacent pixels on the contour, and the area A can be obtained by calculating the number of pixels surrounded by the contour. The closer the circularity value is to 1, the closer the shape of the object is to a circle.

[0020] After detecting the image features of interest, the computer system determines the implicit image representation of each detected image feature of interest in each of the preset multiple defect assessment channels. The defect assessment channel is a channel set to evaluate the defects of the chip package from different angles or aspects, and each channel focuses on different types of defect related information.

[0021] To illustrate with a simple example, assume that there are two defect assessment channels, one channel focuses on the size-related defects of chip package components (such as whether the pin length meets the standard), and the other channel focuses on surface flatness-related defects (such as whether there are bulges or depressions on the package surface). For edge features, which are image features of interest, the implicit representation of the image of interest under the size-related defect assessment channel may be related to the size information of the component defined by the edge. For example, if the length of the pin represented by the edge can be calculated through the edge feature (assuming that the distance between the two end points of the pin is calculated by the coordinates of the edge point), then some normalized representation of this length information can be used as the implicit representation of the image of interest under the size-related defect assessment channel for the edge feature.

[0022] In the surface flatness related defect assessment channel, the implicit representation of the image of interest of the edge feature may be related to the smoothness of the edge. If the edge is relatively smooth, it may indicate that the surface flatness is good; if the edge has more jagged or irregular changes, it may indicate that the surface is uneven. The computer system can obtain an implicit representation related to the surface flatness by calculating the curvature change of the edge. For example, for each point on the edge, its local curvature k can be calculated (calculated by the second-order derivative, etc.), and then the number or proportion of points on the edge with large curvature changes can be counted, and this statistic can be used as the implicit representation of the image of interest of the edge feature in the surface flatness related defect assessment channel.

[0023] For texture features, in the size-related defect assessment channel, if the direction of the texture or the size of the texture unit has some correlation with the size of the component (for example, in certain specific chip packaging processes, the direction of the texture may be consistent with the arrangement direction of the pins, and the size of the texture unit may be related to the width of the pins), then the texture feature can be constructed based on this correlation to form an implicit representation of the image of interest in this channel. In the surface flatness-related defect assessment channel, the uniformity of the texture can be used as part of the implicit representation of the image of interest. If the texture is relatively uniform, it may indicate that the surface flatness is good; if the texture has obvious non-uniformity, it may indicate that there are defects on the surface. Computer systems can measure the uniformity of texture by calculating statistics such as the standard deviation of the texture.

[0024] Shape features also have different implicit representations under different defect assessment channels. Under the size-related defect channel, some size-related parameters of the shape feature (such as area, perimeter, etc.) can themselves be used as part of the implicit representation of the image of interest. Under the surface flatness-related defect assessment channel, features such as shape symmetry may be related to surface flatness. For example, if the shape of a chip packaging component is symmetrical, then ideally its surface flatness should be relatively good; if the shape is asymmetric, it may imply that there are surface unevenness or other defects in the asymmetric part. The computer system can obtain the implicit representation of the image of interest of the shape feature under this channel by calculating the symmetry of the shape (such as by comparing the contours or regional features on both sides of the symmetry axis).

[0025] Step S300: for each defect assessment channel among the multiple defect assessment channels, based on the implicit representation of the image of interest of each image feature of interest in the targeted defect assessment channel, determine a reference defect implicit representation of the component to be identified in the targeted defect assessment channel.

[0026] As mentioned above, the defect assessment channel is a specific way to evaluate chip package defects from different dimensions, and each channel focuses on different aspects of feature information related to the defects. Take the pin components in the chip package as an example. Assume that there are three defect assessment channels, namely the shape assessment channel, the size assessment channel, and the surface condition assessment channel. For the shape assessment channel, the computer system has obtained the implicit image of interest representation of each image feature of interest (such as the edge feature, shape feature, etc. of the pin) under this channel. For edge features, the implicit image of interest may be a value related to the smoothness of the edge; for shape features, it may be a value related to the deviation from the ideal shape of the pin.

[0027] When the computer system determines the implicit representation of the reference defect of the part (pin) to be identified under the shape evaluation channel, it can use a variety of technical means. One possible way is the weighted average method. The computer system assigns different weights to each image feature of interest according to its importance in evaluating the shape defect. Assume that the importance weight of the edge feature for shape evaluation is , the importance weight of the shape feature is ,and . Assume that the image of interest with edge features is implicitly represented as , the shape feature of the image of interest is implicitly represented as , then the implicit representation R of the reference defect of the pin under the shape evaluation channel can be expressed by the formula Calculated. Taking the dimension evaluation channel as an example, for the pin component, the image features of interest may include implicit representations of dimension-related features such as pin length and width extracted from the image. When determining the implicit representation of the reference defect of the pin under the dimension evaluation channel, the computer system may first normalize these different implicit representations of the image of interest so that they are in the same numerical range for subsequent calculation and comparison. For example, if the value range of the implicit representation L of the pin length is [0, 10], and the value range of the implicit representation W of the pin width is [0, 5], the computer system can normalize them to the range of [0, 1] through linear transformation. Assuming that the normalized implicit representation of the pin length is Ln and the implicit representation of the pin width is Wn, the computer system can calculate the reference defect implicit representation D of the pin under the dimension evaluation channel based on their relative importance in dimension evaluation, such as the importance of length is 0.6 and the importance of width is 0.4, then the implicit representation D of the reference defect of the pin under the dimension evaluation channel can be obtained by the formula D=0.6Ln+0.4Wn.

[0028] For the surface condition evaluation channel, it is assumed that the implicit representation of the image of interest of the image feature of interest includes values ​​related to the roughness of the surface texture and the uniformity of the surface color. When the computer system determines the implicit representation of the reference defect of the component (pin) to be identified under this channel, it may first perform some form of quantitative conversion on these implicit representations so that they can participate in the calculation in a unified manner. For example, the value of the surface texture roughness is converted to a relative value based on a standard roughness scale, and the value of the surface color uniformity is converted to the deviation ratio from the standard color uniformity. Then, similar to the previous method, a weighted sum is performed according to their relative importance in evaluating surface condition defects to obtain the implicit representation of the reference defect of the pin under the surface condition evaluation channel.

[0029] In addition to the weighted average method, the computer system can also use other statistical methods to determine the reference defect implicit representation. For example, the median method finds the middle value in the image implicit representation of all image features of interest as the reference defect implicit representation. This method can provide a relatively robust reference value in the presence of some outliers (which may be caused by image noise or local inaccurate feature extraction). Assume that under a certain defect assessment channel, there are n image implicit representations of interest for image features of interest. , first arrange these values ​​in ascending order. If n is an odd number, then the median is the (n+1) / 2th value; if n is an even number, the median is the average of the n / 2th value and the (n / 2+1)th value.

[0030] Another possible method is the majority method, which is to find the value that appears most frequently in the implicit representation of the image of interest of all image features of interest as the reference defect implicit representation. This method is applicable to some discretized implicit representations of the image of interest. For example, when the implicit representation of the image of interest is classified into several discrete levels (such as low risk, medium risk, and high risk), the mode can reflect the most common risk level, thereby providing a reference for the defect situation of the part to be identified under this defect assessment channel.

[0031] Step S400: based on the reference defect implicit representation of the component to be identified in each of the multiple defect assessment channels, determining the defect assessment parameter implicit representations corresponding to the component to be identified under multiple defect assessment parameters deployed in advance.

[0032] In the embodiment of the present invention, the defect assessment parameters are a set of indicators used to comprehensively assess the defect status of the chip packaging component, which characterize the possible defect conditions from different angles. For example, one of the defect assessment parameters may be a parameter related to the probability of defect occurrence, which reflects the probability of defect occurrence in the chip packaging component under a certain condition; another may be a parameter related to the distribution characteristics of the defect, such as whether the defect is concentrated or evenly distributed on the chip packaging component.

[0033] Taking the chip body in the chip package as an example of a component to be identified, it is assumed that in the previous step S300, for three defect evaluation channels (such as the physical structure integrity channel, the electrical performance correlation channel, and the material property channel), the computer system has obtained an implicit representation of the reference defects of the chip body under each channel.

[0034] For defect assessment parameters related to the probability of defect occurrence, the computer system can construct a probability model based on these reference defect implicit representations. For example, assume that the reference defect implicit representation under the physical structure integrity channel is R1, under the electrical performance correlation channel is R2, and under the material property channel is R3. The computer system can determine the implicit representation of the defect assessment parameters of the chip body under the defect assessment parameters through a predetermined functional relationship P=f(R1, R2, R3). This function f can be obtained through training based on a large amount of sample data, such as a mapping relationship obtained through training of a neural network model. If a simple linear weighted function form is adopted, assuming that the weights are w1, w2, and w3 (and w1+w2+w3=1), then P=w1R1+w2R2+w3R3. Here, P is the implicit representation of the defect assessment parameters of the chip body under the defect assessment parameters related to the probability of defect occurrence. It represents the comprehensive probability estimate of the occurrence of defects in the chip body. This probability value can be compared with a preset threshold. If P is greater than a certain threshold, it can be considered that the chip body has a high defect risk in this regard.

[0035] Consider the defect assessment parameters related to defect distribution characteristics. The computer system can analyze the defect distribution trend based on the implicit representation of the reference defects under each defect assessment channel. For example, if the implicit representation of the reference defects under the physical structure integrity channel reflects that there may be more structural defects at the edge of the chip body (for example, obtained through the comprehensive representation of the edge features under this channel), the representation under the electrical performance association channel shows that there may be potential defects related to electrical performance in the central area of ​​the chip (for example, obtained through the comprehensive representation of texture features related to electrical performance, etc.), the representation under the material property channel suggests that certain properties of the material on the entire chip surface may be non-uniform (for example, obtained through the comprehensive representation of shape features related to material properties, etc.).

[0036] The computer system can use a spatial distribution analysis technique to determine the implicit representation of the defect assessment parameter under the defect assessment parameter. For example, the surface of the chip body is divided into several small areas, and the probability of defects in each small area is determined according to the implicit representation of the reference defects under each channel. Assume that there are n small areas in total, and the comprehensive defect impact value of the i-th small area under three channels is (Here g is a function determined according to the specific situation, for example, it can be a simple function such as weighted sum.) The computer system can then calculate these Standard deviation of the values ,in yes The standard deviation σ can be used as an implicit representation of the defect evaluation parameters of the chip body under the defect evaluation parameters related to the defect distribution characteristics. If σ is large, it means that the distribution of defects on the chip body is relatively dispersed; if σ is small, it means that the defect distribution is relatively concentrated.

[0037] Another possible defect assessment parameter is a parameter related to the severity classification of the defect. The computer system can make a classification judgment based on the reference defect implicit representation under each defect assessment channel previously obtained. For example, each reference defect implicit representation has a corresponding numerical range corresponding to different defect severity levels. Assume that the reference defect implicit representation R under the physical structure integrity channel is 1 , when R 1 When it is in the range of [0, 0.3], it indicates a slight defect; when it is in the range of [0.3, 0.7], it indicates a moderate defect; when it is in the range of [0.7, 1], it indicates a serious defect; R under the electrical performance correlation channel and material property channel 2 and R 3 There is a similar grading range.

[0038] The computer system can determine the implicit representation of the defect assessment parameter of the chip body under this defect assessment parameter through a comprehensive judgment mechanism. For example, using the majority voting method, if the reference defect implicit representation of two or more channels in the three channels is determined to be a moderate defect, then the implicit representation of the defect assessment parameter of the chip body under the defect assessment parameter related to the defect severity grading is a moderate defect. Alternatively, a weighted voting method is used to assign different weights to each channel according to its importance in assessing the defect severity, and then the weighted results are calculated to determine the final defect severity level as the implicit representation of the defect assessment parameter.

[0039] Another defect assessment parameter may be related to the defect development trend. The computer system can use the chip packaging images acquired at different times (assuming that the images are collected at different stages of the same chip packaging process) and perform the previous steps at each time to obtain the reference defect implicit representation under each defect assessment channel. For example, let t 1 The reference defect under the physical structural integrity channel at time instant is implicitly expressed as R 11 , R under the electrical performance correlation channel 21 , R in the material property channel 31 ; at t 2 The corresponding expressions at each moment are R 12 , R 22 , R 32 .

[0040] The computer system can analyze the development trend of defects by calculating the difference. Then through a function To determine the implicit representation of the defect assessment parameters of the chip body under the defect assessment parameters related to the defect development trend. If T is positive and large, it means that the defect is developing in a serious direction; if T is negative and the absolute value is large, it means that the defect is improving or being repaired.

[0041] Step S500: obtaining a defect recognition result of the component to be recognized corresponding to the component to be recognized based on the implicit representation of the defect assessment parameters corresponding to the component to be recognized under multiple defect assessment parameters.

[0042] In the embodiment of the present invention, the implicit representation of the defect assessment parameters corresponding to the component to be identified under multiple defect assessment parameters is the result calculated in the previous step S400, and these results reflect the possible defects of the component to be identified (such as the chip body, pins, etc. in the chip package) from different angles. Each defect assessment parameter focuses on a specific aspect of the defect, such as defect probability, defect distribution characteristics, defect severity classification, and defect development trend.

[0043] Taking the pins in the chip package as an example, it is assumed that in step S400, the defect assessment parameter of the pin under the defect probability assessment parameter is implicitly expressed as P (the value range is between 0 and 1, 0 indicates no defect possibility, and 1 indicates that a defect must exist), the defect assessment parameter under the defect distribution characteristic assessment parameter is implicitly expressed as \sigma (for example, obtained by calculating the standard deviation, indicating the degree of discreteness of the defect distribution), the defect assessment parameter under the defect severity grading assessment parameter is implicitly expressed as G (for example, G=1 indicates a slight defect, G=2 indicates a moderate defect, and G=3 indicates a severe defect), and the defect assessment parameter under the defect development trend assessment parameter is implicitly expressed as T (a positive value indicates that the defect development tends to be serious, and a negative value indicates that it tends to improve).

[0044] The process of obtaining the defect identification result of the component to be identified by the computer system can adopt a variety of technical means. A common way is to set a threshold judgment method. For the defect probability P, if the threshold set by the computer system is P th (For example, P th = 0.5), when P > P th When the computer system is 100%, it can preliminarily determine that the pin has a high risk of defect. This is based on the statistics and analysis of a large amount of sample data, and it is determined that the threshold of 0.5 can better distinguish between defective and non-defective situations.

[0045] For the defect distribution characteristic σ, the computer system can set different intervals according to the chip packaging process standards and empirical data. ( is a smaller value), it can be considered that the defect distribution is relatively concentrated; if , it is considered that the defect distribution is relatively uniform; if , then the defect distribution is considered to be very dispersed. Assume , if the computer system calculates that the pin's σ=0.4, then it can be concluded that the pin's defects are distributed evenly. This judgment helps to further analyze the causes of defects. For example, evenly distributed defects may be related to a global factor in the entire production process, such as the overall non-uniformity of the material; while concentrated defects may be related to local process problems, such as processing equipment failure in a specific area.

[0046] In terms of defect severity grading, when G=1, the computer system can determine that the pin has a slight defect, which may mean that although some parameters of the pin deviate from the standard value, they are still within an acceptable small range and have little impact on the overall function of the chip. For example, the length deviation of the pin is within ±0.05mm (assuming the standard length is 1mm). This slight deviation may not affect the connection function of the pin with other components, but it is still marked as a defect for subsequent quality monitoring. When G=2, it indicates a moderate defect, which may be that the shape of the pin is distorted to a certain extent, affecting the stability of its connection with other components, but it has not completely lost its function. When G=3, serious defects indicate that the pin has problems such as breakage and severe deformation, which will directly cause the chip to fail to work properly.

[0047] For the defect development trend T, if T > 0 and the value of T is large (for example, T > 0.3), the computer system can determine that the defects of the pins are developing and worsening, which may indicate that there are some factors that continue to affect the quality of the pins on the production line, such as the continuous increase in ambient temperature leading to increased material deformation. If T < -0.3, it means that the defects are improving, which may be due to the adjustment of the production process or the optimization of the equipment.

[0048] In addition to the threshold judgment method, the computer system can also use a comprehensive evaluation model to obtain the defect identification results of the components to be identified. For example, a model based on fuzzy logic can be constructed. In this model, the defect probability P, defect distribution characteristics σ, defect severity classification G and defect development trend T are used as input variables, and each variable has a corresponding fuzzy set and membership function.

[0049] Taking the defect probability P as an example, three fuzzy sets are defined: "low probability (LP)", "medium probability (MP)", and "high probability (HP)". The membership function can be a triangular membership function. For example, for "low probability (LP)", when P≤0.3, the membership is 1, and when P=0.5, the membership is 0; for "medium probability (MP)", when P=0.4, the membership is 0, when P=0.5, the membership is 1, and when P=0.6, the membership is 0; for "high probability (HP)", when P≥0.7, the membership is 1, and when P=0.5, the membership is 0.

[0050] Similarly, corresponding fuzzy sets and membership functions are defined for the defect distribution characteristics σ, defect severity classification G, and defect development trend T. Then, reasoning is performed based on fuzzy logic rules, such as "If P is high probability (HP) and G is severe (3) and T is development deterioration (large positive value), then the defect condition of the pin is very severe (VS)" and similar rules. Finally, a definite defect identification result is obtained through the defuzzification process, such as "The pin has a serious defect and should be repaired or replaced immediately."

[0051] The computer system can also use a decision tree-based method to obtain the defect recognition result of the component to be identified. The root node of the decision tree can be the defect probability P, and the defect probability P can be obtained according to the difference between P and the set threshold (such as P th ) is divided into two branches. If P≤P th , and then further branch judgment according to the defect severity classification G; if P > P th , then branch judgment is performed according to the defect distribution feature σ. Through such a step-by-step decision-making process, the defect recognition result of the pin is finally obtained.

[0052] As an implementation manner, in step S200, detecting the image feature of interest in each extracted relevant image region may include:

[0053] Step S210: determining each relevant image region as a target relevant image region;

[0054] Step S220: performing an image block operation on the target-related image region to obtain one or more image blocks included in the target-related image region;

[0055] Step S230: For each image block in the one or more image blocks, determine a preset image feature of interest associated with the targeted image block in a pre-deployed set of image features of interest, and determine the preset image feature of interest associated with the targeted image block as the image feature of interest detected in the target-related image area.

[0056] In the previous step S100, the computer system has extracted relevant image areas corresponding to the components to be identified in the chip package image. These relevant image areas are image parts associated with specific components in the chip package (such as the chip body, pins, etc.). Taking the pins in the chip package as an example, it is assumed that through step S100, the computer system has determined the relevant image area where the pins are located, and this area contains visual information such as the shape, color, and texture of the pins. When executing step S210, the computer system separately regards this relevant image area corresponding to the pins as a target relevant image area, so as to subsequently detect the image features of interest for this specific area. The purpose of this step is to decompose the complex chip package image into individual areas that can be analyzed separately corresponding to the components to be identified, laying the foundation for accurately detecting the image features of interest.

[0057] Next is step S220, where the computer system performs an image segmentation operation on the target-related image area to obtain one or more image blocks included in the target-related image area. Image segmentation is an operation that divides the target image (here, the target-related image area) into multiple smaller sub-images (image blocks) according to certain rules. The computer system can adopt a variety of segmentation strategies, such as segmentation according to squares or rectangles of fixed size. Assuming that the target-related image area is a rectangular area of ​​100×100 pixels, if it is segmented according to squares of 10×10 pixels, then this area will be divided into 100 image blocks of 10×10 pixels. This segmentation operation helps to analyze the features of the target-related image area in more detail, because different image blocks may contain different local feature information. For the pin target-related image area in the chip package image, after segmentation, one image block may contain the edge of the pin, and another image block may contain partial information of the connection between the pin and the chip body. This local feature information is very important for accurately detecting image features of interest, because different local areas may correspond to different types of image features of interest. For example, the edge part is more likely to contain information related to the edge feature, while the connection part may contain some texture or shape feature information related to the connection relationship.

[0058] Then comes step S230, where for each image block in the one or more image blocks, the computer system determines, from a pre-deployed set of image features of interest, a preset image feature of interest associated with (i.e., matched to) the targeted image block, and determines the preset image feature of interest associated with the targeted image block as the image feature of interest detected in the target-related image region.

[0059] The pre-deployed image feature set of interest is a set of image features that are pre-defined by the computer system and may be of great significance to the identification of chip packaging defects. This set may contain multiple types of image features, such as edge features, texture features, shape features, etc. Taking edge features as an example, edge features may exist in different forms in this set, such as edge detection templates for different directions (horizontal, vertical, inclined, etc.) or edge feature description forms obtained based on different algorithms (such as Sobel operator, Canny operator, etc.).

[0060] For each image block, the computer system determines the preset image feature of interest associated with it. Assume that an image block in the target-related image area is located at the edge of the chip package pin, and the pixel value of this image block has obvious changes in the horizontal direction. In the pre-deployed image feature set of interest, there is an edge feature description form obtained by detecting the Sobel operator in the horizontal direction. By analyzing the matching relationship between the pixel change pattern of the image block and various preset image features of interest in the set, the computer system finds that this image block is most matched with the edge feature description form based on the Sobel operator in the horizontal direction. The matching relationship here can be determined in a variety of ways, such as comparing the similarity between the pixel gradient direction of the image block and the pixel gradient direction corresponding to the preset image feature of interest.

[0061] When determining the matching relationship, the computer system can use a similarity calculation method based on direction vectors. Assume that the pixel gradient direction of the image block can be represented by a two-dimensional vector Represented by , and the pixel gradient direction corresponding to the preset image feature of interest (taking the horizontal Sobel operator edge feature as an example) is represented by the vector Then the similarity S between them can be measured by calculating the cosine value of the vector angle, and the formula is When the value of S is close to 1, it means that the two direction vectors are very similar, that is, the image block has a high degree of match with the preset image feature of interest.

[0062] When the computer system determines the preset image feature of interest associated with the image block, it determines the preset image feature of interest as the image feature of interest detected in the target-related image area. Continuing with the above example, if it is determined that the edge feature description form based on the horizontal Sobel operator matches the image block, then this edge feature is determined as the image feature of interest detected in the pin target-related image area. For other image blocks in the target-related image area, the computer system will repeat this process and determine the preset image features of interest that match them respectively, thereby obtaining multiple image features of interest in the target-related image area. These image features of interest will serve as an important basis for analyzing chip packaging defects in subsequent steps. They reflect the characteristics of the target-related image area from different local areas and feature types, which helps to conduct more comprehensive and accurate defect assessment.

[0063] For another example, for the target-related image area of ​​the chip body in the chip package image, if one of the image blocks is located in the central area of ​​the chip body and has a relatively complex texture structure. In the pre-deployed set of image features of interest, there are multiple texture features described based on the gray-level co-occurrence matrix (GLCM). The computer system will analyze the relationship between the texture structure of this image block and the GLCM texture feature. This can be done by comparing the statistical parameters of the texture features, such as contrast, correlation, energy, and entropy. Assume that the texture contrast of this image block is high, and in the GLCM texture features, there is a preset texture feature description for high-contrast texture structures. The computer system determines the matching relationship by comparing the degree of proximity between the two in contrast and other relevant texture statistical parameters. If a match is determined, the preset texture feature based on GLCM is determined as the image feature of interest corresponding to the image block.

[0064] For example, in terms of shape features, if an image block in the target-related image area contains a local area of ​​an irregular shape of a chip package, and there is a shape feature description based on contour analysis in the pre-deployed image feature set of interest, the computer system can compare the contour features of the image block (such as the curvature change of the contour, the proportional relationship between the perimeter and the area, etc.) with the matching degree of the corresponding parameters in the preset shape feature description. For example, the perimeter L of the image block contour is calculated. 1 and area A 1 , and calculate the ratio of perimeter to area , and there is a perimeter and area ratio R corresponding to a specific shape in the preset shape feature description 2 , by comparison If the difference is small, it means that the shape features of the two are relatively matched, so the preset shape feature is determined as the image feature of interest corresponding to the image block.

[0065] As an implementation manner, in step S230, determining a preset image feature of interest associated with the targeted image block from a pre-deployed image feature set of interest includes:

[0066] Step S231: determining a feature Euclidean distance between a targeted image block and a preset image feature of interest in a pre-deployed image feature set of image features of interest;

[0067] Step S232: if the set of image features of interest includes a target preset image feature of interest whose feature Euclidean distance meets the first requirement, the target preset image feature of interest is determined as the preset image feature of interest associated with the targeted image block.

[0068] In step S231, the pre-deployed image feature set of interest includes various types of preset image features of interest, which are pre-defined features that are important for chip packaging defect recognition. For example, it may include edge features of different directions and strengths, various texture features, and multiple shape features. For an image block, it is a sub-region obtained from the target-related image region through a block operation, which contains local visual information of the chip packaging component.

[0069] The feature Euclidean distance is a measure of the difference between two features. Assume that the features of the image block can be represented by a feature vector Here n represents the dimension of the feature vector, and each dimension represents an attribute of the feature. For example, for the edge feature vector, may indicate the strength of the edge, It may represent the direction of the edge, etc. For texture feature vectors, may indicate the contrast of the texture, It may represent the energy of texture, etc. For shape feature vectors, may represent the perimeter of a shape, Might represent the area of ​​a shape, etc.

[0070] Similarly, the preset image features of interest can also be represented by feature vectors in the set, set as Then, the computer system calculates the Euclidean distance d between the two feature vectors using the formula .

[0071] Take the pin image block in the chip package as an example. Assume that the feature vector representation of this image block is related to the edge feature, where Indicates that the edge strength is 0.8 (the normalized value ranges from 0 to 1). Indicates that the edge direction is 45° (a value quantized by a certain angle), etc. In the pre-deployed image feature set of interest, there is a preset edge feature vector , where y 1 = 0.7 (edge ​​strength), (edge ​​direction). According to the above Euclidean distance formula, we can get: .

[0072] The calculated Euclidean distance value quantifies the difference between the image block and the preset image feature of interest at the feature level. For different preset image features of interest (such as edge features of different directions and intensities or other types of textures, shape features, etc.), the computer system will calculate the Euclidean distance between them and the image block feature vector.

[0073] Next is step S232, where the first requirement is a judgment standard about the Euclidean distance preset by the computer system. The setting of this standard depends on the specific application scenario and the requirement for feature matching accuracy. For example, in the case of high requirements for chip packaging defect recognition, the first requirement may be set to a smaller Euclidean distance value to ensure that a preset image feature of interest that is very similar to the image block feature is found.

[0074] Continuing with the pin image block as an example, if the first requirement is set to a Euclidean distance less than 5 (the 5 here is just an example value, which is determined according to specific circumstances in actual applications), and in the process of calculating the Euclidean distances with each preset edge feature, it is found that a preset edge feature has a Euclidean distance of 3 with the image block, and this preset edge feature meets the first requirement. Then the computer system determines this preset edge feature as a preset image feature of interest associated with the pin image block.

[0075] Consider another example of texture features. Assume that the image block comes from a certain area on the surface of the chip package, and its texture feature vector Indicates the contrast, energy, entropy and other properties of the texture. In the texture feature part of the pre-deployed image feature set of interest, there are multiple preset texture feature vectors. The computer system calculates the Euclidean distance between the image block and these preset texture feature vectors respectively. If for a preset texture feature vector , the calculated Euclidean distance meets the first requirement (for example, is less than a preset value), then the preset texture feature is determined as the preset image feature of interest associated with the image block.

[0076] The same is true for shape features. For example, an image block contains the shape information of a part of a chip package, and its shape feature vector represents the perimeter, area, circularity and other attributes of the shape. The computer system calculates the Euclidean distance between the image block and various preset shape feature vectors in the pre-deployed set of image features of interest. If the Euclidean distance between a preset shape feature vector and the image block meets the first requirement, it is determined as a preset image feature of interest associated with the image block.

[0077] As an implementation manner, in step S200, determining the implicit image representation of each detected image feature of interest in each defect assessment channel of a plurality of preset defect assessment channels may include:

[0078] Step S240: for each detected image feature of interest, obtaining a preset image of interest implicit representation of a preset image of interest associated with the targeted image feature of interest in each defect assessment channel of a plurality of defect assessment channels deployed in advance;

[0079] Step S250: determining an implicit image of interest representation for the image of interest feature in each defect assessment channel in the plurality of defect assessment channels according to the obtained preset implicit image of interest representation.

[0080] In step S240, the image features of interest are determined in the previous steps (such as steps S210-S230), and these features may include edge features, texture features, shape features, etc. The preset image features of interest are pre-defined and have some association with the detected image features of interest (for example, they have similarities or correspondences in feature types, attributes, etc.).

[0081] Defect assessment channels are specific channels set up to evaluate chip package defects from different angles or aspects. For example, one defect assessment channel may focus on the defect assessment of the geometric structure of the chip package, such as whether the shape of the component meets the standard and whether the size is within a reasonable range; another channel may focus on defects in surface characteristics, such as surface flatness, roughness, and texture uniformity; there may also be defect assessment channels related to electrical performance (if the chip package is directly related to electrical performance), focusing on defects such as electrical connectivity between pins and the impact of packaging materials on electrical performance.

[0082] The preset implicit representation of the image of interest is a pre-set representation form for the preset image of interest features in each defect assessment channel. This representation form is processed and abstracted, and can reflect the relationship between the preset image of interest features and the defect aspects that the corresponding defect assessment channel focuses on. Taking edge features as an example, in the geometric structure defect assessment channel, its preset implicit representation of the image of interest may be related to the continuity, linearity and other attributes of the edge; in the surface feature defect assessment channel, it may be related to the boundary characteristics of the surface texture reflected by the edge.

[0083] Assuming that an edge feature is detected as an image feature of interest in the chip package image, the computer system will search for the preset edge feature (preset image feature of interest) associated with it. For the geometric structure defect assessment channel, the preset image of interest implicit representation of the preset edge feature in the channel may be a quantitative value based on the degree of deviation of the bending degree of the edge from the ideal straight edge. If the edge of the ideal chip package component should be completely straight (under the design requirements of the geometric structure), the implicit representation of the preset edge feature may be a value obtained by calculating the average distance from each point on the edge to an ideal straight line (for example, this value is 0.1, indicating the average deviation from the ideal straight line). For the surface feature defect assessment channel, the preset image of interest implicit representation of the preset edge feature may be a value related to the frequency of texture changes in the area adjacent to the edge (for example, if the texture of the area adjacent to the edge changes frequently, it may indicate that the surface characteristics are abnormal, and the value may be a higher value, such as 0.8).

[0084] Then comes step S250, in which the process of the computer system determining the implicit image representation of the image feature of interest may involve a variety of technical means. One way is to convert based on a proportional relationship or a mapping function. Continuing with the above-mentioned edge feature as an example, assuming that after obtaining the preset implicit image representation of interest of the preset edge feature under the geometric structure defect assessment channel (such as the degree of deviation is 0.1), if the detected edge feature of interest has a certain proportional relationship with the preset edge feature, for example, the curvature of its edge is twice the curvature of the preset edge feature, then the computer system can determine that the implicit image representation of interest of the image feature of interest under this channel is 0.2 through a simple proportional calculation.

[0085] Another way is to use a weighted combination based on features. If the preset implicit representation of the image of interest is composed of multiple implicit representations of sub-features, for the image of interest feature, the computer system can determine the weight according to the relationship between each of its sub-features and the sub-features of the preset image of interest feature, and then perform weighted calculation to obtain the implicit representation of the image of interest. For example, in the surface characteristic defect evaluation channel, the preset implicit representation of the image of interest of the preset edge feature is composed of the texture change frequency of the edge adjacent area (weight is 0.6) and the roughness of the edge (weight is 0.4). For the detected edge feature of interest, the computer system analyzes the relationship between the texture change frequency and edge roughness of the edge adjacent area and the preset value. Assuming that the texture change frequency is 1.5 times the preset value and the edge roughness is 0.8 times the preset value, the implicit representation of the image of interest of the image of interest feature in this channel is (1.5×0.6+0.8×0.4)=0.9+0.32=1.22.

[0086] Taking texture features as an example, in a defect assessment channel related to color uniformity (here it is assumed that texture features are associated with color uniformity, for example, changes in texture may affect the visual perception uniformity of color), the preset image of interest implicit representation of the preset texture feature may be based on the grayscale standard deviation of the texture (assuming it is 0.3, a measure of the grayscale change of the texture). If the grayscale standard deviation of the detected texture feature of interest is 1.3 times the preset value, the computer system can determine that the image of interest implicit representation of the image of interest feature in the defect assessment channel is 0.39.

[0087] For shape features, under a defect assessment channel related to component symmetry, the preset image of interest implicit representation of the preset shape feature may be based on the symmetry deviation of the shape (for example, 0.2, indicating the degree of deviation from the ideal symmetrical shape). If the symmetry deviation of the detected shape feature of interest is 1.2 times the preset value, the computer system determines that the image of interest implicit representation of the image of interest under this channel is 0.24.

[0088] As an implementation manner, step S300, for each defect assessment channel among the multiple defect assessment channels, based on the implicit representation of the image of interest of each image feature of interest in the targeted defect assessment channel, determines a reference defect implicit representation of the component to be identified in the targeted defect assessment channel, including:

[0089] Step S310: for each defect assessment channel among the multiple defect assessment channels, based on the implicit representation of the image of interest of each image feature of interest in the targeted defect assessment channel, determining a mean value of the implicit representation of the defect of the component to be identified in the targeted defect assessment channel;

[0090] Step S320: determining a reference defect implicit representation of the component to be identified in the targeted defect assessment channel according to the mean value of the defect implicit representation of the component to be identified in the targeted defect assessment channel.

[0091] In step S310, the implicit representation of the image of interest is a representation form determined in the previous step (such as step S200) for each image feature of interest under a specific defect assessment channel, which reflects the relationship between the image feature of interest and the defect-related information concerned by the defect assessment channel. Taking the pin component in the chip package as an example, it is assumed that there are two image features of interest, edge features and shape features, under the shape defect assessment channel, and they have their own implicit representations of images of interest under this channel. The implicit representation of the image of interest for the edge feature may be related to the smoothness of the edge, for example, a numerical value is used to represent the smoothness of the edge, and the higher the numerical value, the smoother the edge; the implicit representation of the image of interest for the shape feature may be related to the shape regularity of the pin, for example, a numerical value is used to represent the degree of deviation from the ideal shape, and the smaller the numerical value, the closer the shape is to the ideal shape.

[0092] The computer system can use a simple arithmetic mean method to determine the mean of the implicit representation of defects. Assume that the number of image features of interest is n, and the implicit representation of the image of interest of the i-th image feature of interest in the defect assessment channel is: , then the defect implicitly represents the mean The calculation formula is For example, for the pin shape defect assessment channel mentioned above, if the image of interest of the edge feature implicitly represents (indicates smoother edges), the shape features of the image of interest are implicitly represented (indicates that the shape deviates from the ideal shape to a certain extent), where n=2, the defect implicitly represents the mean .

[0093] In addition to the arithmetic mean method, the computer system can also assign different weights according to the importance of different image features of interest in defect assessment, and use the weighted average method to calculate the mean of the implicit representation of the defect. Suppose the weight of the i-th image feature of interest is (and ), then the weighted average formula is For example, if we determine the importance weight of edge features in shape defect assessment =0.6, the weight of the shape feature =0.4, according to the previous =0.8 and =0.2, then the defect implicitly represents the mean This weighted average approach can more flexibly calculate the mean based on the contribution of different features to defect assessment.

[0094] Next, in step S320, the process of determining the reference defect implicit representation by the computer system may involve comparison and mapping with some pre-set standards or thresholds. Continuing with the pin shape defect evaluation channel as an example, it is assumed that different interval ranges are set to correspond to different defect reference levels based on a large amount of experimental data and chip packaging process requirements. If the defect implicit representation mean In the range of [0, 0.3], it means that the pin has almost no defects in shape. The reference defect implicit representation can be set to a value indicating no defects or low risk, such as R=0.1; if In the range of (0.3, 0.7], it means that there is a certain degree of shape defect risk, and the reference defect implicit representation can be set to R = 0.5; if > 0.7, indicating a high risk of shape defects, and the reference defect implicit representation can be set to R = 0.9.

[0095] Another way to determine the implicit representation of the reference defect is through function mapping. For example, we can define a function R = f( ), this function can be constructed based on the specific process characteristics of chip packaging, historical data and expert experience. Assume that the function is ,when =0.5 (the weighted average defect implicit representation mean value calculated previously), R=0.33, and this R value is used as the reference defect implicit representation of the component (pin) to be identified in the shape defect assessment channel.

[0096] Consider the case of the chip body in the chip package under the material characteristic defect assessment channel. Assume that there are interesting image features such as texture features and color features under this channel, and they each have an implicit representation of the image of interest. If the arithmetic mean is used to calculate the mean of the defect implicit representation After that, the reference defect implicit representation can also be determined according to a pre-set standard or function mapping. For example, according to the requirements of material properties, if is less than a certain threshold, indicating that the material properties are good and the reference defect is implicitly represented as a lower value; if Beyond this threshold, the reference defect implicit representation increases accordingly, indicating that the material properties may be at risk of defects.

[0097] As an implementation mode, the defect assessment parameter includes a reference defect assessment parameter, and the defect assessment parameter implicit representation of the component to be identified under the reference defect assessment parameter includes the reference defect assessment implicit representation. Based on this, step S400, based on the reference defect implicit representation of the component to be identified under each defect assessment channel in multiple defect assessment channels, determines the defect assessment parameter implicit representations corresponding to the component to be identified under multiple defect assessment parameters deployed in advance, including:

[0098] Step S410: for each defect assessment channel in the plurality of defect assessment channels, based on the reference defect implicit representation of the component to be identified in each defect assessment channel in the plurality of defect assessment channels and a pre-deployed probability range, determine a reference defect assessment implicit representation of the component to be identified under reference defect assessment parameters.

[0099] The reference defect implicit representation of the component to be identified in each defect assessment channel is the result calculated in the previous step (such as step S300), and these results reflect the possible defect conditions of the component to be identified from the perspective of different defect assessment channels. For example, in a shape defect assessment channel for chip package pins, the reference defect implicit representation may reflect the degree of deviation between the pin shape and the ideal shape; in a defect assessment channel for chip surface texture, the reference defect implicit representation may reflect the difference between the texture uniformity and the standard texture, etc.

[0100] The pre-deployed probability ranges are numerical ranges related to the probability of defects that are pre-set by the computer system. These ranges are determined based on a large amount of chip packaging sample data, industry standards, and empirical knowledge. For example, there may be a low probability range (such as 0-0.3), which means that within this range, the probability of defects in the component is low; a medium probability range (such as 0.3-0.7) indicates a certain probability of defects; and a high probability range (such as 0.7-1) indicates that the probability of defects in the component is very high.

[0101] Take the chip body in the chip package as an example. Assume that there are three defect assessment channels: structural integrity channel, electrical performance channel and material property channel. Under the structural integrity channel, the reference defect of the chip body calculated by the computer system is implicitly expressed as R 1 =0.4; in the electrical performance channel, the reference defect is implicitly expressed as R 2 = 0.3; in the material property channel, the reference defect is implicitly expressed as R 3 =0.5.

[0102] When the computer system determines the reference defect evaluation implicit representation of the component to be identified (chip body) under the reference defect evaluation parameters, a variety of technical means can be used. One method is based on weighted summation and probability range mapping. It is assumed that the weight of the structural integrity channel is set according to the importance of each defect evaluation channel in the overall defect evaluation. =0.4, the weight of the electrical performance channel =0.3, weight of material property channel =0.3 (and ).

[0103] First calculate the weighted sum .

[0104] Then, the mapping is performed according to the pre-deployed probability range. Since 0.3 < R=0.4 < 0.7, this value falls within the medium probability range. The computer system can further quantize this medium probability range into a specific value as the reference defect assessment implicit representation under the reference defect assessment parameters. For example, within the medium probability range, a linear mapping method can be used. Suppose the quantization interval of the reference defect assessment implicit representation corresponding to the medium probability range is [0.4, 0.6] (this is also set based on experience or data), then the formula .here It is the implicit representation of the reference defect assessment of the chip body under the reference defect assessment parameters.

[0105] Another possible technical approach is based on probability distribution function. Assume that there is a predetermined probability distribution function P(x), which describes the relationship between the reference defect implicit representation and the defect probability. For each defect evaluation channel, the reference defect implicit representation , the computer system substitutes it into the probability distribution function to obtain the corresponding probability value .

[0106] Continuing with the above chip body example, Substituting into the probability distribution function P(x) respectively, we get Then, these probability values ​​are weighted and summed according to the weight of each channel to obtain the comprehensive probability value Finally, this comprehensive probability value Transformed into a reference defect assessment implicit representation through some mapping relationship. For example, if The value of is between 0-0.3, corresponding to the reference defect evaluation implicit representation of 0.1; between 0.3-0.7, corresponding to 0.5; between 0.7-1, corresponding to 0.9, etc.

[0107] Consider the pin components in the chip package. Assume that the reference defect is implicitly expressed as R under the pin shape evaluation channel. s=0.6, R in the electrical connectivity evaluation channel e = 0.2, and the weight of the shape evaluation channel =0.6, the weight of the electrical connectivity evaluation channel =0.4.

[0108] By weighted summation According to the pre-deployed probability range (assuming the same range setting as the chip body), 0.44 falls within the medium probability range (0.3-0.7). If the above linear mapping method is used (assuming that the quantization interval corresponding to the medium probability range is also the same), the reference defect evaluation implicit representation is calculated .

[0109] As another implementation, step S400, based on the reference defect implicit representation of the component to be identified in each defect assessment channel in the multiple defect assessment channels, determines the defect assessment parameter implicit representations corresponding to the component to be identified under the multiple defect assessment parameters deployed in advance, including:

[0110] Step S400A: for each defect assessment channel among the plurality of defect assessment channels, determining a feature error between an implicit representation of the image of interest under the current defect assessment channel and an implicit representation of a reference defect under the current defect assessment channel for each detected image feature of interest;

[0111] Step S400B: Based on the characteristic error determined in the current defect assessment channel, determine the implicit representation of the defect assessment parameters of the to-be-identified component corresponding to the plurality of defect assessment parameters deployed in advance in the current defect assessment channel.

[0112] In step S400A, the implicit representation of the image of interest is a representation form determined in the previous step for each feature of the image of interest under a specific defect assessment channel, which reflects the relationship between the feature of the image of interest and the defect-related information concerned by the defect assessment channel. The implicit representation of the reference defect is a comprehensive representation of the overall defect status of the component to be identified under the defect assessment channel.

[0113] Taking the pins in chip packaging as an example, assume that there are two image features of interest in the shape defect assessment channel: edge features and shape features. For edge features, their implicit representation of the image of interest in the shape defect assessment channel may be related to the smoothness and straightness of the edge, etc., assuming that ; The implicit representation of the image of interest under this channel may be related to the regularity of the pin shape, assuming that In the shape defect evaluation channel, the reference defect implicit representation of the pin is assumed to be R.

[0114] The computer system can use a simple difference calculation method to determine the feature error. For edge features, the feature error ; For shape features, feature error For example, if =0.8 (indicating smoother edges and better straightness), R=0.6 (indicating a comprehensive representation of the overall shape defect degree), then E 1 =|0.8-0.6|=0.2; if =0.7 (indicating a relatively regular shape), then E 2 =|0.7-0.6|=0.1.

[0115] In addition to this simple difference calculation, weighted difference calculation can also be performed based on the importance of the feature. For example, if the importance weight of the edge feature in shape evaluation is determined to be = 0.6, the weight of the shape feature is =0.4, then the weighted feature error According to the previous data, we can calculate that E=0.6×0.2+0.4×0.1=0.12+0.04=0.16.

[0116] Next is step S400B, where the defect assessment parameters are different parameters used to comprehensively assess the defect status of chip packaging components, such as defect distribution evolution parameters, evolution number parameters, evolution degree parameters, evolution speed parameters and defect qualitative parameters.

[0117] Taking the defect distribution evolution parameter as an example, the computer system can determine the implicit representation of the defect distribution evolution corresponding to the defect distribution evolution parameter of the component to be identified under the current defect assessment channel based on the feature error. Assume that in the chip packaging image, the distribution of the image features of interest with larger feature errors is more dispersed in the image, while the distribution of features with smaller feature errors is more concentrated. The computer system can represent the implicit representation of the defect distribution evolution by calculating the distribution variance of the feature error. Assume that there are n image features of interest, and the feature error of the i-th feature is The average value of the characteristic error is , then the defect distribution evolution is implicitly expressed as For example, if there are three image features of interest, their feature errors are ,but .

[0118] For the number of evolution parameters, the computer system can determine the number of changes from the reference defect to other defect states (or vice versa) based on the characteristic error. For example, if the characteristic error exceeds a certain set threshold (such as 0.3), it is considered that an evolution has occurred. Assuming that in a series of tests, there are characteristic error data obtained from multiple tests, the computer system counts the number of times the characteristic error exceeds the threshold as an implicit representation of the number of evolutions. If there are 3 characteristic errors exceeding 0.3 in 10 tests, the number of evolutions is implicitly represented as 3.

[0119] For the evolution degree parameter, the computer system can determine the evolution degree implicit representation based on the size of the feature error. The larger the feature error, the higher the evolution degree. For example, the maximum value of the feature error can be directly used as the evolution degree implicit representation. 1 =0.2, E 2 =0.5, E 3 =0.3, E 2 The maximum, then the evolution degree is implicitly expressed as 0.5. Or a normalized method is adopted, and the maximum possible feature error is set to (determined by experience or data), then the degree of evolution is implicitly expressed ,in It is the maximum characteristic error in actual detection.

[0120] For the evolution speed parameter, the computer system can determine it based on the change of characteristic error at different detection moments. The characteristic error E is obtained by detecting 1 , E 2 , E 3 The evolution rate can be expressed by calculating the ratio of the difference between the characteristic errors at adjacent moments to the time interval. For example, in the time interval from t_{1} to t_{2} Internal, evolution speed ; at t 2 to 3 Time interval Internal, evolution speed For the implicit representation of the trigger speed (evolution from the reference defect to other defects), the forward evolution speed (i.e., V i >0); for implicit representation of callback speed (evolution from other defects to reference defects), it can be determined according to the negative evolution speed (i.e. V i <0) to determine.

[0121] For defect qualitative parameters, the computer system can determine the defect qualitative implicit representation by comparing the characteristic error with the characteristic error range corresponding to different types of defects. For example, assuming that the characteristic error range corresponding to a certain type of defect (such as a scratch defect on the chip surface) is known to be [0.4, 0.6], if the calculated characteristic error falls within this range, the defect qualitative implicit representation is a value related to this type of defect (such as 1, indicating that it is related to the scratch defect); if it is not within this range, it is represented as other values ​​(such as 0, indicating that it is not related to this type of defect).

[0122] As an implementation mode, the defect assessment parameter includes a defect distribution evolution parameter, and the implicit representation of the defect assessment parameter of the component to be identified under the defect distribution evolution parameter includes an implicit representation of the defect distribution evolution. Based on this, step S400B, based on the characteristic error determined under the current defect assessment channel, determines the implicit representation of the defect assessment parameter of the component to be identified under the current defect assessment channel, which corresponds to a plurality of defect assessment parameters deployed in advance, and may include:

[0123] Step S400B1: determining the discreteness in the current defect assessment channel based on the feature error determined in the current defect assessment channel and the number of detected image features of interest;

[0124] Step S400B2: Based on the discreteness under the current defect assessment channel, determine the defect distribution evolution implicit representation corresponding to the defect distribution evolution parameter of the component to be identified under the current defect assessment channel.

[0125] In step S400B1, the feature error is determined in step S400A, which represents the difference between the implicit representation of the image of interest under the current defect assessment channel and the implicit representation of the reference defect under the current defect assessment channel for each feature of the image of interest. A common technical means for determining the discreteness is to measure it by calculating the variance. Of course, in addition to the variance, the computer system can also use other similar statistics to measure the discreteness, such as the mean absolute deviation (MAD).

[0126] Next is step S400B2, where the defect distribution evolution parameter focuses on how the distribution of defects on the component evolves over time or under different conditions. Discreteness is used as a key indicator here, reflecting the degree of dispersion of feature errors between different image features of interest, and thus reflecting the changes in defect distribution.

[0127] Assume that the computer system establishes a mapping relationship between discreteness and the implicit representation of defect distribution evolution based on a large amount of sample data and prior knowledge. For example, when the discreteness D is between 0 and 0.05, the implicit representation of defect distribution evolution is R 1=0.1, indicating that the defect distribution is relatively stable and changes little; when D is between 0.05-0.1, R 2 =0.3, indicating that the defect distribution has evolved to a certain extent; when D is greater than 0.1, R 3 =0.5, indicating that the evolution of defect distribution is more obvious.

[0128] The discreteness calculated above According to this mapping relationship, the computer system determines the defect distribution evolution of the component to be identified (chip body) under the defect evaluation channel related to surface flatness, which corresponds to the defect distribution evolution parameter and is implicitly expressed as R 1 =0.1.

[0129] Taking the pins in chip packaging as an example, in a defect assessment channel related to the pin shape, assuming that there are four image features of interest, their feature errors are calculated as follows: First, calculate the average value of the feature error . According to the variance formula, the dispersion D=0.003125.

[0130] According to the mapping relationship set previously (assuming that when the discreteness D is between 0 and 0.03, the defect distribution evolution is implicitly expressed as R 1 = 0.05), the computer system determines that the defect distribution evolution of the pin under this shape-related defect assessment channel corresponds to the defect distribution evolution parameter implicitly expressed as R 1 =0.05.

[0131] As an implementation mode, the defect assessment parameter includes an evolution number parameter, and the implicit representation of the defect assessment parameter of the component to be identified under the evolution number parameter includes the implicit representation of the evolution number. Based on this, step S400B, based on the characteristic error determined under the current defect assessment channel, determines the implicit representation of the defect assessment parameter of the component to be identified under the current defect assessment channel, which corresponds to a plurality of defect assessment parameters deployed in advance, and further includes:

[0132] Step S400Ba: based on the feature error determined in the current defect assessment channel, a target image feature of interest corresponding to the feature error that meets the preset critical value requirement is selected from each detected image feature of interest;

[0133] Step S400Bb: determining the number of errors generated in the current defect assessment channel based on the number of target image features of interest;

[0134] Step S400Bc: Based on the number of errors generated in the current defect assessment channel, determine an implicit representation of the number of evolutions of the component to be identified corresponding to the number of evolutions parameter in the current defect assessment channel.

[0135] In step S400Ba of the embodiment of the present invention, the computer system selects a target image feature of interest corresponding to a feature error that meets a preset critical value requirement from each detected image feature of interest based on the feature error determined in the current defect assessment channel.

[0136] The feature error is determined in the previous step (such as step S400A), and represents the difference between the implicit representation of the image of interest under the current defect assessment channel and the implicit representation of the reference defect under the current defect assessment channel for each image feature of interest. The preset critical value is a limit value pre-set by the computer system for screening the target image feature of interest. The setting of this value is based on factors such as prior knowledge of chip packaging defects, empirical data, and tolerance for defects of different degrees.

[0137] Taking the chip body in the chip package as an example, assume that in a defect assessment channel related to electrical performance, five image features of interest are detected, namely edge features, texture features, and shape features of two different regions (shape feature 1 and shape feature 2). For these features, the calculated feature errors are E 1 =0.1 (edge ​​feature), E 2 =0.35 (texture feature), E 3 =0.2 (shape feature 1), E 4 =0.4 (shape feature 2), E 5 =0.15 (other relevant features).

[0138] If the preset critical value of the computer system is set to 0.3, then the target image feature of interest that meets the preset critical value requirement (i.e., the feature error is greater than 0.3) is the texture feature (E 2 = 0.35) and shape feature 2 (E 4 =0.4). The computer system selects each image feature of interest by comparing the feature error of each image feature with a preset critical value.

[0139] Next is step S400Bb. Continuing with the example of the chip body, after selecting the two target image features of interest, texture feature and shape feature 2, the number of target image features of interest is 2. This number is regarded as the number of errors generated under the current defect assessment channel. Because each target image feature of interest that meets the preset critical value requirements represents a possible error generation situation, these errors may be caused by different factors in the chip packaging process, such as process fluctuations, material unevenness, etc.

[0140] In some cases, in order to more accurately reflect the actual situation of error generation, the computer system may classify and count the target image features of interest. For example, if there are multiple types of texture features (such as horizontal texture features and vertical texture features), and multiple feature errors in the horizontal texture features exceed the preset critical value, the computer system may regard multiple feature errors of the same type as one or more error generation situations according to the actual situation. This depends on the specific error generation mechanism and the correlation between them. Assume that in the defect evaluation channel related to electrical performance, the feature errors of three sub-features in the horizontal texture feature all exceed the preset critical value, but these three sub-features are caused by problems in the same process link, then the computer system may regard it as a single error generation situation; if these three sub-features are caused by different process links or different reasons, then it may be regarded as three error generation situations.

[0141] Finally, in step S400Bc, the computer system determines the implicit representation of the evolution number of the component to be identified under the current defect assessment channel corresponding to the evolution number parameter based on the number of errors generated under the current defect assessment channel. The evolution number parameter focuses on the number of transitions from one state to another (possibly from non-defective to defective, or from one defect degree to another defect degree) of the chip package component under the current defect assessment channel. The number of errors generated is closely related to the evolution number, and the error generation is often accompanied by a change in state, which can reflect the evolution number.

[0142] One way for a computer system to determine the implicit representation of the number of evolutions is to directly use the number of errors as the implicit representation of the number of evolutions. Taking the previous chip body under the defect evaluation channel related to electrical performance as an example, the number of errors is 2, so the implicit representation of the number of evolutions is 2. This means that under this defect evaluation channel, the chip body may have experienced two state evolutions, for example, the electrical performance may have changed from a better state to a worse state twice, and these two evolutions may be caused by different reasons (such as one related to texture features and one related to shape feature 2).

[0143] Another way is to determine the implicit representation of the number of evolutions based on the number of error occurrences and some pre-set standards or mapping relationships. Assume that based on a large amount of chip packaging sample data and research on the evolution of electrical performance defects, the computer system sets a mapping relationship: when the number of error occurrences is 0, the number of evolutions is implicitly represented as 0; when the number of error occurrences is 1, the number of evolutions is implicitly represented as 0.3; when the number of error occurrences is 2, the number of evolutions is implicitly represented as 0.6; when the number of error occurrences is greater than 2, the number of evolutions is implicitly represented as 1. Then for the previous example of the chip body, the number of error occurrences is 2, and according to this mapping relationship, the number of evolutions is implicitly represented as 0.6. The setting of this mapping relationship is based on statistical analysis and empirical summary of the relationship between different error occurrences and the actual chip packaging component performance evolution.

[0144] Taking the pins in chip packaging as an example, in a defect assessment channel related to mechanical connection stability, it is assumed that there are six image features of interest, namely three sub-features of the pin edge (edge ​​feature 1, edge feature 2, edge feature 3) and three shape-related features of the pin body (shape feature 1, shape feature 2, shape feature 3). The feature errors are calculated to be E 1 =0.25 (edge ​​feature 1), E 2 =0.1 (edge ​​feature 2), E 3 =0.32 (edge ​​feature 3), E 4 =0.08 (shape feature 1), E 5 =0.35 (shape feature 2), E 6 =0.12 (shape feature 3).

[0145] If the preset threshold is set to 0.3, then the target image feature of interest that meets the requirements is edge feature 3 (E 3 =0.32) and shape feature 2 (E 5 =0.35), the number of target image features of interest is 2, that is, the number of errors generated is 2.

[0146] If the method of directly using the number of error occurrences as the implicit representation of the number of evolutions is used, the number of evolutions corresponding to the number of evolutions parameter of the pin under this defect assessment channel related to mechanical connection stability is implicitly represented as 2. If another method of setting the mapping relationship is used (assuming that when the number of error occurrences is 0, the number of evolutions is implicitly represented as 0; when the number of error occurrences is 1, the number of evolutions is implicitly represented as 0.4; when the number of error occurrences is 2, the number of evolutions is implicitly represented as 0.8; when the number of error occurrences is greater than 2, the number of evolutions is implicitly represented as 1), the number of evolutions is implicitly represented as 0.8.

[0147] Through steps S400Ba-S400Bc, the computer system selects the target image features of interest based on the feature error, determines the number of times the error occurs, and then determines the implicit representation of the evolution number, and evaluates the defect status of the chip packaging components in the current defect assessment channel from the perspective of the evolution number. This helps to have a more comprehensive understanding of the quality evolution of the chip packaging components and provides an important basis for chip packaging defect identification.

[0148] As an implementation mode, the defect assessment parameter includes an evolution degree parameter, and the implicit representation of the defect assessment parameter of the component to be identified under the evolution degree parameter includes the implicit representation of the evolution degree. Based on this, step S400B, based on the characteristic error determined under the current defect assessment channel, determines the implicit representation of the defect assessment parameter of the component to be identified under the current defect assessment channel, which corresponds to a plurality of defect assessment parameters deployed in advance, and further includes:

[0149] Step S400B′: based on the norm of the characteristic error determined in the current defect assessment channel, determine the implicit representation of the evolution degree of the component to be identified corresponding to the evolution degree parameter in the current defect assessment channel.

[0150] The feature error is determined in the previous step (such as step S400A), which represents the difference between the implicit representation of the image of interest under the current defect assessment channel and the implicit representation of the reference defect under the current defect assessment channel for each image feature of interest. The norm is a measure of the length of a vector (here the feature error can be regarded as a vector), which is important in measuring the overall size of the feature error.

[0151] Taking the chip body in the chip package as an example, assume that in a defect assessment channel related to the connectivity of the internal circuit of the chip, there are three image features of interest: the texture feature of the circuit, the shape feature of the circuit, and the feature of the connection between the circuit and the pin. The feature errors calculated for each feature are E_1=(0.1, 0.2), E_2=(0.3, 0.1), and E_3=(0.2, 0.3) (here it is assumed that the feature error is a two-dimensional vector, for example, the first dimension may be related to the length deviation of the circuit, and the second dimension may be related to the angle deviation of the circuit).

[0152] The computer system calculates the norm of the characteristic error to determine the degree of evolution implicit representation. , the common norm calculation method is the Euclidean norm (L_2 norm), and its formula is For E 1 =(0.1, 0.2), its Euclidean norm is ;for , ;for , .

[0153] The computer system can determine the implicit representation of the degree of evolution based on the values ​​of these norms. One possible way is to directly use the maximum norm value as the implicit representation of the degree of evolution. In the above example, the maximum norm value is , then this value can be used as an implicit representation of the evolution degree of the chip body corresponding to the evolution degree parameter under the defect assessment channel related to the internal circuit connectivity of the chip. This means that under this defect assessment channel, the defect evolution degree of the chip body is relatively high because the norm of its characteristic error is large, reflecting that there may be obvious state changes in the internal circuit connection.

[0154] Another way is to determine the implicit representation of the evolution degree by mapping the norm value to a pre-set range. Assume that based on a large amount of chip packaging sample data and research on the evolution of internal line connectivity defects, the computer system sets a mapping relationship: when the norm value is between 0-0.2, the evolution degree is implicitly represented as 0.1; when the norm value is between 0.2-0.4, the evolution degree is implicitly represented as 0.3; when the norm value is greater than 0.4, the evolution degree is implicitly represented as 0.5. For the norm value calculated above, ,According to this mapping relationship, the evolution degree of the chip body corresponding to the evolution degree parameter under this defect assessment channel is implicitly expressed as 0.3.

[0155] Taking the pins in chip packaging as an example, in a defect assessment channel related to the mechanical strength of the pins, it is assumed that there are two image features of interest: the shape feature of the pin and the texture feature of the pin surface. The calculated feature errors are . Compute their Euclidean norms, , .

[0156] If the method of directly taking the maximum norm value is adopted, the degree of evolution is implicitly expressed as If we follow the set mapping relationship (assuming that when the norm value is between 0-0.3, the degree of evolution is implicitly expressed as 0.2; when the norm value is between 0.3-0.5, the degree of evolution is implicitly expressed as 0.4; when the norm value is greater than 0.5, the degree of evolution is implicitly expressed as 0.6), then according to , the degree of evolution is implicitly expressed as 0.2.

[0157] As an implementation method, the defect assessment parameter includes an evolution speed parameter, and the implicit representation of the defect assessment parameter of the component to be identified under the evolution speed parameter includes an implicit representation of the trigger speed and an implicit representation of the callback speed. Based on this, step S400B, based on the characteristic error determined under the current defect assessment channel, determines the implicit representation of the defect assessment parameter of the component to be identified under the current defect assessment channel, which corresponds to a plurality of defect assessment parameters deployed in advance, and further includes:

[0158] Step S400B01: determining a defect error mode of each detected image feature of interest based on the error property of the feature error determined in the current defect assessment channel; the defect error mode includes a trigger mode and a callback mode;

[0159] Step S400B02: determining a first number of the image feature of interest in the trigger mode, and determining a trigger speed implicit representation corresponding to the evolution speed parameter of the component to be identified in the current defect assessment channel based on the first number and the evolution degree implicit representation in the current defect assessment channel;

[0160] Step S400B03: Determine the second number of the image feature of interest in the callback mode, and determine the implicit representation of the callback speed of the component to be identified corresponding to the evolution speed parameter under the current defect assessment channel based on the second number and the implicit representation of the evolution degree under the current defect assessment channel.

[0161] In step S400B01 of the embodiment of the present invention, the computer system determines the defect error mode of each detected image feature of interest based on the error property of the characteristic error determined in the current defect assessment channel (for example, whether it is a positive error or a negative error); the defect error mode includes a trigger mode (that is, evolving from a reference defect to other defects) and a callback mode (that is, evolving from other defects to a reference defect).

[0162] The feature error is calculated in the previous step and represents the difference between the image feature of interest in the current defect assessment channel and the implicit representation of the reference defect. The positive or negative nature of the error is a key factor in determining the direction of defect evolution. For example, in the chip body in the chip package, for a defect assessment channel related to the chip surface flatness, it is assumed that there are three image features of interest: texture features, edge features, and shape features. The feature error of the texture feature is calculated as E 1 = 0.2 (assuming that the reference defect is implicitly represented as 0, 0.2 here means that it deviates from the reference state by 0.2, which is a positive error). The characteristic error of the edge feature is E 2 = -0.1 (this is a negative error), the characteristic error of the shape feature is E 3 =0.3 (positive error).

[0163] For positive errors (such as E 1 and E 3 ), the computer system determines that it is in trigger mode, which means that the chip surface flatness corresponding to these features may evolve from a good reference state to a defective state. 2 ), the computer system determines that it is in callback mode, that is, it may be evolving from a defective state towards a reference better state.

[0164] A technical means to determine the nature of the error is to set a reference base value (such as the reference defect assumed above is implicitly represented as 0). When the characteristic error is greater than this base value, it is judged as a positive error, and when it is less than this value, it is judged as a negative error.

[0165] Next is step S400B02, where the computer system determines the first number of the image feature of interest in the trigger mode, and based on the first number and the implicit representation of the degree of evolution under the current defect assessment channel, determines the implicit representation of the trigger speed corresponding to the evolution speed parameter of the component to be identified under the current defect assessment channel.

[0166] Continuing with the example of the surface flatness of the chip body, the image features of interest in the trigger mode are texture features and shape features, and the first number is 2. The evolution degree implicit representation is calculated in the previous step, reflecting the degree of defect evolution of the chip package component under the current defect assessment channel. Assume that in this example, the evolution degree of the chip body under the defect assessment channel related to surface flatness is implicitly represented as D=0.4.

[0167] One way for the computer system to determine the trigger speed implicit representation may be through a simple proportional relationship. Assuming that without the influence of other factors, the trigger speed implicit representation V t The number of interesting image features n in the trigger mode t is positively correlated with the implicit representation of the evolution degree D. For example, we can set V t =k t ×n t ×D, where k t is a proportionality coefficient determined based on experience or experimental data. Assume that k t =0.1, then V t =0.1×2×0.4=0.08. This 0.08 is used as an implicit representation of the trigger speed of the chip body corresponding to the evolution speed parameter in the current defect assessment channel, indicating that the evolution speed from the reference defect to other defects is relatively fast (here is a relative concept based on the set calculation relationship).

[0168] Taking the pins in the chip package as an example, in a defect assessment channel related to the electrical connectivity of the pins, assuming that there are four image features of interest, their feature errors are calculated to be E 1 =0.3 (forward error, trigger mode), E 2 =-0.1 (negative error, callback mode), E 3 =0.2 (forward error, trigger mode), E 4 = 0.1 (positive error, trigger mode). The first number n of the image features of interest located in the trigger mode t =3, assuming that the evolution degree in this defect assessment channel is implicitly expressed as D=0.3, and k t = 0.1, the trigger speed implicitly represents V t =0.1×3×0.3=0.09.

[0169] Finally, in step S400B03, the computer system determines the second number of the image feature of interest in the callback mode, and based on the second number and the implicit representation of the degree of evolution under the current defect assessment channel, determines the implicit representation of the callback speed of the component to be identified corresponding to the evolution speed parameter under the current defect assessment channel.

[0170] Back to the example of the flatness of the chip surface, the only interesting image features in the callback mode are the edge features, and the second number is 1. Assume that the callback speed implicitly represents V c The number of interesting image features n in callback mode c There is a similar relationship between the implicit expression D and the evolution degree. Let V c =k c ×n c ×D, where k c It is also a proportionality coefficient determined based on experience or experimental data. Assuming k c = 0.05 (because the callback speed may be different from the trigger speed under the same conditions, so the coefficient is different), then V c =0.05×1×0.4=0.02. This 0.02 is the implicit representation of the callback speed of the chip body corresponding to the evolution speed parameter under the current defect assessment channel, indicating that the speed of returning from other defect states to the reference state is relatively slow.

[0171] For the pin example, the number of interesting image features n in the callback mode under the defect assessment channel related to the pin electrical connectivity c =1, the evolution degree implicitly represents D=0.3, and k c =0.05, then the callback speed implicitly represents V c =0.05×1×0.3=0.015.

[0172] As another implementation, the defect assessment parameters include defect qualitative parameters (which can help understand the main defect tendency in the image), and the defect assessment parameter implicit representation of the component to be identified under the defect qualitative parameters includes the defect qualitative implicit representation. Based on this, step S400, based on the reference defect implicit representation of the component to be identified under each defect assessment channel in multiple defect assessment channels, determines the defect assessment parameter implicit representations corresponding to the component to be identified under multiple defect assessment parameters deployed in advance, including:

[0173] Step S4001: obtaining multi-channel defect qualitative implicit representations corresponding to each of the plurality of defect qualitative image blocks deployed in advance;

[0174] Step S4002: for each defect qualitative image block in the multiple defect qualitative image blocks, determine the defect similarity between the component to be identified and the targeted defect qualitative image block according to the multi-channel defect qualitative implicit representation of the targeted defect qualitative image block and the reference defect implicit representation of the component to be identified in each defect assessment channel in the multiple defect assessment channels;

[0175] Step S4003: based on the defect similarity between the component to be identified and each defect qualitative image block in the multiple defect qualitative image blocks, a target defect qualitative image block is selected from the multiple defect qualitative image blocks;

[0176] Step S4004: Based on the target defect qualitative image block, determine the defect qualitative implicit representation of the component to be identified under the defect qualitative parameters.

[0177] In step S4001 of the embodiment of the present invention, the computer system obtains the multi-channel defect qualitative implicit representation corresponding to each of the multiple defect qualitative image blocks deployed in advance. The defect qualitative image blocks are pre-defined image blocks with specific defect characteristics or tendencies. The definition of these image blocks is based on an in-depth understanding of various defect types that may occur in chip packaging and a large number of sample analyses. For example, in chip packaging, there may be surface unevenness defects caused by uneven packaging materials, and then there will be one or more defect qualitative image blocks corresponding to this surface unevenness defect.

[0178] Multi-channel defect qualitative implicit representation means that for each defect qualitative image block, there are corresponding representations under multiple different defect assessment channels. Assume that there are three defect assessment channels: structural integrity channel, electrical performance channel, and surface characteristic channel. For a defect qualitative image block corresponding to a scratch defect on the chip surface, its multi-channel defect qualitative implicit representation under the structural integrity channel may be related to the potential impact of the scratch on the chip structure, for example, expressed as ; In the electrical performance channel, it may be related to the interference of scratches on circuit connections, recorded as ; In the surface property channel, it may be related to the change in surface roughness caused by scratches, recorded as For another defect qualitative image block corresponding to the chip pin bending defect, the multi-channel defect qualitative implicit representation under three channels is .

[0179] The computer system may obtain these multi-channel defect qualitative implicit representations by storing them based on the results of previous experiments and analysis. These results may come from feature extraction and analysis of a large number of chip packaging samples with known defects, and quantitative representation according to different defect evaluation channels, thereby obtaining the multi-channel defect qualitative implicit representation of each defect qualitative image block under each channel.

[0180] Next is step S4002, where the chip body in the chip package is taken as the component to be identified. Assuming that the reference defect of the chip body in the structural integrity channel obtained in the previous step is implicitly represented as R 1 , R under the electrical performance channel 2 , R under the surface characteristic channel 3 For the defect qualitative image block corresponding to the surface scratch defect mentioned above (its multi-channel defect qualitative implicit representation is ), one way for a computer system to determine defect similarity is to express the similarity by calculating the inverse of the Euclidean distance. Let the defect similarity be , the calculation formula is For example, if ; ,but .

[0181] For the defect qualitative image block corresponding to the pin bending defect (the multi-channel defect qualitative implicit representation is ), the defect similarity with the chip body is calculated according to the above formula .

[0182] In addition to the inverse of the Euclidean distance, the computer system can also use other methods to determine the defect similarity. For example, based on cosine similarity. Suppose the reference defect implicit representation of the part to be identified under three channels consists of vectors , the multi-channel defect qualitative implicit representation of the defect qualitative image patch constitutes the vector , then the cosine similarity ,in , .

[0183] Then, step S4003 is performed. Continuing with the above example, it is assumed that the defect similarities between the chip body and the multiple defect qualitative image blocks calculated by the computer system are (Here, it is assumed that there are three defect qualitative image blocks) etc. The computer system can select the defect qualitative image block with the greatest defect similarity as the target defect qualitative image block. In this example, the defect qualitative image block corresponding to the surface scratch defect (whose similarity ) is selected as the target defect qualitative image patch.

[0184] Another method of selecting the target defect qualitative image block is to set a threshold. For example, if the defect similarity is greater than a certain threshold (such as 1.5), the corresponding defect qualitative image block is used as the target defect qualitative image block. In the above example, >1.5, so the defect qualitative image block corresponding to the surface scratch defect is selected.

[0185] Finally, in step S4004, once the target defect qualitative image block is determined (such as the defect qualitative image block corresponding to the surface scratch defect mentioned above), the computer system can determine the defect qualitative implicit representation of the component to be identified under the defect qualitative parameters based on the specific attributes of the target defect qualitative image block.

[0186] One way is to directly use the multi-channel defect qualitative implicit representation of the target defect qualitative image block under a specific channel as the defect qualitative implicit representation of the part to be identified. For example, after selecting the defect qualitative image block corresponding to the surface scratch defect, its multi-channel defect qualitative implicit representation under the surface characteristic channel is used as the defect qualitative implicit representation of the part to be identified. (Assumption =0.45) as the defect qualitative implicit representation of the chip body under the defect qualitative parameters. This means that the chip body shows a similar degree of characteristics in terms of defect qualitative characteristics as the surface scratch defect qualitative image block in terms of surface characteristics.

[0187] Another way is to adjust the defect qualitative implicit representation based on the relationship between the target defect qualitative image block and the part to be identified. For example, based on the defect similarity between the target defect qualitative image block and the part to be identified Perform weighted adjustment. Let D be the defect qualitative implicit representation of the component to be identified under the defect qualitative parameters, .

[0188] Let’s take the pins in the chip package as an example of the components to be identified. Assume that there are defect qualitative image blocks corresponding to defects such as pin breakage and pin oxidation. The reference defect of the pin in the structural integrity channel is implicitly represented as , the electrical performance channel is , the surface characteristic channel is The defect qualitative image block corresponding to the pin fracture defect is implicitly expressed in the multi-channel defect qualitative representation under three channels as follows: , corresponding to the pin oxidation defect is .

[0189] The computer system calculates the defect similarity with the pin fracture defect qualitative image block according to the method mentioned above and defect similarity with the pin oxidation defect qualitative image block Assumptions ,According to the selection rules (such as selecting the one with the greatest similarity), the defect qualitative image block corresponding to the pin fracture defect is selected as the target defect qualitative image block.

[0190] If the method of directly using the representation of the target defect qualitative image block under a certain channel as the qualitative implicit representation of the defect of the part to be identified is adopted, assuming that the target defect qualitative image block under the structural integrity channel is selected (Assumption =0.35) as the implicit representation of the defect quality of the pin under the defect quality parameters. If the weighted adjustment method is adopted, let D' be the implicit representation of the defect quality of the pin under the defect quality parameters, .

[0191] Through steps S4001-S4004, the computer system determines the defect qualitative implicit representation of the component to be identified under the defect qualitative parameters from the defect qualitative perspective, based on the relationship between the reference defect implicit representation of the component to be identified under different defect assessment channels and the multi-channel defect qualitative implicit representation of the defect qualitative image block deployed in advance. This helps to more deeply understand the defect type tendency of the component to be identified, provides an important qualitative analysis basis for chip packaging defect identification, and thus more comprehensively evaluates the quality status of chip packaging components.

[0192] An embodiment of the present invention provides a computer system, such as Figure 2 As shown, the computer system 100 includes: a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, such as through a bus 102. Optionally, the computer system 100 may also include a transceiver 104. It should be noted that in actual applications, the transceiver 104 is not limited to one, and the structure of the computer system 100 does not constitute a limitation on the embodiments of the present invention.

[0193] An embodiment of the present invention provides a computer system. The computer system in the embodiment of the present invention includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors. When the one or more programs are executed by the processor, the above-mentioned chip packaging defect recognition method based on machine vision is implemented.

Claims

1. A chip packaging defect recognition method based on machine vision, characterized in that: The method comprises: Determine a component to be identified in a chip package image, and extract a relevant image region corresponding to the component to be identified in the chip package image; Detecting an image feature of interest in each extracted relevant image region, and determining an implicit image representation of interest for each detected image feature of interest in each defect assessment channel of a plurality of preset defect assessment channels; For each defect assessment channel of the plurality of defect assessment channels, determining a reference defect implicit representation of the component to be identified under the targeted defect assessment channel based on an implicit representation of the image of interest of each of the image features of interest under the targeted defect assessment channel; Determining, based on the reference defect implicit representation of the component to be identified under each of the multiple defect assessment channels, the defect assessment parameter implicit representations corresponding to the component to be identified under multiple defect assessment parameters deployed in advance; Based on the implicit representation of the defect assessment parameters corresponding to the component to be identified under the multiple defect assessment parameters, a defect identification result of the component to be identified corresponding to the component to be identified is obtained.

2. The method according to claim 1, characterized in that The image features of interest are detected in each extracted relevant image region, including: Determine each relevant image region as a target relevant image region; Performing an image block operation on the target-related image region to obtain one or more image blocks included in the target-related image region; For each of the one or more image blocks, a preset image feature of interest associated with the targeted image block is determined in a pre-deployed set of image features of interest, and the preset image feature of interest associated with the targeted image block is determined as the image feature of interest detected in the target-related image area.

3. The method according to claim 2, characterized in that The determining of a preset image feature of interest associated with the targeted image block from a pre-deployed image feature set of interest includes: Determine a feature Euclidean distance between the targeted image block and a preset image feature of interest in a pre-deployed image feature set of interest; If the set of image features of interest includes a target preset image feature of interest whose feature Euclidean distance meets the first requirement, the target preset image feature of interest is determined as the preset image feature of interest associated with the targeted image block.

4. The method according to claim 1, characterized in that: The step of determining an implicit image representation of each detected image feature of interest in each defect assessment channel of a plurality of preset defect assessment channels comprises: For each of the detected image features of interest, obtaining a preset image of interest implicit representation of a preset image of interest associated with the targeted image feature of interest in each defect assessment channel of a plurality of defect assessment channels deployed in advance; Determine, according to the obtained preset implicit representation of the image of interest, an implicit representation of the image of interest for the targeted feature of the image of interest in each defect assessment channel of the multiple defect assessment channels; The step of determining, for each defect assessment channel among the plurality of defect assessment channels, a reference defect implicit representation of the component to be identified under the targeted defect assessment channel based on an implicit representation of the image of interest of each of the image features of interest under the targeted defect assessment channel, comprises: For each defect assessment channel of the plurality of defect assessment channels, based on an implicit representation of the image of interest of each of the image features of interest in the targeted defect assessment channel, determining a mean value of an implicit representation of a defect of the component to be identified in the targeted defect assessment channel; According to the mean value of the defect implicit representation of the component to be identified in the targeted defect assessment channel, a reference defect implicit representation of the component to be identified in the targeted defect assessment channel is determined.

5. The method according to claim 1, characterized in that: The defect assessment parameters include reference defect assessment parameters, and the implicit representation of the defect assessment parameters of the component to be identified under the reference defect assessment parameters includes a reference defect assessment implicit representation; and determining the implicit representation of the defect assessment parameters corresponding to the component to be identified under the multiple defect assessment parameters deployed in advance based on the reference defect implicit representation of the component to be identified under each defect assessment channel in the multiple defect assessment channels, comprises: For each defect assessment channel in the multiple defect assessment channels, a reference defect assessment implicit representation of the component to be identified under the reference defect assessment parameters is determined based on the reference defect implicit representation of the component to be identified under each defect assessment channel in the multiple defect assessment channels and a pre-deployed probability range.

6. The method according to claim 1, characterized in that The step of determining the implicit representations of defect assessment parameters corresponding to the component to be identified under a plurality of defect assessment parameters deployed in advance based on the implicit representation of the reference defect of the component to be identified under each of the plurality of defect assessment channels comprises: For each defect assessment channel of the plurality of defect assessment channels, determining a feature error between an implicit representation of the image of interest under a current defect assessment channel and an implicit representation of a reference defect under the current defect assessment channel for each detected feature of the image of interest; Based on the characteristic error determined in the current defect assessment channel, an implicit representation of defect assessment parameters of the to-be-identified component corresponding to a plurality of defect assessment parameters deployed in advance in the current defect assessment channel is determined.

7. The method according to claim 6, characterized in that The defect assessment parameters include defect distribution evolution parameters, and the implicit representation of the defect assessment parameters of the to-be-identified component under the defect distribution evolution parameters includes an implicit representation of defect distribution evolution; The step of determining, based on the characteristic error determined in the current defect assessment channel, the implicit representation of defect assessment parameters of the to-be-identified component corresponding to the plurality of defect assessment parameters deployed in advance in the current defect assessment channel, comprises: Determine the discreteness under the current defect assessment channel based on the feature error determined under the current defect assessment channel and the number of the detected image features of interest; Determine, based on the discreteness under the current defect assessment channel, an implicit representation of defect distribution evolution of the component to be identified corresponding to the defect distribution evolution parameter under the current defect assessment channel; or; The defect assessment parameter includes an evolution number parameter, and the implicit representation of the defect assessment parameter of the component to be identified under the evolution number parameter includes an implicit representation of the evolution number; The step of determining, based on the characteristic error determined in the current defect assessment channel, the implicit representation of defect assessment parameters of the to-be-identified component corresponding to the plurality of defect assessment parameters deployed in advance in the current defect assessment channel, further includes: Based on the feature error determined in the current defect assessment channel, a target image feature of interest corresponding to the feature error that meets a preset critical value requirement is selected from each of the detected image features of interest; Determining the number of error occurrences in the current defect assessment channel based on the number of the target image features of interest; Based on the number of errors generated in the current defect assessment channel, determining an implicit representation of the number of evolutions of the component to be identified corresponding to the number of evolutions parameter in the current defect assessment channel; or; The defect assessment parameter includes an evolution degree parameter, and the implicit representation of the defect assessment parameter of the to-be-identified component under the evolution degree parameter includes an implicit representation of the evolution degree; The step of determining, based on the characteristic error determined in the current defect assessment channel, the implicit representation of defect assessment parameters of the to-be-identified component corresponding to the plurality of defect assessment parameters deployed in advance in the current defect assessment channel, further includes: Based on the norm of the characteristic error determined under the current defect assessment channel, an implicit representation of the evolution degree of the to-be-identified component corresponding to the evolution degree parameter under the current defect assessment channel is determined.

8. The method according to claim 7, characterized in that The defect assessment parameters include an evolution speed parameter, and the implicit representation of the defect assessment parameters of the to-be-identified component under the evolution speed parameter includes an implicit representation of a trigger speed and an implicit representation of a callback speed; The step of determining, based on the characteristic error determined in the current defect assessment channel, the implicit representation of defect assessment parameters of the to-be-identified component corresponding to the plurality of defect assessment parameters deployed in advance in the current defect assessment channel, further includes: Determine a defect error mode of each of the detected image features of interest based on the error properties of the feature errors determined in the current defect assessment channel; the defect error mode includes a trigger mode and a callback mode; Determine a first number of the image feature of interest in the trigger mode, and determine an implicit representation of the trigger speed of the component to be identified corresponding to the evolution speed parameter under the current defect assessment channel based on the first number and the implicit representation of the evolution degree under the current defect assessment channel; A second number of the image feature of interest in the callback mode is determined, and based on the second number and the implicit representation of the evolution degree under the current defect assessment channel, an implicit representation of the callback speed of the component to be identified corresponding to the evolution speed parameter under the current defect assessment channel is determined.

9. The method according to claim 1, characterized in that: The defect assessment parameters include defect qualitative parameters, and the defect assessment parameter implicit representation of the to-be-identified component under the defect qualitative parameters includes defect qualitative implicit representation; The step of determining the implicit representations of defect assessment parameters corresponding to the component to be identified under a plurality of defect assessment parameters deployed in advance based on the implicit representation of the reference defect of the component to be identified under each of the plurality of defect assessment channels comprises: Obtaining multi-channel defect qualitative implicit representations corresponding to multiple defect qualitative image blocks deployed in advance; For each defect qualitative image block in the multiple defect qualitative image blocks, determine the defect similarity between the component to be identified and the targeted defect qualitative image block according to the multi-channel defect qualitative implicit representation of the targeted defect qualitative image block and the reference defect implicit representation of the component to be identified under each defect assessment channel in the multiple defect assessment channels; Based on the defect similarity between the component to be identified and each of the plurality of defect qualitative image blocks, a target defect qualitative image block is selected from the plurality of defect qualitative image blocks; Based on the target defect qualitative image block, a defect qualitative implicit representation of the to-be-identified component under defect qualitative parameters is determined.

10. A computer system, characterized in that: include: one or more processors; Memory; one or more computer programs; The one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method according to any one of claims 1 to 9 is implemented.

Images