Chip wire bonding reliability detection method and system based on multi-source test fusion

Abstract

The present invention provides a chip wire bonding reliability detection method and system based on multi-source test fusion, which obtains chip multi-source test images, loads them into a target wire bonding reliability detection network for image recognition, and obtains reliability detection results. The target wire bonding reliability detection network is obtained by calibrating a preset deep neural network based on a first training error and a second training error. The first training error is one or more of an image descriptor training error and a defect reasoning training error. The image descriptor training error is used to indicate the error between image semantic descriptors corresponding to the training image descriptors for the same defect type area, and the defect reasoning training error is used to indicate the error between wire bonding defect reasoning results corresponding to the same defect type area in the training image defect reasoning results. Based on this application, the network's recognition uniformity of the same defect type area can be increased, and the universality and robustness of defect fusion recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a chip wire bonding reliability detection method and system based on multi-source test fusion. Background Art

[0002] In today's highly developed electronics industry, chips are the core components of various computer systems. The chip manufacturing process is a highly precise and complex project, in which the wire bonding process is crucial to ensure the normal function and overall performance of the chip.

[0003] During the chip wire bonding process, there are many types of defects that may occur, such as wire breakage, cold solder joints, irregular solder ball shapes, and bond position offsets. Moreover, these defects do not exist in isolation. In actual chips, they may appear in the same area or different areas in a variety of different combinations. This complex combination of defects greatly increases the difficulty of detection.

[0004] Traditional chip wire bonding reliability detection methods have obvious shortcomings when dealing with such complex defect combinations. For example, some rule-based or template matching methods may be effective in identifying specific, single defect types, but when faced with multiple defect combinations, they often cannot accurately judge the overall reliability of the chip due to the lack of understanding and adaptability of the relationship between different defects.

[0005] With the development of artificial intelligence technology, neural networks have been introduced into the field of chip wire bonding reliability detection. However, existing neural network models face the problem of insufficient generalization ability when dealing with complex defect combinations in chip wire bonding. When neural networks are trained to identify specific, limited defect types, in actual production environments, the performance of the model will be significantly reduced when faced with defect combinations that are not fully covered in the training set. This is because the model lacks the ability to uniformly identify areas of the same defect type, as well as the universality and robustness to accurately identify different defect combination scenarios. Summary of the invention

[0006] The purpose of the present invention is to provide a chip wire bonding reliability detection method and system based on multi-source test fusion, and the present application is implemented as follows:

[0007] In a first aspect, the present application provides a chip wire bonding reliability detection method based on multi-source test fusion, the method comprising: acquiring a chip multi-source test image; loading the chip multi-source test image into a target wire bonding reliability detection network for image recognition, and obtaining a reliability detection result corresponding to the chip multi-source test image; wherein the target wire bonding reliability detection network is obtained by calibrating a preset deep neural network based on a first training error and a second training error, the first training error being one or more of an image descriptor training error and a defect inference training error, the image descriptor training error being used to indicate the same chip in a multi-source test training image set. The error between the image semantic descriptors corresponding to the defect type area in the training image descriptor, the defect reasoning training error is used to indicate the error between the wire bonding defect reasoning results corresponding to the same defect type area in the chip multi-source test training image set in the training image defect reasoning result, the training image descriptor and the training image defect reasoning result are obtained by loading the chip multi-source test training image set into the preset deep neural network for image recognition, and the second training error is determined based on the training image defect reasoning result and the prior defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set.

[0008] In a second aspect, the present application provides a computer system comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method as described above is implemented.

[0009] Beneficial effects of the present invention: The present application obtains a chip multi-source test image, loads the chip multi-source test image into a target wire bonding reliability detection network for image recognition, and obtains a reliability detection result corresponding to the chip multi-source test image, wherein the target wire bonding reliability detection network is obtained by calibrating a preset deep neural network based on a first training error and a second training error, the first training error is one or more of an image descriptor training error and a defect reasoning training error, the image descriptor training error is used to indicate the error between image semantic descriptors corresponding to the same defect type area in a chip multi-source test training image set in a training image descriptor, and the defect reasoning training error is used to indicate the wire bonding corresponding to the same defect type area in a chip multi-source test training image set in a training image defect reasoning result. The error between the defect reasoning results, the training image descriptor and the training image defect reasoning result are obtained by loading the chip multi-source test training image set into a preset deep neural network for image recognition. The second training error is determined based on the training image defect reasoning result and the prior defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set. Based on the adjustment based on the first training error to obtain the target wire bonding reliability detection network, the recognition uniformity for the same defect type area can be improved, the image recognition reliability of the target wire bonding reliability detection network can be increased, and the universality and robustness of the defect fusion recognition of the wire bonding reliability detection network can be improved. Furthermore, there is no need to change the network structure of the wire bonding reliability detection network, which can prevent the waste of computing power resources caused by this operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of a chip wire bonding reliability detection method based on multi-source test fusion provided in an embodiment of the present application.

[0011] Figure 2 It is a calibration flow chart of a target wire bonding reliability detection network provided in an embodiment of the present application.

[0012] Figure 3 It is a schematic diagram of the composition of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The execution subject of the chip wire bonding reliability detection method based on multi-source test fusion in the embodiment of the present application is a computer system, including but not limited to a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The embodiment of the present application provides a chip wire bonding reliability detection method based on multi-source test fusion, such as Figure 1 As shown, the method includes:

[0014] Step S100: Acquire a chip multi-source test image.

[0015] In the embodiment of the present application, the chip multi-source test image is image information related to chip wire bonding obtained from multiple different sources. Multi-source means that the image sources are diverse, for example, it may include different types of images such as optical microscope images, electron microscope images, and X-ray imaging images.

[0016] From the perspective of optical microscope images, it can provide intuitive visual information of the chip surface. When detecting chip wire bonding, the connection form and direction of the wires on the chip surface can be clearly seen. Optical microscopes use the principle of light reflection to form images, and their resolution depends on the parameters of the optical system. Assuming that the resolution of an optical microscope is d, according to the Rayleigh criterion, its resolution formula is ,in is the wavelength of the illumination light, and NA is the numerical aperture of the objective lens. The computer system can acquire these image data by connecting to an image acquisition device (such as a CCD or CMOS camera) of the optical microscope.

[0017] Electron microscope images provide more microscopic structural information. It uses the interaction between the electron beam and the sample to form an image, and there are types such as scanning electron microscope (SEM) and transmission electron microscope (TEM). For example, a scanning electron microscope can provide high-resolution surface morphology images of the chip lead bonding area, which is very useful for detecting tiny defects or observing the microstructure of the contact interface between the lead and the chip. The computer system also needs to use a special image acquisition interface to obtain electron microscope images, which converts the electrical signals generated by the electron microscope into digital image signals that can be processed by the computer.

[0018] X-ray imaging images are another source. X-rays have strong penetrating power and can penetrate the different layers of the chip structure. When testing the reliability of wire bonding, X-ray imaging can reveal the connection status of internal leads, such as whether there are internal broken wires, cold solder joints and other defects hidden inside the chip. The computer system obtains the image data generated by the X-ray imaging device that reflects the internal structure of the chip by connecting to the data output port. After the computer system obtains these multi-source test images, it performs preliminary sorting and preprocessing on the images. For images from different sources, format conversion may be required to unify them into a format that the computer system can handle, such as common TIFF, JPEG and other formats. At the same time, due to the different imaging parameters of different devices, there may be differences in the brightness and contrast of the images. The computer system also needs to normalize these images to ensure that subsequent analysis and processing can be carried out on the same basis. For example, for the normalization of brightness, a simple linear transformation formula can be used: ,in is the normalized brightness value, I is the original brightness value, are the minimum and maximum brightness in the image, respectively.

[0019] In addition, to facilitate subsequent image recognition and analysis, the computer system can also perform operations such as cropping and rotating the acquired multi-source test images to remove unnecessary background information or adjust the image to a suitable direction. For example, if the optical microscope image contains a large number of irrelevant areas around the chip, the computer system can crop the image according to the pre-set chip area coordinates and only retain the image portion of the chip lead bonding area.

[0020] Step S200: loading the chip multi-source test image into the target wire bonding reliability detection network for image recognition to obtain the reliability detection result corresponding to the chip multi-source test image.

[0021] In an embodiment of the present application, the target wire bonding reliability detection network is a calibrated and optimized network that has the ability to effectively analyze multi-source test images of chips. This network is structurally built based on a deep neural network in deep learning. The deep neural network contains multiple hidden layers and can automatically learn feature representations in images. For example, a typical convolutional neural network (CNN) structure includes a convolution layer, a pooling layer, and a fully connected layer. The convolution layer performs a convolution operation by sliding the convolution kernel on the image to extract local features of the image. Assuming that the size of the convolution kernel is k×k (k is a positive integer, such as 3×3), for a certain position (x, y) of the input image I, the output O(x, y) after the convolution operation can be expressed by the formula Calculation, where K is the convolution kernel. The pooling layer is used to reduce the data dimension and reduce the amount of calculation. For example, the maximum pooling layer will select the maximum value in the local area as the output.

[0022] When the computer system loads the multi-source test images of the chip into the target wire bonding reliability detection network, the network will process the images from different sources (such as the optical microscope images, electron microscope images, X-ray imaging images, etc. mentioned earlier) in a unified manner. Taking the optical microscope image as an example, each pixel in the image contains information such as the brightness and color of the wire bonding area on the surface of the chip, which is represented in digital form. After the input layer of the network receives these image data, it gradually extracts features through the hidden layer and analyzes them. For electron microscope images, due to their higher resolution and microscopic structural information, the network can identify more subtle features at the wire bonding, such as atomic-level structural changes or extremely small defects. The internal structural information in the X-ray imaging image can also be effectively analyzed by the network, such as detecting whether there are voids or incomplete connections inside the wire.

[0023] During the image recognition process, the target wire bonding reliability detection network uses the learned patterns and features to determine the reliability of chip wire bonding. This reliability detection result may be a binary classification result (such as reliable or unreliable), or a multi-classification result (such as no defects, minor defects, serious defects, etc.) or the type of defect (cold solder joints, short circuits, breaks, offsets, etc.). The computer system determines the reliability detection result corresponding to the chip multi-source test image based on the results of the network output layer. For example, if the network outputs a probability vector , corresponding to the probability of no defect, slight defect, and severe defect respectively. The computer system can > 0.8 is considered as no defect) to determine the final reliability test result.

[0024] In order to achieve accurate image recognition, the computer system may need to perform some preprocessing operations on the image before loading it into the network to adapt it to the network's input requirements. For example, adjust the image size to unify the chip multi-source test images of different sizes into a fixed size required by the network input layer. If the network input requires an image size of M×N (M and N are positive integers), the computer system can adjust the image size through an interpolation algorithm (such as bilinear interpolation). The bilinear interpolation formula is: For a point (x, y) in the target image, its pixel value , where i and j are the corresponding coordinates in the source image, and dx and dy are the fractional parts in the x and y directions.

[0025] In addition, the computer system needs to consider the type of image data (such as grayscale image or color image) during processing. If it is a color image, it may need to be converted to a format suitable for network processing. For example, to convert an RGB color image to a grayscale image, the conversion formula is (R, G, B are the values ​​of the red, green, and blue channels respectively).

[0026] In an embodiment of the present application, a target wire bonding reliability detection network is obtained by calibrating a preset deep neural network based on a first training error and a second training error. The first training error is one or more of an image descriptor training error and a defect reasoning training error. The image descriptor training error is used to indicate the error between image semantic descriptors corresponding to the same defect type area in a chip multi-source test training image set in a training image descriptor. The defect reasoning training error is used to indicate the error between wire bonding defect reasoning results corresponding to the same defect type area in a chip multi-source test training image set in a training image defect reasoning result. The training image descriptor and the training image defect reasoning result are obtained by loading the chip multi-source test training image set into a preset deep neural network for image recognition. The second training error is determined based on the training image defect reasoning result and the prior defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set.

[0027] Specifically, Figure 2 As shown in FIG. 1 , the calibration process of the target wire bonding reliability detection network includes the following steps:

[0028] Step S10: obtaining a chip multi-source test training image set and a priori defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set;

[0029] Step S20: loading the chip multi-source test training image set into a preset deep neural network for image recognition, obtaining a training image descriptor corresponding to the chip multi-source test training image set, and a training image defect inference result corresponding to the chip multi-source test training image set;

[0030] Step S30: Determine the descriptor error of the chip multi-source test training image set according to the training image descriptor to obtain the image descriptor training error;

[0031] Step S40: determining the inference error of the chip multi-source test training image set according to the training image defect inference result to obtain the defect inference training error;

[0032] Step S50: determining a second training error according to the training image defect inference result and the prior defect detection result;

[0033] Step S60: calibrating a preset deep neural network according to the first training error and the second training error to obtain a target wire bonding reliability detection network.

[0034] In step S10, for the acquisition of the chip multi-source test training image set, the computer system screens and organizes from a large amount of existing chip test image data. These images come from a wide range of sources and may be collected at different stages of the chip manufacturing process and under different test environments, covering a variety of imaging methods (such as the optical microscope images, electron microscope images, X-ray imaging images, etc. mentioned above). For example, in the database of a chip manufacturing plant, there are thousands of test images of chips from different batches. The computer system selects those images that can reflect the wire bonding conditions. These images contain various possible wire bonding states, from perfect connection to conditions with varying degrees of defects, to form a chip multi-source test training image set.

[0035] Prior defect detection results refer to the defects in each training image that have been previously determined by other means (such as manual inspection or automatic inspection based on traditional algorithms). For example, for an optical microscope image of a chip, an experienced engineer has previously manually inspected and marked the defect of a cold solder joint at the wire bond. This cold solder joint mark is the corresponding prior defect detection result. The computer system matches these prior defect detection results with the corresponding chip multi-source test training images one by one, providing a reference standard for the subsequent evaluation of the accuracy of network calibration.

[0036] In actual operation, a computer system can establish a data management system to store and manage these images and the corresponding prior defect detection results. This data management system can be implemented using a database management system (DBMS), such as using a relational database (such as MySQL), storing the image data in the form of a binary large object (BLOB) in a database table, and storing the corresponding prior defect detection result description information in another field.

[0037] In step S20, the computer system uses a preset deep neural network to perform feature extraction and defect reasoning on the training image. The preset deep neural network is a pre-built network structure with multiple layers of neuron connections that can automatically learn complex features in the image. When the computer system loads the chip multi-source test training image set into this network, the network will process the image according to its internal algorithm and structure.

[0038] Take a simple multi-layer perceptron (MLP) structure as an example. Assume that the input layer has n neurons, corresponding to the pixel values ​​of the image or the feature values ​​after preprocessing. The hidden layer has m neurons and the output layer has p neurons. For each input training image, the network will transfer information and calculate through the weighted connections between neurons.

[0039] In obtaining training image descriptors, the network will extract features from the image and convert the original information of the image into a more abstract and representative description form. For example, for an electron microscope image of a chip, the network can extract the texture features and shape features of the wire bonding area as training image descriptors. These features can be represented in the form of vectors. Assume that the training image descriptor vector is ,in are different feature values, which reflect the properties of the image in certain feature dimensions.

[0040] For the training image defect inference results, the network will infer whether there is a defect in the wire bond in the image and the type of defect based on the learned pattern. For example, the output may be a probability vector ,in represents the probability of no defects, Indicates the probability of a certain type of defect (such as a broken lead), etc. The computer system obtains the defect inference results of these training images based on the output layer results of the network.

[0041] In the actual calculation process, for the input of neuron j in a certain layer , whose output The calculation of can be achieved through the activation function f, such as the commonly used Sigmoid function ,in is the weight connecting neuron i in the previous layer to neuron j, and b_j is the bias term of neuron j.

[0042] In step S30, the computer system performs a more detailed analysis on the images in the chip multi-source test training image set. For example, for a set of training images containing different chip lead bonding conditions, each image has its corresponding training image descriptor. Assume that some of these images are about the same type of defects (such as lead cold soldering) but in different chips or different imaging angles.

[0043] The computer system performs block operations on these images, dividing each image into multiple small blocks, in order to more accurately locate and analyze the defect type area. For example, an optical microscope image of a chip is divided into multiple small blocks according to a fixed size (such as 10×10 pixels as a small block), so there may be some small blocks in the image that just contain the defect type area of ​​lead cold soldering.

[0044] After the block operation, the computer system selects blocks related to a specific defect type, that is, the defect type area. This screening process may be based on some preset rules, such as the statistical characteristics of the pixel values ​​in the block (such as average brightness, variance, etc.) meet the characteristics of the known defect type area.

[0045] For each defect type area to be screened, the computer system determines the corresponding related training image, that is, the chip multi-source test training image containing the defect type area. Then, the computer system determines the image semantic descriptors of each defect type area to be screened in these related training images to form a descriptor set.

[0046] When evaluating semantic consistency, the computer system will traverse each descriptor in this descriptor set. For example, for the descriptor (representing the descriptor of the i-th screened defect type area in the j-th related training image), the computer system will search for a descriptor with lower similarity among other related descriptors.

[0047] There are many ways to evaluate similarity, such as calculating the Euclidean distance. , the Euclidean distance between them If the distance is greater than a certain threshold, it is considered that the two descriptors have significant semantic differences.

[0048] The computer system performs such an analysis on the descriptors of all the screened defect type areas and calculates the overall image descriptor training error. This error reflects the degree of inconsistency between the descriptors generated by the network when extracting features for the same defect type area.

[0049] In step S40, the computer system first determines a plurality of defect type region binary groups. For example, consider two optical microscope training images of a chip, one image has a slight crack defect in a wire bonding area (recorded as the first defect type area), and the other image is its corresponding clone image or an image under similar test conditions, and the area also has the same defect in this image (recorded as the second defect type area), thus forming a defect type region binary group.

[0050] For each defect type region binary, the computer system determines its corresponding defect type reasoning binary, that is, the wire bonding defect reasoning result obtained after the network performs defect reasoning on these two defect type regions. Assume that in the network output, the probability that the reasoning result for the first defect type region is a defect is , the inference result for the second defect type area is that the probability of the existence of defects is ,So This is the defect type region binary corresponding to the defect type reasoning binary. Then, the computer system performs error calculation on each defect type reasoning binary. Simple difference calculation can be used, such as error e = The defect reasoning training error is obtained by averaging the reasoning errors corresponding to all defect type region pairs. This error reflects the fluctuation of the network's defect reasoning results for the same defect type region, that is, the stability and accuracy of the network in defect reasoning.

[0051] In step S50, the computer system determines the second training error by combining the network's own reasoning results with the a priori known correct results.

[0052] For each image in the chip multi-source test training image set, the computer system has both the training image defect inference results given by the network (such as the possibility of defect type represented by the probability vector mentioned earlier), and the corresponding prior defect detection results (such as the accurate defect type manually marked).

[0053] For example, for an electron microscope training image of a chip, the prior defect detection result shows that there is a broken wire defect at the wire bonding site, while the network's training image defect reasoning result gives a probability of a broken wire defect of p. The computer system can calculate the error according to a certain rule. A simple method is that if the prior result shows that a certain defect exists (1 indicates existence, 0 indicates non-existence), the difference between the probability p and 1 given by the network |p-1| can be used as an error component. If the prior result shows that there is no defect, the difference between the probability p and 0 given by the network |p-0| is used as an error component. These error components of all training images are comprehensively calculated, such as averaging, to obtain the second training error. This second training error reflects the overall degree of deviation between the network's reasoning results and the prior correct results.

[0054] In step S60, the computer system uses the first training error (including one or more of the image descriptor training error and the defect inference training error) and the second training error calculated previously to adjust the parameters of the preset deep neural network, thereby obtaining an optimized target wire bonding reliability detection network.

[0055] During the calibration process, the computer system can use the backpropagation algorithm. Assume that the network's loss function L is a combination of the first training error and the second training error (e.g., a simple addition of ,in is the first training error, is the second training error).

[0056] The back propagation algorithm adjusts the weight of each neuron in the network according to the loss function. and the bias term Take the derivative and get the gradient information. For example, for the weight , whose gradient It means that the loss function L changes with The computer system uses this gradient information and follows a certain learning rate (a pre-set small positive number, such as 0.01) to update the weights and bias terms. The update formula is: .

[0057] Through multiple iterations of this process, the parameters of the network are continuously adjusted to gradually reduce the loss function, that is, the first training error and the second training error gradually decrease until the preset stop condition is reached (such as the error is less than a certain threshold or the number of iterations reaches the set value). The final tuned network is the target wire bonding reliability detection network, which has higher accuracy and reliability in chip wire bonding reliability detection.

[0058] In summary, steps S10-S60 in the calibration process of the target wire bonding reliability detection network is a complex and systematic process. Each step involves in-depth processing of image data, network structure and error calculation. Through the synergistic effect of these steps, the network can be effectively optimized and the performance of chip wire bonding reliability detection can be improved.

[0059] As an implementation manner, step S10, obtaining a chip multi-source test training image set and a priori defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set, includes:

[0060] Step S11: acquiring a plurality of first chip multi-source test training images in a set image set;

[0061] Step S12: cloning each first chip multi-source test training image to obtain a second chip multi-source test training image corresponding to each first chip multi-source test training image;

[0062] Step S13: generating a chip multi-source test training image set according to the plurality of first chip multi-source test training images and the second chip multi-source test training images respectively corresponding to the plurality of first chip multi-source test training images;

[0063] Step S14: Determine the priori defect detection results corresponding to the chip multi-source test training image set according to the priori defect detection results corresponding to the set image set.

[0064] In step S11, the image set is a predetermined set of images containing a large number of chip-related images. This set may be an image database from a chip manufacturing company, which stores images obtained using different test equipment at various stages of the chip production process. These images are multi-source, for example, images of chip wire bonding appearance observed under an optical microscope, images of microstructures obtained by an electron microscope, and images of chip internal structures displayed by X-ray imaging.

[0065] The computer system selects a plurality of first chip multi-source test training images from the huge set of set images. The selection process may be based on certain specific criteria. For example, in order to ensure the diversity and representativeness of the training images, the computer system may select according to different batches of chips, different production processes or different chip models.

[0066] Assume that the image set contains 10,000 chip-related images, covering images from 10 different batches of chip production processes. The computer system can perform stratified sampling based on batches, extracting a certain proportion (e.g., 10%) of images from each batch as the first chip multi-source test training images. For optical microscope images, if there are 500 such images in a batch, the computer system will extract 50 of them.

[0067] From a technical point of view, the computer system can use database query statements to implement this filtering operation. If the set of images is stored in a relational database (such as MySQL), the computer system can write SQL query statements to filter out qualified images based on the image metadata (such as batch number, imaging device type, etc.). For example, the query statement may be similar to "SELECT * FROM chip_images WHERE batch_number LIKE '%batch1%' ANDimaging_device='optical_microscope' LIMIT 50", where it is assumed that 50 images are selected from images whose batch number contains "batch1" and the imaging device is an optical microscope.

[0068] In step S12, the computer system performs a cloning operation to increase the amount of training data and maintain a certain relevance of the data. For each selected first chip multi-source test training image, the computer system creates an identical copy thereof through a specific algorithm and technical means, and this copy is the corresponding second chip multi-source test training image.

[0069] Taking a simple image storage format as an example, if the image stores pixel values ​​in the form of a two-dimensional array (assuming it is a grayscale image, each pixel value is between 0-255), the computer system can directly copy this two-dimensional array to achieve cloning. For example, for a first chip multi-source test training image of size M×N (M is the height of the image, N is the width of the image), its pixel value array is , the computer system creates a new two-dimensional array J[m][n] and obtains the cloned image by cyclically assigning J[m][n]=I[m][n].

[0070] In actual image data formats (such as TIFF or JPEG), the computer system follows the corresponding image file format specifications to perform cloning operations. For example, for a TIFF image, the computer system parses the information in the TIFF file header, including the image size, color mode, etc., and then creates a new TIFF file with the same parameters and copies the pixel data of the original image to the new file.

[0071] This cloning operation can provide more data samples for subsequent training processes without changing the content of the image itself. For example, in deep learning training, more data samples help improve the generalization ability of the model. Assuming that some feature information in the original first chip multi-source test training image may be ignored by the model due to the small amount of data, after increasing the amount of data through cloning, the model has more opportunities to learn the relationship between these features and the reliability of chip wire bonding.

[0072] In step S13, the computer system combines the previously acquired and cloned images to form a chip multi-source test training image set, which includes the original first chip multi-source test training images and their corresponding cloned images (second chip multi-source test training images).

[0073] For example, assuming that 100 first chip multi-source test training images are obtained in step S11, after the cloning operation in step S12, 100 corresponding second chip multi-source test training images are obtained. The computer system combines these 200 images (100 first chip multi-source test training images and 100 second chip multi-source test training images) into a new set, which is the chip multi-source test training image set.

[0074] From the perspective of data structure, the computer system can use array or list data structure to store this image set. If you use array structure, for example in the programming language Python, you can create an array containing these image objects. Assuming that the image object is an instance of the `Image` class, you can use the following code to create an array: `training_image_set=[Image() for _ in range(200)]`, where 200 instances of the `Image` class are created first, and then the actual image data can be loaded into these instances.

[0075] This chip multi-source test training image set will serve as the basic data source for subsequent network training and tuning. Its diversity (including original images and cloned images) helps the network learn more comprehensive chip wire bonding features and patterns, both in normal wire bonding situations and in the presence of various defects.

[0076] In step S14, the computer system determines the prior defect detection result for the chip multi-source test training image set just generated. Since the chip multi-source test training image set is selected and cloned from the set image set, the prior defect detection result of the set image set can be used to determine the prior defect detection result corresponding to the chip multi-source test training image set.

[0077] For example, the prior defect detection results of the set image set may be stored in a table in a database, where each row corresponds to an image, which includes the image identifier (such as a file name or a unique identifier) ​​and the corresponding defect detection result (such as no defect, broken lead, poor solder joint, etc.). The computer system maps the prior defect detection results in the set image set to the chip multi-source test training image set by matching the images in the chip multi-source test training image set with the identifiers of the images in the set image set. Assume that the prior defect detection results corresponding to the set image set are stored in a table called `prior_defect_detection`, which has fields `image_id` (image identifier) ​​and `defect_result` (defect result). For each image in the chip multi-source test training image set, the computer system searches for the corresponding `image_id` in the `prior_defect_detection` table, and then obtains the corresponding `defect_result` as the prior defect detection result of this image in the chip multi-source test training image set.

[0078] If you use database connection operations to implement this process, in a relational database (such as MySQL), you can use the JOIN operation or nested queries to achieve this matching. For example, assuming that the chip multi-source test training image set is stored in a table called training_images, which has a field image_id, you can use the following query statement: "SELECT training_images.image_id, prior_defect_detection.defect_result FROMtraining_images JOIN prior_defect_detection ON training_images.image_id=prior_defect_detection.image_id".

[0079] In this way, each image in the chip multi-source test training image set has a corresponding prior defect detection result. These results will be used as a reference standard in the subsequent network calibration process to evaluate the training effect and accuracy of the network.

[0080] In summary, sub-steps S11-S14 in step S10 are a complete data preparation process, from obtaining the original first chip multi-source test training image, to cloning the second chip multi-source test training image, and then to generating a complete chip multi-source test training image set and determining its corresponding prior defect detection results. Each step provides an important data basis for the adjustment of the target wire bonding reliability detection network.

[0081] As an implementation mode, step S20, loading the chip multi-source test training image set into a preset deep neural network for image recognition, obtaining a training image descriptor corresponding to the chip multi-source test training image set, and a training image defect inference result corresponding to the chip multi-source test training image set, including:

[0082] Step S21: loading the chip multi-source test training image set into the training image semantic extraction component in the preset deep neural network to perform feature extraction to obtain a training image descriptor;

[0083] Step S22: Load the training image descriptor into the defect type inference component in the preset deep neural network to perform defect type inference to obtain the training image defect inference result.

[0084] When the computer system executes step S21, the preset deep neural network is a neural network model with a multi-layer structure specially constructed for chip wire bonding reliability detection. The training image semantic extraction component is responsible for extracting features with semantic information from the input chip multi-source test training image set, and these features are presented in the form of training image descriptors.

[0085] When the computer system loads the chip multi-source test training image set into the training image semantic extraction component, taking the convolutional neural network (CNN) as an example, its working principle is as follows. For each image in the chip multi-source test training image set, it is assumed to be an image of chip wire bonding under an optical microscope, and the image exists in the form of a pixel matrix. The convolution layer is one of the core parts of CNN, and the convolution kernel slides on the image to perform convolution operations. For example, a 3×3 convolution kernel starts from the upper left corner of the image and slides in sequence according to a certain step size (such as a step size of 1), performing a weighted sum operation on the covered 3×3 pixel area. This operation can be expressed by the formula: , where O(x,y) is the value of the convolution result at the coordinate (x,y), I(x+i,y+j) is the pixel value of the input image at the corresponding coordinate, and K(i,j) is the weight value of the convolution kernel at the (i,j) position.

[0086] Through convolution operations, the network can automatically extract local features in the image, such as the edge and shape of the chip lead. As the number of network layers increases, these local features will gradually be combined into more abstract and advanced features. For example, in the subsequent convolutional layers, the overall texture features of the wire bonding area can be extracted, which are of great significance for judging the reliability of wire bonding.

[0087] In addition to the convolutional layer, the pooling layer also plays a role in the feature extraction process. The maximum pooling layer is one of the common pooling methods. It selects the maximum value in a small area (such as a 2×2 area) as the output. For example, for a 2×2 pixel area [a,b;c,d], the output after maximum pooling is max(a,b,c,d). The role of the pooling layer is to reduce the amount of data while retaining the most important feature information to prevent overfitting.

[0088] After multiple convolutional layers and pooling layers, the fully connected layer integrates the previously extracted features. Suppose a feature vector is obtained after the previous operation. , this feature vector is a form of training image descriptor. It contains information about the chip multi-source test training image at the semantic level, which reflects various characteristic attributes of chip wire bonding in the image.

[0089] In step S22, after obtaining the training image descriptor, the computer system loads it into a defect type inference component in a preset deep neural network. This component infers the possible defect types of chip wire bonding based on the previously extracted training image descriptor with semantic information.

[0090] Continuing with the convolutional neural network as an example, assume that the training image descriptor is an n-dimensional vector , the fully connected layer in the defect type inference component processes this vector. Each neuron in the fully connected layer has a connection weight with each element of the input vector. For example, for a fully connected layer with m neurons, the input of the i-th neuron is It can be expressed as ,in is the connection weight, is the bias term.

[0091] Then, the output of the neuron is converted into a probability value through an activation function (such as the Softmax function). The Softmax function is used for the input vector , the output probability of its kth element for This probability value indicates the likelihood of a certain type of defect existing in the chip wire bonds.

[0092] For example, if m=3, representing three situations: no defect, lead breakage, and poor solder joint, then the output probability vector is It represents the probability that the chip wire bonding in the training image belongs to these three situations. This probability vector is the defect reasoning result of the training image.

[0093] In actual chip wire bonding inspection, different defect types may have different feature patterns reflected in the training image descriptors. For example, for wire break defects, the training image descriptors may show specific values ​​or combinations in certain feature dimensions (such as features related to the wire shape). The defect type inference component learns the relationship between a large number of training image descriptors and the corresponding prior defect detection results, so that it can accurately infer the defect type of new training image descriptors.

[0094] In summary, steps S21-S22 are the key steps for the preset deep neural network to extract features and infer defect types from the chip multi-source test training image set. The training image semantic extraction component extracts the training image descriptors, and then the defect type inference component performs defect type inference based on these descriptors to obtain the training image defect inference results, which provides important data basis for subsequent network calibration.

[0095] As an implementation mode, step S30, based on the training image descriptor, determines the descriptor error of the chip multi-source test training image set to obtain the image descriptor training error, including:

[0096] Step S31: performing an image block operation on each chip multi-source test training image in the chip multi-source test training image set to obtain a plurality of training defect type regions corresponding to the chip multi-source test training image set;

[0097] Step S32: screening multiple training defect type regions to obtain one or more screened defect type regions;

[0098] Step S33: determining one or more relevant training images corresponding to each screening defect type region, wherein the one or more relevant training images corresponding to each screening defect type region are chip multi-source test training images in the chip multi-source test training image set that contain each screening defect type region;

[0099] Step S34: determining one or more relevant descriptors corresponding to each screening defect type region and a screening descriptor set corresponding to one or more screening defect type regions according to one or more relevant training images corresponding to each screening defect type region; wherein any relevant descriptor corresponding to each screening defect type region is an image semantic descriptor corresponding to each screening defect type region in any relevant training image corresponding to each screening defect type region, and the screening descriptor set includes one or more relevant descriptors corresponding to one or more screening defect type regions respectively;

[0100] Step S35: determining a relevant descriptor set corresponding to each relevant descriptor according to one or more relevant descriptors corresponding to each screening defect type region; the relevant descriptor set corresponding to any relevant descriptor includes relevant descriptors other than any relevant descriptor in the one or more relevant descriptors corresponding to the target screening defect type region; the target screening defect type region is a screening defect type region corresponding to any relevant descriptor in the one or more screening defect type regions;

[0101] Step S36: According to the relevant descriptor set and the screening descriptor set corresponding to each relevant descriptor, a semantic consistency evaluation is performed on one or more screened defect type regions to obtain an image descriptor training error.

[0102] In step S31, the computer system analyzes the characteristic information in the chip multi-source test training image in more detail, especially the defect type area related to the wire bonding reliability. For each image in the chip multi-source test training image set, the image block operation is to divide the image into several smaller sub-areas, which are so-called blocks.

[0103] Take an optical microscope image of a chip as an example. The image may have a high resolution, such as 1000×1000 pixels. The computer system can divide it into blocks of fixed size (such as 100×100 pixels). In this way, the entire image is divided into =10×10=100 blocks.

[0104] This block operation can be implemented based on the coordinate system of the image. The computer system can determine the range of each block by defining the starting coordinates of each block in the image (such as the coordinates (x, y) of the upper left corner vertex) and the size of the block (width and height).

[0105] Some of these blocks may contain information related to chip wire bonding defects, such as wire breakage, cold solder joints, etc., which may appear in one or some blocks. These blocks containing defect-related information constitute multiple training defect type regions corresponding to the chip multi-source test training image set. This block operation helps the computer system locate and analyze defects more accurately, because different defects may present specific characteristic patterns in different local areas of the image.

[0106] In step S32, after obtaining a plurality of training defect type regions, the computer system screens them to obtain regions that are more representative and valuable for analysis.

[0107] First, the computer system can perform commonality assessment on each training defect type area and the set image. The set image here can be an image with standard features or known normal features, which is used as a benchmark for comparison. For example, the set image is an optical microscope image of a completely normal chip wire bonding. For each training defect type area, the computer system can perform commonality assessment by comparing the differences in pixel value distribution, texture features, etc. with the set image.

[0108] Assume that for a training defect type area, the mean pixel value is , the variance is , and the mean pixel value of the corresponding area of ​​the image is set to , the variance is The computer system can use some statistical indicators to measure the differences between them, such as calculating the relative difference If these differences exceed a pre-set threshold (e.g. ), it is considered that the area is significantly different from the set image and may be the area that actually contains defect information.

[0109] After completing the commonality assessment, the computer system removes the training defect type areas that belong to the set image from multiple training defect type areas based on the commonality assessment results. Here, "belonging to the set image" refers to those areas that are too similar to the set image in features (i.e., the difference does not exceed the threshold), and these areas may not contain real defect information. For example, if a part of the area , these areas are considered to contain no defect information and are cleared.

[0110] Finally, the cleaned training defect type regions are de-redundant. De-redundant means removing regions that are very similar or repeated in features. For example, there may be multiple adjacent training defect type regions whose pixel value distributions, texture features, etc. are almost the same. This may be due to repeated information caused by some factors in the image acquisition process. The computer system can determine whether it is a redundant region by calculating the similarity index between regions (such as cosine similarity or Euclidean distance, etc.). For two training defect type regions A and B, if the Euclidean distance is used to measure the similarity, let the feature vector of region A be , the eigenvector of region B is , then the Euclidean distance between them .if ( is a pre-set threshold), the two regions are considered redundant and only one of them is retained. After such a screening process, the computer system finally obtains one or more screening defect type regions.

[0111] In step S33, after obtaining the screening defect type area, the computer system needs to determine the training image related to each screening defect type area. Since the chip multi-source test training image set includes multiple different chip test images, each screening defect type area may appear in one or more of the images.

[0112] For example, suppose a specific screening defect type area is screened out, which represents a cold solder joint area on the chip lead. The computer system will traverse the chip multi-source test training image set and check whether each image contains this cold solder joint area. If the characteristics of this cold solder joint area are found in an optical microscope image of a chip (for example, through the previously mentioned pixel value distribution, texture features, etc., which match the screening defect type area), then this image is the relevant training image corresponding to the screening defect type area.

[0113] This process can be achieved by performing feature matching on the image. For the feature vector of the screening defect type area (such as the specific feature representation obtained by the previous image segmentation and screening process), the computer system compares it with the corresponding area of ​​each image in the chip multi-source test training image set (the area may be obtained through the same segmentation operation). The comparison method can be to calculate the similarity between the feature vectors, such as using the Euclidean distance or cosine similarity mentioned above. If the similarity reaches a certain threshold (for example, the cosine similarity is greater than 0.8), the image is considered to be a related training image containing the screening defect type area.

[0114] In step S34, after the computer system determines the relevant training images corresponding to each screening defect type area, it needs to further obtain the relevant descriptors in these images. In the previous step, the training image semantic extraction component in the preset deep neural network has been used to extract features from the chip multi-source test training image set to obtain the training image descriptors.

[0115] For each screening defect type region and its corresponding related training image, the computer system extracts the image semantic descriptor corresponding to the screening defect type region from these related training images. For example, for a screening defect type region representing a lead break, in a corresponding optical microscope related training image, a training image descriptor in the form of a feature vector was previously obtained by the training image semantic extraction component. , the part of this descriptor related to the lead break area (which can be determined by a pre-defined mapping relationship or feature selection algorithm) is the relevant descriptor of the screened defect type area in this relevant training image.

[0116] For all the screening defect type areas, the computer system collects the relevant descriptors corresponding to each screening defect type area to form a set of screening descriptors corresponding to one or more screening defect type areas. For example, if there are three screening defect type areas A, B, and C, and area A has two relevant training images, the corresponding relevant descriptors are ; Region B has three related training images, and their corresponding related descriptors are ; Region C has a related training image, and its corresponding related descriptor is . Then the filter descriptor set is .

[0117] In step S35, after constructing the screening descriptor set, the computer system determines the corresponding related descriptor set for each related descriptor. For any related descriptor, its corresponding target screening defect type area is clear. For example, for the related descriptors in the screening defect type area A mentioned above, , the target defect type area is A. Then The corresponding set of related descriptors is , that is, except In addition to itself, other related descriptors corresponding to the defect type areas are filtered.

[0118] This process helps in the subsequent analysis of the relationship between different descriptors, especially for evaluating the degree of semantic consistency between each relevant descriptor and other relevant descriptors.

[0119] In step S36, after determining the relevant descriptor set and the screened descriptor set, the computer system performs a semantic consistency evaluation to obtain the image descriptor training error. The semantic consistency evaluation is intended to measure the semantic similarity or consistency between different relevant descriptors.

[0120] For each relevant descriptor, the computer system compares it with other descriptors in the corresponding relevant descriptor set and all descriptors in the filtered descriptor set. , the computer system will calculate it and the related descriptor set Each descriptor in and the filtered descriptor set Some similarity measure between all descriptors in .

[0121] A commonly used similarity measure is cosine similarity. , the cosine similarity between them .

[0122] If there are m descriptors in the filter descriptor set, for one of the relevant descriptors , the computer system calculates its similarity value with other m-1 descriptors Then, the contribution of the relevant descriptor to semantic consistency is determined based on the distribution of these similarity values. For example, if most of the similarity values ​​are relatively low (below a certain threshold, such as 0.5), it means that the relevant descriptor has a large semantic difference with other descriptors, which has a large negative impact on semantic consistency.

[0123] The computer system performs such an analysis on all relevant descriptors, and combines these analysis results according to certain rules to obtain the image descriptor training error. This error reflects the consistency between the image semantic descriptors for the same or similar screening defect type areas in different training images. If this error is large, it means that when the network extracts features for these areas, the descriptors generated are not stable enough or have poor semantic consistency, and it is necessary to further adjust the network parameters or improve the feature extraction method.

[0124] In summary, sub-steps S31-S36 are a process of gradually and deeply analyzing the defect type areas in the chip multi-source test training images, by image segmentation, screening, determining related images and descriptors, and finally performing semantic consistency evaluation to obtain the image descriptor training error.

[0125] As an implementation mode, step S36, based on the relevant descriptor set and the screening descriptor set corresponding to each relevant descriptor, performs semantic consistency evaluation on one or more screened defect type regions to obtain the image descriptor training error, including:

[0126] Step S361: walking through multiple related descriptors in the screening descriptor set one by one;

[0127] Step S362: determining, based on the filtered descriptor set, a plurality of first related descriptors corresponding to the currently visited related descriptor; wherein the plurality of first related descriptors are related descriptors other than the currently visited related descriptor in the filtered descriptor set;

[0128] Step S363: determining a target related descriptor set corresponding to the currently wandered related descriptor in the related descriptor set corresponding to each related descriptor; the target related descriptor set includes a plurality of second related descriptors;

[0129] Step S364: perform similarity evaluation on the currently visited related descriptor and each first related descriptor to obtain a first evaluation result corresponding to each first related descriptor; the first evaluation result corresponding to each first related descriptor represents the similarity between each first related descriptor and the currently visited related descriptor;

[0130] Step S365: perform similarity evaluation on the currently visited related descriptor and each second related descriptor to obtain a second evaluation result corresponding to each second related descriptor; the second evaluation result corresponding to each second related descriptor represents the similarity between each second related descriptor and the currently visited related descriptor;

[0131] Step S366: determining a third training error corresponding to the currently wandered related descriptor according to the second evaluation result corresponding to each second related descriptor and the first evaluation results corresponding to the plurality of first related descriptors;

[0132] Step S367: After the walk is completed, the plurality of third training errors obtained during the walk are averaged to obtain the image descriptor training error.

[0133] In step S361, the computer system starts to traverse the relevant descriptors in the screening descriptor set. The screening descriptor set contains multiple descriptors corresponding to the screening defect type areas, which are image semantic descriptors extracted from relevant training images before and exist in the form of vectors.

[0134] For example, suppose the filter descriptor set is ,in represents the dimension of the descriptor vector. The computer system starts from the first descriptor At the beginning, each descriptor is operated in sequence, just like visiting elements one by one in a list. This one-by-one walking method is to comprehensively evaluate the relationship between each descriptor and other descriptors, so as to perform subsequent semantic consistency evaluation.

[0135] The computer system can implement this walking operation by setting an index variable, starting from index 0 and adding 1 each time until the entire set of filter descriptors is traversed.

[0136] In step S362, when the computer system wanders to a relevant descriptor (e.g. ), it is necessary to determine multiple first relevant descriptors corresponding to it. These first relevant descriptors are other descriptors in the descriptor set that are filtered except the descriptor currently wandered to.

[0137] Continuing with the above example, when the computer system wanders to When , the corresponding multiple first related descriptors are { The purpose of this determination is to distinguish the current descriptor from other descriptors in the set so as to compare the relationship between them and evaluate the semantic consistency.

[0138] The computer system can determine the first relevant descriptor by determining the position of the descriptor in the set. If an index is used to represent the position of the descriptor in the set, for the descriptor with index i that is wandered to, the first relevant descriptor is all descriptors whose index is not equal to i.

[0139] In step S363, for each relevant descriptor, there is a corresponding relevant descriptor set (determined in the previous step). ), it is necessary to determine a target related description subset in its corresponding related description subset.

[0140] For example, suppose the relevant descriptor The corresponding related description subset is ,The computer system determines a subset from this set as the target-related description subset according to certain rules (e.g., according to the current analysis requirements or some previously set conditions), such as , the elements in this subset That is, multiple second related descriptors. The determination of this target related descriptor set helps to analyze the relationship between the current wandering related descriptor and other related descriptors from a more specific scope, and further refine the evaluation of semantic consistency.

[0141] In actual operation, the computer system can determine the target-related descriptor set based on certain characteristics between descriptors (such as the range of values ​​on certain feature dimensions) or pre-set weights.

[0142] In step S364, after determining the first relevant descriptor, the computer system needs to search for the currently visited relevant descriptor (e.g. ) and each first relevant descriptor for similarity evaluation. Similarity evaluation is an operation that measures how close two descriptors are in semantics.

[0143] A common similarity evaluation method is to calculate the cosine similarity. Assume that the relevant descriptor currently wandered is , a first relevant descriptor , the cosine similarity between them (i.e. the first evaluation result) The formula can be Calculated.

[0144] For example, if The closer this value is to 1, the higher the similarity between the two descriptors; the closer it is to 0, the lower the similarity.

[0145] By performing such calculations on each first relevant descriptor, the computer system can obtain a first evaluation result corresponding to each first relevant descriptor, and these results reflect the degree of similarity between them and the currently wandered relevant descriptor.

[0146] In step S365, similar to step S364, after determining the second relevant descriptor, the computer system performs a search on the currently visited relevant descriptor (e.g. ) and each second related descriptor are evaluated for similarity.

[0147] The same cosine similarity calculation method is used, assuming that the relevant descriptor currently wandered to , a second relevant descriptor , the cosine similarity between them (i.e. the second evaluation result) By formula Calculated.

[0148] For example, if According to the above formula, we can calculate the This value reflects the similarity between the second relevant descriptor and the currently wandered relevant descriptor, providing an important basis for the subsequent comprehensive evaluation of semantic consistency.

[0149] In step S366, after obtaining the second evaluation result corresponding to each second related descriptor and the first evaluation results corresponding to multiple first related descriptors, the computer system needs to combine these results to determine the third training error corresponding to the currently wandered related descriptor.

[0150] For example, suppose the relevant descriptor currently wandered to is , and its corresponding second related descriptor is , the corresponding first related descriptor is Their corresponding second evaluation results are The first evaluation results are .

[0151] The computer system can determine the third training error in a weighted manner. Assume that the weight given to the second related descriptor is , the weight given to the first relevant descriptor is ( and It is pre-set according to specific analysis requirements, e.g. =0.6, =0.4).

[0152] First, calculate the contribution of the second relevant descriptor to the error (Here ri represents the second relevant descriptor), and then calculate the contribution of the first relevant descriptor to the error (Here rj represents the first relevant descriptor).

[0153] Finally, the relevant descriptor currently wandered to The corresponding third training error This calculation method comprehensively considers the similarity between the second relevant descriptor and the first relevant descriptor and the currently wandered relevant descriptor, thereby reflecting the error of the relevant descriptor in terms of semantic consistency.

[0154] In step S367, after the computer system performs the above operation on each relevant descriptor in the screened descriptor set and obtains the third training error corresponding to each relevant descriptor, it is necessary to perform comprehensive processing on these errors to obtain the image descriptor training error.

[0155] For example, suppose that during the walk, for N related descriptors, the third training error is obtained The computer system calculates their average value to obtain the image descriptor training error .

[0156] This image descriptor training error reflects the overall semantic consistency of the entire screening descriptor set. If this error is large, it means that in different related training images, for the same or similar screening defect type areas, there is a large semantic inconsistency between the image semantic descriptors extracted by the network, which may require adjustments to the preset deep neural network, such as adjusting the network structure or training parameters, to improve the accuracy and stability of the network in feature extraction.

[0157] In summary, sub-steps S361-S367 are a process of gradually and deeply analyzing the relationship between related descriptors to evaluate semantic consistency and finally obtain the image descriptor training error. This process comprehensively and carefully examines the descriptors in the screened descriptor set through a variety of similarity evaluation and error calculation methods, providing an important basis for optimizing the preset deep neural network.

[0158] As an implementation mode, step S366, determining a third training error corresponding to the currently wandered related descriptor according to the second evaluation result corresponding to each second related descriptor and the first evaluation results corresponding to the plurality of first related descriptors, includes:

[0159] Step S3661: determining the inference confidence corresponding to each second-related descriptor according to the second evaluation result corresponding to each second-related descriptor and the first evaluation results corresponding to the plurality of first-related descriptors; the inference confidence corresponding to each second-related descriptor is used to indicate the confidence that each second-related descriptor is correlated with the currently wandered related descriptor;

[0160] Step S3662: Determine a fourth training error corresponding to each second-related descriptor according to the inference confidence corresponding to each second-related descriptor; the fourth training error is used to indicate the error between each second-related descriptor and the currently wandered related descriptor;

[0161] Step S3663: adding the fourth training errors corresponding to the plurality of second related descriptors respectively to obtain the third training error.

[0162] In step S3661, the computer system determines the inference confidence corresponding to each second related descriptor based on the similarity evaluation results previously obtained (i.e., the second evaluation results and the first evaluation results). The inference confidence reflects the reliability of the correlation between the second related descriptor and the currently wandered related descriptor.

[0163] Assume that the relevant descriptor currently wandered to is , there are m second related descriptors , and n first relevant descriptors For each second relevant descriptor , and its corresponding second evaluation result is (express ), the first evaluation results corresponding to the first relevant descriptor are .

[0164] The computer system can use a relative comparison method to determine the confidence of the inference. For example, first calculate a comprehensive similarity measure , which is the second evaluation result The average first evaluation result of the first related descriptor One possible combination is weighted sum. Let the weight of the second evaluation result be , the weight of the first evaluation result is ( and It is set based on experience or specific problems, e.g. =0.6, =0.4), then .

[0165] Then, this comprehensive similarity measure Mapped to a confidence interval [0,1], the inference confidence is obtained A simple mapping method is to use a logistic function (such as the Sigmoid function). ,in is a threshold parameter (e.g. = 0.5), which is used to adjust the sensitivity of the confidence level. When larger, Close to 1, it means that the confidence of the inference is high, that is, There is a high confidence level that there is a correlation between When smaller, Close to 0, it means the inference confidence is low.

[0166] For example, suppose First calculate , which means the second relevant descriptor Related descriptors to the current location There is a high confidence level of correlation between them.

[0167] This way of determining the inference confidence comprehensively considers the direct similarity between the second relevant descriptor and the currently wandered relevant descriptor (second evaluation result) and the relative similarity with other first relevant descriptors (first evaluation result), thereby more comprehensively evaluating the confidence of their correlation.

[0168] In step S3662, after obtaining the inference confidence corresponding to each second relevant descriptor, the computer system uses the confidence to determine the fourth training error corresponding to each second relevant descriptor. The fourth training error is intended to quantify the degree of difference between the second relevant descriptor and the currently wandered relevant descriptor, and this difference is not only based on simple similarity, but also combined with the previously determined inference confidence.

[0169] For each second relevant descriptor , and its corresponding inference confidence is The computer system can use a confidence-based error calculation method. For example, assuming that there is an ideal perfect correlation (inference confidence is 1), the difference between the two descriptors is 0; when the inference confidence is 0, the difference is the largest, set ( It is based on the specific descriptor feature space or a pre-set larger value. For example, for a descriptor vector with a dimension of n, ).

[0170] Then, each second relevant descriptor The corresponding fourth training error The formula can be Calculated.

[0171] For example, continuing with the previous example, suppose (Assuming the descriptor vector dimension is 5), for the second related descriptor , its inference confidence This value indicates that, taking into account the confidence of inference, Related descriptors to the current location The degree of error between .

[0172] This method of calculating the fourth training error directly links the inference confidence with the error, making the error calculation more consistent with the actual situation, that is, the more unrelated (the lower the inference confidence) the descriptors, the greater the error.

[0173] In step S3663, after determining the fourth training error corresponding to each second related descriptor, the computer system adds these errors to obtain a third training error. The third training error comprehensively reflects the overall error between all second related descriptors and the currently wandered related descriptor.

[0174] Assume that for the relevant descriptor currently wandering , there are m second related descriptors , and their corresponding fourth training errors are .

[0175] Then the third training error (against ) through the formula Calculated.

[0176] For example, suppose there are three second relevant descriptors , and their corresponding fourth training errors are , then the third training error .

[0177] This third training error plays a key role in the entire semantic consistency evaluation process. It reflects the overall error from the perspective of reasoning confidence and error calculation when considering the relationship between the second related descriptor and the currently wandered related descriptor. This error will be further used in subsequent steps (such as the average calculation in step S367) to finally obtain the image descriptor training error, thereby providing an important basis for evaluating and optimizing the preset deep neural network.

[0178] To sum up, sub-steps S3661-S3663 gradually calculate the inference confidence, the fourth training error, and finally obtain the third training error. This is a meticulous process of evaluating the relationship between related descriptors from multiple aspects to quantify the error. This process is of great significance for accurately evaluating the semantic consistency and training error of image descriptors.

[0179] As an implementation mode, step S32, screening multiple training defect type regions to obtain one or more screened defect type regions, includes:

[0180] Step S321: performing commonality assessment on each training defect type region and a set image to obtain commonality assessment results corresponding to a plurality of training defect type regions;

[0181] Step S322: according to the commonality evaluation result, clearing the training defect type regions belonging to the set image among the multiple training defect type regions to obtain the cleared training defect type regions;

[0182] Step S323: performing redundancy elimination on the cleaned training defect type regions to obtain one or more screening defect type regions.

[0183] In step S321, when the computer system executes this step, its purpose is to determine the similarity or commonality of features between each training defect type area and the set image. The set image here is a reference image with a specific meaning, which may represent a chip image in a normal state or an image with certain standard features.

[0184] For each training defect type area, the computer system will conduct commonality assessment from multiple aspects. Taking the optical microscope image of the chip as an example, it is assumed that the training defect type area refers to the area on the chip lead where there is suspected cold solder joint, and the set image is an optical microscope image of a normal chip lead bonding.

[0185] First, at the pixel value level, the computer system can calculate the statistical characteristics of the pixel values ​​in the region. For example, calculate the mean and variance. Suppose the pixel values ​​in the training defect type region are (n is the number of pixels in the region), then the mean ,variance Similarly, the corresponding area of ​​the set image (the area at the same position and size) is also calculated to obtain the mean and variance .

[0186] The computer system can then measure the commonality of pixel value features by calculating relative differences. For example, calculating the relative difference of the mean , the relative difference in variance These two values ​​are part of the commonality evaluation results at the pixel value level.

[0187] In addition to pixel values, texture features are also an important aspect. Computer systems can use texture analysis algorithms, such as gray-level co-occurrence matrix (GLCM) to extract texture features. For the training defect type area and the corresponding area of ​​the set image, GLCM is calculated respectively, and then characteristic parameters such as contrast, correlation, energy and homogeneity are extracted from GLCM. For example, the contrast C is calculated as , where L is the grayscale level of the image, and p(i,j) is the element value in GLCM at grayscale levels i and j. By comparing the differences in these texture feature parameters between the two regions, the commonality evaluation results at the texture feature level are obtained.

[0188] Shape features can also be used for commonality assessment. If the training defect type region is related to the shape of the lead, such as the bend of the lead, the computer system can use shape descriptors to quantify the shape features. For example, using Fourier descriptors, the boundary contour of the region is represented as the coefficients of the Fourier series. A set of Fourier descriptors is obtained for the training defect type region. , the corresponding area of ​​the set image is obtained . Then, by calculating the Euclidean distance To measure the difference in shape features, this distance value is part of the commonality evaluation results at the shape feature level.

[0189] Combining these evaluation results in terms of pixel value, texture features, shape features, etc., the computer system obtains the commonality evaluation results between each training defect type area and the set image. These results will provide a basis for subsequent judgment of which areas may actually contain defect information.

[0190] In step S322, after completing the commonality assessment, the computer system needs to clean up those training defect type areas that are too similar to the set image according to the commonality assessment results. Here, "belonging to the set image" means that these areas have too high commonality with the set image in terms of features and may not contain real defect information.

[0191] Taking the commonality evaluation results calculated previously as an example, if the relative difference of the mean at the pixel value level is The relative difference between the are all less than a pre-set threshold (e.g. ), and the differences in texture features and shape features are also smaller than their corresponding thresholds (for example, the relative difference in texture feature parameters is less than 0.1, and the Euclidean distance of shape features is less than a fixed value), then the computer system believes that this training defect type area is too similar to the set image and is an area that should be cleaned.

[0192] For example, suppose there are 100 training defect type regions, and after evaluating the commonality between each region and the set image, it is found that the differences in each of the above characteristics of 20 regions are less than the corresponding threshold. The computer system removes these 20 regions from the training defect type region set, and the remaining 80 regions are the cleaned training defect type regions.

[0193] This cleaning operation helps reduce the interference information in the subsequent analysis, so that the computer system can focus more on those areas that may actually contain defect information for further analysis. Technically speaking, the computer system can maintain an index list or a tag array to record which areas need to be cleaned, and then update the set of training defect type areas based on this record.

[0194] In step S323, after obtaining the cleaned training defect type regions, the computer system needs to eliminate redundancy in these regions. The purpose of redundancy elimination is to remove those regions that are very similar or repeated in features, so as to further streamline the objects of analysis and improve the efficiency and accuracy of analysis.

[0195] Continuing with the optical microscope image of the chip as an example, suppose that there are several areas in the cleaned training defect type area that represent suspected areas of poor solder joints on the chip leads, but their positions in the image are very close and they are very similar in terms of pixel values, texture features, and shape features.

[0196] The computer system can use a variety of methods to eliminate redundancy. One method is based on the similarity calculation of feature vectors. First, for each cleaned training defect type area, the computer system represents its features as a feature vector. For example, the mean, variance, texture feature parameters (such as contrast, correlation, etc.) of the pixel value, and the shape feature descriptor (such as some important coefficients of the Fourier descriptor) are combined into a feature vector. .

[0197] Then, calculate the similarity between the feature vectors of any two regions. You can use the cosine similarity formula , n is the dimension of the feature vector.

[0198] If the cosine similarity between the feature vectors of two regions is greater than a pre-set threshold (e.g. ), then the two regions are considered redundant. The computer system can choose to keep one of the regions, for example, the region with clearer features or a more representative position in the image.

[0199] For example, among the 50 training defect type regions after cleaning, by calculating the feature vector similarity between each pair, it was found that the similarity of 10 pairs of regions was greater than 0.9. The computer system processes each pair of redundant regions and finally retains one of them, thus obtaining 40 regions after redundancy elimination, which are one or more screening defect type regions.

[0200] This redundancy elimination operation helps to reduce the duplication of data, making the subsequent analysis of the defect type screening area more focused and effective, and can more accurately reflect the different types of defects in chip wire bonding, providing a more representative and accurate data basis for subsequent steps (such as determining relevant training images, relevant descriptors, etc.).

[0201] In summary, sub-steps S321-S323 are a process of gradually screening and streamlining the training defect type areas. Through commonality evaluation, cleaning and redundancy elimination, the computer system can obtain more valuable and representative screening defect type areas from a large number of training defect type areas.

[0202] As an implementation manner, in step S323, after performing redundancy elimination on the cleaned training defect type regions and obtaining one or more screening defect type regions, the method may further include:

[0203] Step S324: determining one or more universal screening defect type regions among the one or more screening defect type regions; the number of occurrences of any universal screening defect type region among the one or more universal screening defect type regions in the chip multi-source test training image set is greater than a number threshold;

[0204] Step S33, determining one or more relevant training images corresponding to each screening defect type area, including:

[0205] Step S331: Determine one or more relevant training images corresponding to each commonly screened defect type area.

[0206] In step S324, the computer system finds those areas that appear more frequently in the chip multi-source test training image set, namely, the commonly screened defect type areas, from the previously screened defect type areas (including the screened defect type areas after commonality evaluation, cleaning and redundancy elimination).

[0207] Taking the wire bonding inspection of a chip as an example, suppose that after the previous steps, several screening defect type regions are obtained, and these regions represent different possible defect situations, such as wire breakage, cold solder joints, uneven solder joints, etc. For each screening defect type region, the computer system counts its occurrence times in the chip multi-source test training image set.

[0208] For example, the computer system may maintain a counter array, the length of which is equal to the number of the screening defect type regions. When traversing the chip multi-source test training image set, for each screening defect type region in each image, if a screening defect type region (such as a region representing a lead break) is found to exist in the current image, the corresponding counter is incremented by 1.

[0209] Assume that the set number threshold is 5 times. After completing the traversal statistics of the entire chip multi-source test training image set, the computer system checks the value of each counter. If the counter value corresponding to a certain screening defect type area is greater than 5, then this screening defect type area is determined to be a universal screening defect type area. For example, it is found through statistics that the screening defect type area representing the false solder joint appears 8 times in the chip multi-source test training image set, which is greater than the threshold of 5 times, so this screening defect type area for the false solder joint is a universal screening defect type area.

[0210] The significance of determining the commonly screened defect type areas is that these areas frequently appear in the training image set and may be typical or easy-to-occur defect types in the chip wire bonding process. In the subsequent analysis and network tuning process, these commonly screened defect type areas are focused on.

[0211] In addition to the counter array method described above, database query techniques (if the training image set is stored in a database) can also be used to count the number of occurrences of each screening defect type area. For example, query and count operations can be performed in the database table of the image set based on the feature identifier of the screening defect type area (such as a specific texture feature value range or shape feature encoding, etc.).

[0212] In step S331, after the computer system determines the commonly screened defect type areas, it needs to find relevant training images containing these commonly screened defect type areas. This step helps to further analyze the performance of these typical defects in different training images and provide more targeted data for subsequent feature extraction and network calibration.

[0213] Continuing with the chip wire bonding inspection as an example, for each common screening defect type area (such as the common screening defect type area of ​​the previously mentioned cold solder joint), the computer system traverses the chip multi-source test training image set again. During the traversal process, for each training image, the computer system detects whether it contains a specific common screening defect type area. This can be achieved through the feature matching method previously used to determine the screening defect type area. For example, if the common screening defect type area of ​​the cold solder joint has a specific pixel value distribution pattern (such as the variation characteristics of the pixel values ​​around the cold solder joint area), texture features (such as specific grayscale co-occurrence matrix features) or shape features (such as the contour shape of the cold solder joint), the computer system will compare these features with the corresponding areas in each training image.

[0214] The comparison method can be to calculate the similarity measure between features. For example, for texture features, the contrast, correlation and other characteristic parameters of the gray level co-occurrence matrix mentioned above can be used for comparison. Assume that the texture characteristic parameters of the general defect type area are , the texture feature parameter of a certain area in the training image is , the difference measure between them can be calculated If the difference measure is less than a preset threshold (e.g., d < 0.1), and the comparison in pixel value and shape features also meets the corresponding threshold conditions, then it is considered that this training image contains the common screening defect type area, and this training image is the relevant training image corresponding to this common screening defect type area.

[0215] For example, for the common screening defect type area of ​​cold solder joints, when the computer system traverses the 100 images in the chip multi-source test training image set, it is found that 20 of the images have a good match with the features of the common screening defect type area in terms of pixel values, texture features, and shape features of the cold solder joint area (that is, the difference measure is less than the corresponding threshold). Then these 20 images are the relevant training images corresponding to the common screening defect type area of ​​cold solder joints.

[0216] This operation of determining relevant training images can associate the generally screened defect type areas with specific training images, so that feature information related to these typical defects (such as training image descriptors, etc.) can be extracted from these relevant training images later, further providing accurate data support for network calibration, which helps to improve the accuracy of the network in detecting chip wire bonding reliability.

[0217] As an implementation mode, step S40, based on the defect reasoning result of the training image, determines the reasoning error of the chip multi-source test training image set to obtain the defect reasoning training error, including:

[0218] Step S41: Determine a plurality of defect type region binary groups, wherein a defect type region binary group includes a first defect type region and a second defect type region, wherein the first defect type region is any defect type region in the first chip multi-source test training image, and the second defect type region is a defect type region corresponding to the first defect type region in the third chip multi-source test training image, and the third chip multi-source test training image is a second chip multi-source test training image corresponding to the first chip multi-source test training image;

[0219] Step S42: determining a defect type reasoning tuple corresponding to each defect type region tuple, wherein the defect type reasoning tuple corresponding to each defect type region tuple includes a wire bonding defect reasoning result corresponding to the first defect type region in each defect type region tuple and a wire bonding defect reasoning result corresponding to the second defect type region in each defect type region tuple;

[0220] Step S43: Determine the defect type reasoning error of the chip multi-source test training image set according to the defect type reasoning tuple corresponding to each defect type region tuple, and obtain the defect reasoning training error.

[0221] In step S41, the computer system constructs a special region relationship combination, namely, a defect type region binary. In the chip multi-source test training image set, there is a first chip multi-source test training image and its corresponding second chip multi-source test training image (obtained through the previous cloning operation).

[0222] Taking the wire bonding inspection of a chip as an example, assuming that the first chip multi-source test training image is a chip image taken by an optical microscope, there is an area indicating wire breakage, which is the first defect type area. In the corresponding second chip multi-source test training image (because it is a cloned image, in theory, it is very similar to the first chip multi-source test training image except for some possible noise or slight differences), the area indicating wire breakage at the same position or with the same identification (determined by a specific positioning or matching method) is the second defect type area. These two areas (the first defect type area and the second defect type area) constitute a defect type area binary.

[0223] The computer system traverses all first chip multi-source test training images and their corresponding second chip multi-source test training images, and determines multiple defect type area tuples according to such rules. For example, in a set including 100 sets of first chip multi-source test training images and their corresponding second chip multi-source test training images, the computer system checks the defect type areas in each set of images one by one. Assuming that the corresponding defect type areas are successfully found in 50 sets of images and the tuples are constructed, 50 defect type area tuples are obtained.

[0224] In order to accurately determine these binary groups, the computer system can use some image matching techniques. For optical microscope images, the feature points of the image can be used for matching. For example, the feature points are first detected in the first chip multi-source test training image through an algorithm (such as the Harris corner detection algorithm). The positions of these feature points in the image and the pixel features around them are unique. Then, similar feature points are found in the corresponding second chip multi-source test training image, and the matching relationship is determined by comparing the pixel value distribution and texture features around the feature points, thereby determining the corresponding defect type area and constructing a binary group.

[0225] In step S42, after obtaining the defect type region binary, the computer system needs to further determine the corresponding defect type reasoning binary. In the previous step, the chip multi-source test training image set has been loaded into the preset deep neural network for image recognition, and the training image defect reasoning result has been obtained.

[0226] Continuing with the wire bonding detection as an example, for each defect type region binary (such as the region binary representing wire breakage mentioned above), in the first defect type region (the wire breakage region in the first chip multi-source test training image), after the network's defect reasoning, a result indicating the probability of the existence of a wire breakage defect may be obtained as p_1 (assuming that the network output is a probability value, indicating the possibility of various defect types existing in the region). Similarly, for the second defect type region (the corresponding wire breakage region in the second chip multi-source test training image), after the network's defect reasoning, a result indicating the probability of the existence of a wire breakage defect is p_2. Then [p_1, p_2] constitutes the defect type reasoning binary corresponding to this defect type region binary.

[0227] The computer system performs such an operation on each constructed defect type region binary to determine their respective corresponding defect type reasoning binary. This process involves accurately extracting the results related to each defect type region binary from the defect reasoning results of many training images previously output by the network. For example, if the output results of the network for the entire chip multi-source test training image set are stored in the form of a matrix, where rows represent different training images and columns represent different defect type reasoning results (such as the probabilities of different types of defects), the computer system obtains the corresponding reasoning results from this matrix based on the image identification and region location information in the defect type region binary to construct the defect type reasoning binary.

[0228] In step S43, after determining the defect type reasoning tuples corresponding to all defect type region tuples, the computer system starts to determine the defect type reasoning error, and finally obtains the defect reasoning training error. This error reflects the consistency and accuracy of the network's results when reasoning about the same or corresponding defect type regions.

[0229] For each defect type inference binary [p_1, p_2], the computer system calculates its error according to certain rules. A simple method is to calculate the absolute value of the difference between the two probability values, that is, the error e=|p_1-p_2|. For example, if p_1=0.8 (indicating that the probability that the first defect type area is inferred to be defective is 0.8), p_2=0.6 (indicating that the probability that the second defect type area is inferred to be defective is 0.6), then the error corresponding to this defect type inference binary is e=|0.8-0.6|=0.2.

[0230] The computer system calculates the error of the defect type reasoning tuples corresponding to all defect type region tuples in this way, and then comprehensively processes these errors to obtain the defect reasoning training error. For example, the average calculation method can be used. Assume that there are n defect type region tuples, and the corresponding errors are , then the defect inference training error .

[0231] This method of calculating the defect reasoning training error can effectively evaluate the stability of the network's reasoning results when processing the same or corresponding defect type areas. If the defect reasoning training error is large, it means that the network has large fluctuations when reasoning about defects in these areas, and it may be necessary to further adjust the network parameters or optimize the network structure to improve the accuracy and stability of the network in chip wire bonding reliability detection.

[0232] In actual operation, in addition to the simple difference calculation method mentioned above, other more complex error calculation methods can be used according to specific application scenarios and requirements. For example, if the importance weights of different defect types are taken into account, different weights can be assigned to the errors corresponding to the binary groups of different defect types and then summed and averaged. In addition, if you want to measure the distribution of errors more accurately, you can also calculate statistical indicators such as the standard deviation of the errors, which can help to more comprehensively evaluate the defect reasoning performance of the network.

[0233] In summary, substeps S41-S43 are a process from constructing defect type region binary groups to determining the corresponding defect type reasoning binary groups, and finally calculating the defect reasoning training error. This process is of great significance for evaluating the accuracy and stability of defect reasoning of the preset deep neural network on the chip multi-source test training image set, and provides a key basis for network calibration.

[0234] As an implementation mode, step S43 determines the defect type reasoning error of the chip multi-source test training image set according to the defect type reasoning tuple corresponding to each defect type region tuple to obtain the defect reasoning training error, including:

[0235] Step S431: performing binary error determination on the defect type reasoning binary corresponding to each defect type region binary to obtain the reasoning error corresponding to each defect type region binary;

[0236] Step S432: averaging multiple reasoning errors corresponding to multiple defect type region binary groups to obtain a defect reasoning training error.

[0237] In step S431, the computer system accurately determines the inference error of the defect type reasoning tuple corresponding to each defect type region tuple. In the context of chip wire bonding reliability testing, the defect type reasoning tuple is composed of the results of network reasoning of the same defect type in different but related images (the first chip multi-source test training image and its corresponding second chip multi-source test training image).

[0238] For example, suppose that for a specific defect type region binary group, the binary group is about the cold solder defect on the chip lead. After the cold solder region (first defect type region) in the first chip multi-source test training image is subjected to defect reasoning by the preset deep neural network, the probability of the existence of the cold solder is p_1=0.7; the probability of the existence of the cold solder region (second defect type region) in the corresponding second chip multi-source test training image is obtained through network reasoning is p_2=0.5.

[0239] The computer system can use a variety of methods to determine the inference error of this defect type inference binary. A common method is to directly calculate the absolute value of the difference between the two probability values, that is, the inference error e=|p_1-p_2|=|0.7-0.5|=0.2. This method is simple and intuitive, and directly reflects the degree of difference in the network's inference results for the same defect in two related images.

[0240] However, in some cases, more factors may need to be considered to determine the inference error. For example, a weight coefficient w can be set according to the actual situation. If the detection accuracy of a defect type such as a cold solder joint is considered to be more critical based on past experience or prior knowledge, then w > 1 can be set. In this case, the inference error e=w×|p_1-p_2|. Assuming w=1.5, then e=1.5×0.2=0.3.

[0241] In addition, the inference error can also be determined from the perspective of probability distribution. For example, if the existence of a cold solder joint is regarded as a binomial distribution problem, for the probability p_1 of the first defect type area and the probability p_2 of the second defect type area, their distance under the binomial distribution can be calculated as the inference error. Let n be the number of samples (in this case, it can be assumed to be a fixed large value to represent the concept of multiple inspections), according to the probability mass function of the binomial distribution (where C(n, k) is the number of combinations), calculate the probability distribution corresponding to p_1 and p_2 , and then by calculating the distance between the two distributions (such as KL divergence, the formula is ) to determine the inference error. However, KL divergence requires Not 0. In practical applications, some processing may be required to avoid this problem.

[0242] The computer system determines the inference error of the defect type reasoning tuple corresponding to each defect type region tuple according to the selected method. This step helps to accurately quantify the difference in the network reasoning results of each tuple, providing a basis for the subsequent comprehensive evaluation of the network's defect reasoning accuracy.

[0243] In step S432, after determining the reasoning error corresponding to each defect type region binary group, the computer system combines these reasoning errors to obtain a defect reasoning training error that can reflect the overall situation.

[0244] Assume that the computer system has determined the inference errors corresponding to m defect type area pairs, which are The defect reasoning training error E can be obtained by the average calculation method, and the calculation formula is: .

[0245] For example, suppose there are 5 defect type region pairs, and the corresponding inference errors are Then the defect inference training error .

[0246] This defect reasoning training error E reflects the error of the preset deep neural network when reasoning about the defect type region binary in the chip multi-source test training image set. If the value of E is large, it means that the network's reasoning results fluctuate greatly when processing these related defect type regions, that is, the stability and accuracy of the network in defect reasoning needs to be improved.

[0247] When calculating the average error, the computer system needs to ensure that all the inference errors involved in the calculation are determined according to the same standard and method (such as the method used in the previous step S431). In addition, in order to more comprehensively evaluate the performance of the network, in addition to calculating the average error, statistical indicators such as the standard deviation of the error can also be calculated at the same time. , which reflects the discrete degree of reasoning error. If the standard deviation is large, it means that the reasoning errors between the binary pairs of different defect types are quite different, and the network may have poor reasoning effect on some types of defect area binary pairs, which also provides useful information for further optimizing the network.

[0248] To summarize, sub-steps S431-S432 determine the reasoning error corresponding to each defect type area binary group, and then perform an average calculation to obtain the defect reasoning training error. This process comprehensively and effectively evaluates the defect reasoning performance of the preset deep neural network on the chip multi-source test training image set, and provides a key quantitative basis for network calibration.

[0249] The present application embodiment provides a computer system, such as Figure 3 As shown, the computer system 100 includes: a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, such as through a bus 102. Optionally, the computer system 100 may also include a transceiver 104. It should be noted that in actual applications, the transceiver 104 is not limited to one, and the structure of the computer system 100 does not constitute a limitation on the embodiments of the present application.

[0250] An embodiment of the present application provides a computer system, and the computer system in the embodiment of the present application includes: one or more processors; a memory; one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more programs are executed by the processor, the above-mentioned chip lead bonding reliability detection method based on multi-source test fusion is implemented.

Claims

1. A chip wire bonding reliability detection method based on multi-source test fusion, characterized in that: The method comprises: Acquire chip multi-source test images; Loading the chip multi-source test image into a target wire bonding reliability detection network for image recognition to obtain a reliability detection result corresponding to the chip multi-source test image; Wherein, the target wire bonding reliability detection network is obtained by calibrating a preset deep neural network based on a first training error and a second training error, the first training error is one or more of an image descriptor training error and a defect reasoning training error, the image descriptor training error is used to indicate the error between image semantic descriptors corresponding to the same defect type area in the chip multi-source test training image set in the training image descriptor, the defect reasoning training error is used to indicate the error between wire bonding defect reasoning results corresponding to the same defect type area in the chip multi-source test training image set in the training image defect reasoning result, the training image descriptor and the training image defect reasoning result are obtained by loading the chip multi-source test training image set into the preset deep neural network for image recognition, and the second training error is determined based on the training image defect reasoning result and the prior defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set; The calibration process of the target wire bonding reliability detection network includes: Acquire the chip multi-source test training image set and a priori defect detection results corresponding to each chip multi-source test training image in the chip multi-source test training image set; Loading the chip multi-source test training image set into the preset deep neural network for image recognition, obtaining the training image descriptor corresponding to the chip multi-source test training image set, and the training image defect inference result corresponding to the chip multi-source test training image set; Determine the descriptor error of the chip multi-source test training image set according to the training image descriptor to obtain the image descriptor training error; According to the defect reasoning result of the training image, determining the reasoning error of the chip multi-source test training image set to obtain the defect reasoning training error; Determining the second training error according to the training image defect inference result and the priori defect detection result; The preset deep neural network is calibrated according to the first training error and the second training error to obtain the target wire bonding reliability detection network.

2. The method according to claim 1, characterized in that The step of determining the descriptor error of the chip multi-source test training image set based on the training image descriptor to obtain the image descriptor training error includes: Performing an image block operation on each chip multi-source test training image in the chip multi-source test training image set to obtain a plurality of training defect type regions corresponding to the chip multi-source test training image set; Screening the plurality of training defect type regions to obtain one or more screened defect type regions; Determine one or more relevant training images corresponding to each screening defect type area, wherein the one or more relevant training images corresponding to each screening defect type area are chip multi-source test training images containing each screening defect type area in the chip multi-source test training image set; According to one or more relevant training images corresponding to each of the screening defect type regions, determine one or more relevant descriptors corresponding to each of the screening defect type regions, and a screening descriptor set corresponding to the one or more screening defect type regions; wherein any relevant descriptor corresponding to each of the screening defect type regions is an image semantic descriptor corresponding to each of the screening defect type regions in any relevant training image corresponding to each of the screening defect type regions, and the screening descriptor set includes one or more relevant descriptors corresponding to the one or more screening defect type regions, respectively; According to the one or more relevant descriptors corresponding to each of the screening defect type regions, a relevant descriptor set corresponding to each relevant descriptor is determined; the relevant descriptor set corresponding to any relevant descriptor includes relevant descriptors other than any relevant descriptor in the one or more relevant descriptors corresponding to the target screening defect type region; the target screening defect type region is a screening defect type region corresponding to any relevant descriptor in the one or more screening defect type regions; According to the relevant descriptor set corresponding to each relevant descriptor and the screening descriptor set, a semantic consistency evaluation is performed on the one or more screening defect type regions to obtain the image descriptor training error.

3. The method according to claim 2, characterized in that The step of performing semantic consistency evaluation on the one or more screened defect type regions according to the relevant descriptor set corresponding to each relevant descriptor and the screened descriptor set to obtain the image descriptor training error includes: Walking through multiple related descriptors in the screening descriptor set one by one; Determine, according to the filtered descriptor set, a plurality of first related descriptors corresponding to the currently wandered related descriptor; wherein the plurality of first related descriptors are related descriptors in the filtered descriptor set other than the currently wandered related descriptor; Determine a target related descriptor set corresponding to the related descriptor currently wandered to in the related descriptor set corresponding to each related descriptor; the target related descriptor set includes a plurality of second related descriptors; Performing similarity evaluation on the currently wandered related descriptor and each first related descriptor to obtain a first evaluation result corresponding to each first related descriptor; the first evaluation result corresponding to each first related descriptor represents the similarity between each first related descriptor and the currently wandered related descriptor; Performing similarity evaluation on the currently wandered related descriptor and each second related descriptor to obtain a second evaluation result corresponding to each second related descriptor; the second evaluation result corresponding to each second related descriptor represents the similarity between each second related descriptor and the currently wandered related descriptor; Determine a third training error corresponding to the currently wandered related descriptor according to the second evaluation result corresponding to each of the second related descriptors and the first evaluation results respectively corresponding to the plurality of first related descriptors; After the walking is completed, multiple third training errors obtained during the walking are averaged to obtain the image descriptor training error.

4. The method according to claim 3, characterized in that The determining, based on the second evaluation result corresponding to each of the second related descriptors and the first evaluation results respectively corresponding to the plurality of first related descriptors, a third training error corresponding to the currently wandered related descriptor comprises: Determine the inference confidence corresponding to each second-related descriptor according to the second evaluation result corresponding to each second-related descriptor and the first evaluation results corresponding to each of the plurality of first-related descriptors; the inference confidence corresponding to each second-related descriptor is used to indicate the confidence that each second-related descriptor is correlated with the currently wandered related descriptor; Determine, according to the inference confidence corresponding to each second-related descriptor, a fourth training error corresponding to each second-related descriptor; the fourth training error is used to indicate the error between each second-related descriptor and the currently wandered related descriptor; The fourth training errors respectively corresponding to the multiple second related descriptors are added to obtain the third training error.

5. The method according to claim 2, characterized in that: The step of screening the plurality of training defect type regions to obtain one or more screened defect type regions includes: Performing commonality assessment on each training defect type region and a set image to obtain commonality assessment results corresponding to the plurality of training defect type regions; According to the commonality evaluation result, clearing the training defect type regions belonging to the set image among the multiple training defect type regions to obtain cleared training defect type regions; Redundancy elimination is performed on the cleaned training defect type regions to obtain the one or more screening defect type regions.

6. The method according to claim 5, characterized in that After performing redundancy elimination on the cleaned training defect type regions to obtain the one or more screening defect type regions, the method further includes: Determine one or more common screening defect type regions among the one or more screening defect type regions; the number of occurrences of any common screening defect type region among the one or more common screening defect type regions in the chip multi-source test training image set is greater than a number threshold; The step of determining one or more relevant training images corresponding to each defect type screening area includes: One or more relevant training images corresponding to each common screening defect type region are determined.

7. The method according to claim 1, characterized in that The obtaining of the chip multi-source test training image set and the priori defect detection result corresponding to each chip multi-source test training image in the chip multi-source test training image set includes: In a set image set, a plurality of first chip multi-source test training images are obtained; Cloning each first chip multi-source test training image to obtain a second chip multi-source test training image corresponding to each first chip multi-source test training image; Generate the chip multi-source test training image set according to the plurality of first chip multi-source test training images and the second chip multi-source test training images respectively corresponding to the plurality of first chip multi-source test training images; Determine the prior defect detection result corresponding to the chip multi-source test training image set according to the prior defect detection result corresponding to the set image set; The step of loading the chip multi-source test training image set into the preset deep neural network for image recognition, obtaining the training image descriptor corresponding to the chip multi-source test training image set, and the training image defect inference result corresponding to the chip multi-source test training image set, includes: Loading the chip multi-source test training image set into the training image semantic extraction component in the preset deep neural network to perform feature extraction to obtain the training image descriptor; The training image descriptor is loaded into the defect type inference component in the preset deep neural network to perform defect type inference to obtain the training image defect inference result.

8. The method according to claim 7, characterized in that The step of determining the inference error of the chip multi-source test training image set based on the defect inference result of the training image to obtain the defect inference training error includes: Determine a plurality of defect type region binary groups, wherein a defect type region binary group includes a first defect type region and a second defect type region, wherein the first defect type region is any defect type region in the first chip multi-source test training image, and the second defect type region is a defect type region corresponding to the first defect type region in a third chip multi-source test training image, and the third chip multi-source test training image is a second chip multi-source test training image corresponding to the first chip multi-source test training image; Determine a defect type reasoning tuple corresponding to each defect type region tuple, wherein the defect type reasoning tuple corresponding to each defect type region tuple includes a wire bonding defect reasoning result corresponding to a first defect type region in each defect type region tuple and a wire bonding defect reasoning result corresponding to a second defect type region in each defect type region tuple; Determine the binary error of the defect type reasoning binary corresponding to each defect type region binary to obtain the reasoning error corresponding to each defect type region binary; The multiple reasoning errors corresponding to the multiple defect type area binary groups are averaged to obtain the defect reasoning training error.

9. A computer system, characterized in that: include: one or more processors; Memory; one or more computer programs; The one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method according to any one of claims 1 to 8 is implemented.

Images