Methods, apparatus, terminal equipment and storage media for detecting defects in chip bonding processes

Abstract

This invention relates to the field of image recognition technology and provides a method and apparatus for detecting defects in chip bonding processes. The method includes: after the chip bonding points are formed, controlling a bonding actuator to apply a pre-set periodic vibration mechanical excitation to the chip, and collecting a set of image sequences of the bonding points under the periodic vibration mechanical excitation; based on the image sequence set, extracting a first feature vector and a second feature vector; fusing the first feature vector and the second feature vector and inputting them into a classification decision model to obtain the confidence scores corresponding to multiple defect categories output by the classification decision model; if any of the confidence scores is greater than a threshold, then outputting the defect anomaly information corresponding to the confidence score. The combination of dynamic features and static appearance features makes defect identification more comprehensive and detailed, significantly improving the accuracy of detection.

Description

Technical Field

[0001] This invention belongs to the technical field of image recognition, and particularly relates to a method, apparatus, terminal device, and storage medium for detecting defects in chip bonding processes. Background Technology

[0002] With the rapid development of microelectronics technology, the complexity of chip packaging processes is constantly increasing, especially the importance of chip bonding in integrated circuit manufacturing. Chip bonding is a crucial step in reliably connecting multiple chips together through solder joints, wires, or other connection methods, directly affecting chip performance, reliability, and production yield. However, due to various factors such as material properties, process parameters, and environmental conditions, various defects can easily occur during chip bonding, such as poor bonding, bubbles, and misalignment. These defects can not only lead to chip malfunction but may also trigger a chain reaction in subsequent processing, causing serious economic losses.

[0003] Traditional defect detection methods largely rely on manual visual inspection or simple automated inspection techniques, which often fall short of the high efficiency and precision requirements of modern production. Furthermore, limited by human resources and equipment, traditional methods suffer from significant shortcomings in defect recognition rate and classification accuracy, especially when detecting minute and hidden defects. Therefore, there is an urgent need for a novel, effective, and highly automated defect detection solution for chip bonding processes to improve the accuracy and reliability of defect detection. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for detecting defects in chip bonding processes, in order to solve the technical problem that traditional methods are significantly insufficient in terms of defect recognition rate and classification accuracy due to limitations in human resources and equipment.

[0005] A first aspect of this invention provides a method for detecting defects in a chip bonding process, the method comprising: After the chip bonding points are formed, the bonding actuator is controlled to apply a pre-set periodic vibration mechanical excitation to the chip, and a set of image sequences of the bonding points under the periodic vibration mechanical excitation is collected. Based on the image sequence set, a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point are extracted. The first feature vector and the second feature vector are fused and input into the classification decision model to obtain the confidence scores corresponding to each of the multiple defect categories output by the classification decision model. If any of the confidence levels is greater than the threshold, the defect anomaly information corresponding to the confidence level is output; wherein, the defect anomaly information includes the defect category, the location of the bonding point and the chip serial number corresponding to the confidence level.

[0006] Further, the step of extracting a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point based on the image sequence set includes: For the first frame image in the image sequence set, identify the region of interest corresponding to the bonding point and obtain its initial bounding box; Using the initial bounding box as a reference, subpixel-level image registration is performed on each frame of the image sequence set to obtain the bond point aligned region of interest sequence; At least one frame of image is selected from the aligned region of interest sequence as structural analysis input, and input into a pre-trained first convolutional neural network for feature encoding, and outputs a first feature vector representing the static appearance features of the bonding point; For each pixel position in the aligned region of interest sequence, extract its grayscale value change in the entire region of interest sequence to generate the corresponding pixel-level temporal signal; Based on the pixel-level time-series signal, a second feature vector characterizing the dynamic response properties of the bonding point is extracted.

[0007] Further, the step of performing sub-pixel-level image registration on each frame of the image sequence set based on the initial bounding box to obtain the sequence of regions of interest with precisely aligned bonding points includes: Based on the region within the first frame image of the initial bounding box, a set of feature points are extracted; Construct a Gaussian image pyramid between the first frame image and the current frame; wherein, the current frame refers to the frame image to be registered in the image sequence set; Starting from the highest layer of the Gaussian image pyramid, based on the position of the initial bounding box or the initial transformation parameters passed from the previous layer, the feature points are projected onto the current frame of the current layer, up to the original resolution layer. For each projected feature point, the optical flow method is used to calculate the local displacement with sub-pixel accuracy in its neighborhood through iterative optimization. Based on the local displacements calculated from all feature points, fit the affine transformation model of the current layer; The fitted affine transformation model is passed to the next layer as the initial transformation parameters for the next layer. Based on the affine transformation model finally optimized at the original resolution layer, the current frame is reverse-mapped and resampled to align it with the reference frame. Based on the initial bounding box, the precisely aligned region of interest of the bonding points is cropped from the aligned current frame.

[0008] Furthermore, the first convolutional neural network includes a multi-scale feature pyramid module, a spatial attention module, and a first feature encoding layer; The multi-scale feature pyramid module is used to extract and fuse multi-scale visual features from the input image by setting multiple dilated convolutional branches with different receptive fields in parallel. The parallel dilated convolutional branches of the multi-scale feature pyramid module include a first branch, a second branch, a third branch, and a global context branch. The first branch uses a 1x1 standard convolutional layer; the second branch uses a 3x3 convolutional layer with a dilation rate of 2; and the third branch uses a 3x3 convolutional layer with a dilation rate of 4. The global context branch includes a global average pooling layer, a 1x1 convolutional layer, and an upsampling layer. The outputs of each branch are concatenated along the channel dimension. The spatial attention module performs global max pooling and global average pooling on the input feature map, and concatenates the pooling results along the channel dimension. The spatial attention module passes the stitched feature map through a sub-network including convolutional layers and activation functions to generate a single-channel spatial weight map with the same spatial size as the input feature map; wherein, the spatial weight map is used to enhance the feature response of the bonding point region; The first feature encoding layer is used to pool and transform the single-channel spatial weight map after spatial attention enhancement to generate the first feature vector.

[0009] Further, the step of extracting a second feature vector characterizing the dynamic response properties of the bonding point based on the pixel-level temporal signal includes: For each pixel-level timing signal, calculate its dynamic response characteristics with respect to the periodic micromechanical excitation signal; The dynamic response features calculated from all pixel positions are rearranged according to their spatial positions to generate a multi-channel spatiotemporal feature map. The spatiotemporal feature map is input into the second neural network for feature encoding, and the output is a second feature vector representing the dynamic response characteristics of the bonding point.

[0010] Furthermore, the step of calculating the dynamic response characteristics between each pixel-level time-series signal and the periodic micromechanical excitation signal includes: Acquire the reference timing signal for the periodic micromechanical excitation applied to the chip; Based on the acquisition timestamp corresponding to the region of interest and the generation timestamp of the reference timing signal, the two are unified to the same time axis; Perform cross-correlation analysis between pixel-level time-series signals and reference time-series signals to find the time offset corresponding to the maximum cross-correlation value; The pixel-level timing signal is shifted according to the time offset to achieve time-domain synchronization. Fourier transforms are performed on the synchronized pixel-level timing signal and the reference timing signal respectively to obtain their complex spectrum values ​​at the fundamental frequency of periodic micromechanical excitation. The phase angle difference between the fundamental frequency complex spectrum value of the pixel-level timing signal and the fundamental frequency complex spectrum value of the reference timing signal is calculated and used as the dynamic response feature of the pixel position.

[0011] Furthermore, the second neural network includes a spatiotemporal feature decoupling coding module, a deep spatiotemporal feature extraction module, a channel attention module, and a second feature coding layer; The spatiotemporal feature decoupling encoding module has parallel spatial relationship paths and channel relationship paths. The spatial relationship path is used to encode the spatial distribution pattern of dynamic features, and the channel relationship path is used to encode the correlation pattern between different dynamic features. The spatial relationship path consists of at least one 2D convolutional layer, and the channel relationship path consists of at least one 1x1 convolutional layer. The outputs of the spatial relationship path and the channel relationship path are fused by element-wise addition or channel concatenation. The deep spatiotemporal feature extraction module consists of multiple stacked separate convolutional blocks, each of which includes a deep convolutional layer for spatial filtering and a pointwise convolutional layer for channel fusion. The channel attention module is used to perform global average pooling on the input feature map to obtain a scalar description for each channel; The channel attention module is used to input a scalar description sequence into a neural network containing two fully connected layers and output the weight coefficients of each channel; wherein, the first fully connected layer is followed by a ReLU activation function and the second fully connected layer is followed by a Sigmoid activation function. The second feature encoding layer is used to pool and transform the features after channel attention weighting to generate the second feature vector.

[0012] A second aspect of the present invention provides a chip bonding process defect detection device, comprising: The acquisition unit is used to control the bonding actuator to apply a pre-set periodic vibration mechanical excitation to the chip after the chip bonding points are formed, and to acquire a set of image sequences of the bonding points under the periodic vibration mechanical excitation. The extraction unit is used to extract a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point based on the image sequence set. A classification unit is used to fuse the first feature vector and the second feature vector and input them into a classification decision model to obtain the confidence scores of each of the multiple defect categories output by the classification decision model. The judgment unit is configured to output defect anomaly information corresponding to any confidence level if any of the confidence levels is greater than a threshold; wherein the defect anomaly information includes the defect category, bonding point location and chip serial number corresponding to the confidence level.

[0013] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the chip bonding process defect detection method described in the first aspect above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the chip bonding process defect detection method described in the first aspect.

[0015] The beneficial effects of this invention compared to existing technologies are as follows: by applying a pre-set periodic vibration mechanical excitation to the chip bonding point, the dynamic response characteristics of the bonding point can be effectively captured. This combination of dynamic and static appearance features makes defect identification more comprehensive and detailed, significantly improving detection accuracy. This method extracts feature vectors of two different dimensions—static and dynamic features. By fusing these two types of features, the classification decision model can more effectively learn and identify various defect categories. Compared to single-feature detection methods, the feature fusion approach significantly enhances the model's classification ability, making the identification of complex defects clearer. This method, through automated image acquisition and processing, can rapidly detect defects after chip bonding, providing immediate feedback. This real-time monitoring capability helps to promptly identify and adjust problems, thereby reducing production losses caused by defects and improving the overall efficiency of the production line. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic flowchart of a chip bonding process defect detection method provided by the present invention is shown. Figure 2 This diagram illustrates a chip bonding process defect detection device according to an embodiment of the present invention. Figure 3 A schematic diagram of a terminal device provided in an embodiment of the present invention is shown. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0019] This invention provides a method and apparatus for detecting defects in chip bonding processes, which addresses the technical problem that traditional methods are significantly insufficient in defect identification rate and classification accuracy due to limitations in human resources and equipment.

[0020] First, this invention provides a method for detecting defects in chip bonding processes. Please refer to [link / reference]. Figure 1 , Figure 1 A schematic flowchart of a chip bonding process defect detection method provided by the present invention is shown. Figure 1 As shown, the chip bonding process defect detection method may include the following steps: Step 101: After the chip bonding points are formed, control the bonding actuator to apply a pre-set periodic vibration mechanical excitation to the chip, and collect a set of image sequences of the bonding points under the periodic vibration mechanical excitation. By utilizing existing bonding actuators as the excitation source, there is no need to introduce additional complex excitation devices. The frequency, amplitude, and waveform of the periodic vibration mechanical excitation can be optimized for different chip types or expected defects, which is the basis for the method's repeatability and standardization.

[0021] The collected images are a series of images under forced vibration, rather than a single image. This provides a data foundation for subsequent analysis of dynamic characteristics. The images may come from high-speed cameras, microscopic vision systems, etc.

[0022] Step 102: Based on the image sequence set, extract a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point; The raw image data is extracted into two types of feature vectors: a first feature vector and a second feature vector, each with clear physical / engineering significance. The static and dynamic features complement each other, significantly enhancing the dimensionality and depth of detection, particularly in the detection of potential and hidden defects.

[0023] Specifically, step 102 includes steps 1021 to 1025: Step 1021: Identify the region of interest corresponding to the bonding point in the first frame image of the image sequence set, and obtain its initial bounding box; Using the first frame as a reference ensures consistency in the processing.

[0024] Identifying regions of interest can be accomplished using traditional image processing algorithms (such as thresholding and contour detection) or a lightweight initial detection model.

[0025] Obtaining the initial bounding box provides spatial anchor points for subsequent fine-grained tracking and alignment. This step ensures that the system can automatically locate each bonding point to be detected.

[0026] Step 1022: Using the initial bounding box as a reference, perform subpixel-level image registration on each frame of the image sequence set to obtain the bond point aligned region of interest sequence; Image registration is crucial for eliminating interference caused by minor camera shake, stage drift, or overall rigid body displacement due to excitation. This is a key preprocessing step for extracting a pure dynamic response.

[0027] Subpixel-level registration ensures that background displacement is suppressed to the maximum extent in the analysis of minute vibrations, so that the changes in the extracted gray values ​​truly reflect the deformation or vibration of the bonding points themselves, rather than the overall movement.

[0028] This yields a spatially aligned sequence of regions of interest, where each pixel in the sequence corresponds to the same physical location on the bonding point.

[0029] Specifically, step 1022 includes steps A1 to A8: Step A1: Based on the region of the initial bounding box within the first frame image, extract a set of feature points; Feature points are extracted only within the initial bounding box (ROI), focusing on the bonding points themselves, eliminating background interference, and improving computational efficiency and the specificity of registration.

[0030] Feature points, corner points, or spots that exhibit significant gradient changes and are easy to track, such as SIFT, SURF, ORB feature points, or Shi-Tomasi corner points. These points are more stable and reliable in subsequent optical flow tracking.

[0031] Step A2: Construct the Gaussian image pyramid of the first frame image and the current frame; wherein, the current frame refers to the frame image to be registered in the image sequence set; The pyramid structure is the infrastructure for implementing a coarse-to-fine strategy. By repeatedly applying Gaussian blur and downsampling to the image (layer 0), a series of images with resolution halved layer by layer are generated, forming a pyramid structure. The top image has the lowest resolution but the largest motion scale.

[0032] Large displacements (such as overall movement caused by mechanical vibration) can be captured quickly at the top level (low resolution), avoiding getting stuck in local optima or calculation failure when calculating directly at the original resolution.

[0033] Step A3: Starting from the highest layer of the Gaussian image pyramid, based on the position of the initial bounding box or the initial transformation parameters passed from the previous layer, project the feature points to the current frame of the current layer, until the original resolution layer; This is the source of the initial guesses for the iteration, and there are two scenarios: In the highest layer (first iteration), there is no previous layer, so the initial bounding box position is used. The original coordinates of the feature points in the first frame (reference frame) are simply divided by the downsampling coefficient of the top layer of the pyramid (e.g., if each layer of the pyramid is scaled down by a factor of 2, and there are 3 layers, then the top layer coefficient is 4) to obtain the approximate positions of these feature points in the top layer reference image. Simultaneously, it is assumed that the initial positions of the feature points in the top layer image of the current frame (the frame to be registered) are the same as those in the top layer of the reference frame. This is a zero-displacement initial assumption. Since the displacement of the top layer image has been scaled down, this assumption is usually not far from the true positions.

[0034] "Use the initial transformation parameters passed from the previous layer in non-highest layers (subsequent iterations)." This is the essence of iterative optimization. The previous layer (a coarser layer) has already computed an affine transformation model that describes the overall motion from the reference frame to the current frame at that coarse scale. This model is scaled (because the resolution changes) and used as the initial transformation estimate for the current finer layer. This provides a very accurate starting point for the computation of the current layer.

[0035] Based on the initial guess, the estimated position of the feature point in the current frame of the current layer is calculated. At the highest layer: Projected position = Coordinates of the feature point at the top layer of the reference frame (i.e., assuming no displacement). At lower layers: Projected position = Coordinates of the feature point in the current layer of the reference frame, applied with the (scaled) affine transformation model of the previous layer.

[0036] This provides a starting point for the optical flow search performed on the current layer. The optical flow method performs a fine search within a small neighborhood (e.g., ±3 pixels) at each projected location to find the best match. This projection step ensures that the optical flow search starts from a point very close to the real location, thus guaranteeing a fast and accurate search.

[0037] Step A4: For each projected feature point, the optical flow method is used to iteratively optimize and calculate the local displacement with sub-pixel accuracy in its neighborhood; The algorithm starts at the top of the pyramid. At the top, the position of the initial bounding box is used as the initial guessed position of the feature points. Then, in the current layer, for each feature point, an optical flow method (such as the Lucas-Kanade algorithm) is used to compute its subpixel displacement vector from the reference frame to the current frame.

[0038] Optical flow assumes that the pixel grayscale of the same object remains constant between adjacent frames, and calculates pixel motion by solving a constant brightness equation. Its iterative optimization process can achieve sub-pixel accuracy, which is key to achieving high-precision registration.

[0039] After completing the displacement calculation for the current layer, these displacement results (usually in the form of a transformation model, see the next step) are scaled up by one time and used as the initial positions for feature point tracking in the next layer (higher resolution). This process is repeated layer by layer until the original resolution layer is reached. This ensures that at each layer, the optical flow method only needs to calculate a very small displacement correction, thus balancing speed and accuracy.

[0040] Step A5: Based on the local displacements calculated from all feature points, fit the affine transformation model of the current layer; In chip bonding inspection scenarios, the object under test (chip or substrate) may undergo rigid overall motion (translation, rotation) under excitation, or it may introduce non-rigid but linear deformation (scaling, shearing) due to mounting flatness, perspective, or slight deformation. Affine transformation is the simplest linear geometric transformation that can describe these phenomena. The affine transformation model is calculated using the RANSAC algorithm, which is a traditional technique and will not be elaborated upon here.

[0041] Step A6: Pass the fitted affine transformation model to the next layer as the initial transformation parameters for the next layer; The optical flow method in step A4 yields the local displacement of each feature point, which may contain noise and outliers. This step uses robust fitting (such as the RANSAC algorithm) to obtain a global affine transformation model (including translation, rotation, scaling, and shearing), which can filter out erroneous matching points and obtain a smooth mathematical model describing the overall motion of the entire ROI region.

[0042] Assuming that the motion of the bonding point between two frames is mainly caused by rigid body displacement and small changes in viewpoint, affine transformation is a good approximation and is more general than the simple translation model.

[0043] The parameters of the affine transformation model fitted in this layer are passed to the next layer as the initial global motion estimate for optical flow tracking in the next layer, thus realizing cross-layer information inheritance and refinement.

[0044] Step A7: Based on the affine transformation model finally optimized at the original resolution layer, perform reverse mapping and resampling on the current frame to align it with the reference frame; Step A8: Based on the initial bounding box, crop out the precisely aligned region of interest of the bonding points from the aligned current frame.

[0045] Using the optimal affine transformation matrix obtained from the original layer, the entire current frame image is reverse-mapped (to avoid holes) and resampled (e.g., by bilinear interpolation) to generate a new image that is spatially perfectly aligned with the first frame (reference frame).

[0046] Since the transformation is global, after alignment, the precisely corresponding ROIs can be cropped from the aligned current frame using the initial bounding box coordinates of the first frame. This results in a pixel-level aligned ROI sequence, providing clean data for subsequent static and dynamic analysis.

[0047] In the embodiments corresponding to steps A1 to A8, by using a layer-by-layer pyramid processing method combined with optical flow optimization of feature points, sub-pixel level accuracy can be achieved while ensuring computational efficiency. This method not only improves the stability of registration but also ensures that the final generated region of interest accurately reflects the actual situation of the area to be inspected, providing a solid foundation for subsequent defect detection.

[0048] Step 1023: Select at least one frame image from the aligned region of interest sequence as structural analysis input, input it into a pre-trained first convolutional neural network for feature encoding, and output a first feature vector representing the static appearance features of the bonding point; Because the sequences are aligned, any frame (or the average frame, the clearest frame) can represent its static structure. A pre-trained first convolutional neural network is used. CNNs excel at extracting hierarchical abstract features of images, such as edges, textures, and shape patterns, which are perfect descriptors of static appearance features (such as solder ball shape, lead outline, and uniformity of colloid distribution).

[0049] This step upgrades traditional, manually designed static features (such as area and roundness) to richer and more discriminative deep feature vectors that are automatically learned by deep networks, significantly improving the detection capability of static defects (such as deformation, contamination, and bridging).

[0050] Specifically, the first convolutional neural network includes a multi-scale feature pyramid module, a spatial attention module, and a first feature encoding layer; The multi-scale feature pyramid module is used to extract and fuse multi-scale visual features from the input image by setting multiple dilated convolutional branches with different receptive fields in parallel. The parallel dilated convolutional branches of the multi-scale feature pyramid module include a first branch, a second branch, a third branch, and a global context branch. The first branch uses a 1x1 standard convolutional layer; the second branch uses a 3x3 convolutional layer with a dilation rate of 2; and the third branch uses a 3x3 convolutional layer with a dilation rate of 4. The global context branch includes a global average pooling layer, a 1x1 convolutional layer, and an upsampling layer. The outputs of each branch are concatenated along the channel dimension. The spatial attention module performs global max pooling and global average pooling on the input feature map, and concatenates the pooling results along the channel dimension. The spatial attention module passes the stitched feature map through a sub-network including convolutional layers and activation functions to generate a single-channel spatial weight map with the same spatial size as the input feature map; wherein, the spatial weight map is used to enhance the feature response of the bonding point region; The first feature encoding layer is used to pool and transform the single-channel spatial weight map after spatial attention enhancement to generate the first feature vector.

[0051] The multi-scale feature pyramid module (which addresses the scale and context awareness problem) extracts and fuses multi-scale visual features from the input image by setting up multiple dilated convolution branches with different receptive fields in parallel.

[0052] Defects at bonding points can manifest as features at different scales (e.g., tiny cracks require a small receptive field to capture details, while overall deformation or relative relationship with surrounding structures requires a large receptive field to capture context). Parallel multi-branch structures can capture this information simultaneously.

[0053] The application of dilated convolution is a key technical approach in this module. Dilated convolution can exponentially expand the receptive field of the convolution kernel without increasing parameters or computational cost.

[0054] The first branch (1x1 convolution) has the smallest receptive field, used to capture the most local details and perform channel transformations. The second branch (3x3 convolution, dilation 2) has a medium receptive field, perceiving patterns over a slightly larger area while maintaining resolution. The third branch (3x3 convolution, dilation 4) has a large receptive field, capable of integrating contextual information from a larger region, helping to understand the overall relationship between bonding points and surrounding pads and leads. The global context branch uses global average pooling to obtain global statistical information (global context) of the entire feature map.

[0055] A 1x1 convolutional layer combined with upsampling transforms the global information and restores it to the same spatial size as the other branches. The outputs of the four branches are concatenated along the channel dimension to form a rich feature map that simultaneously contains local details, mid-level structure, large-scale context, and global statistical information. This design makes the network extremely robust to scale variations in defects.

[0056] The spatial attention module (which addresses the region focusing problem) performs global max pooling and global average pooling on the input feature map to generate a single-channel spatial weight map with the same spatial size as the input feature map; wherein, the spatial weight map is used to enhance the feature response of the bonding point region.

[0057] In bond point images, defective regions (such as cracks and voids) are typically sparse and localized, while most areas are normal. The role of spatial attention mechanisms is to teach the network to focus on spatial locations that are more likely to contain anomalous information, while suppressing responses to irrelevant background or normal regions.

[0058] Dual-channel pooling uses both global max pooling (which highlights the most salient feature points) and global average pooling (which reflects the overall feature distribution) to obtain two different global spatial contextual cues.

[0059] The concatenation and convolution methods combine the results of two pooling operations and then pass them through a small sub-network (typically containing convolutional layers and a sigmoid activation function) to generate a spatial weight map. This sub-network learns to determine the importance of each spatial location based on the global context.

[0060] The generated weight map has values ​​between 0 and 1, and it is multiplied point-by-point with the original multi-scale feature map. Important regions (such as suspected defects and bonding point edges) have weights close to 1, and the features are enhanced; unimportant regions (such as uniform backgrounds) have weights close to 0, and the features are suppressed. This greatly improves the feature representation efficiency and the signal-to-noise ratio of defects.

[0061] The first feature encoding layer (which generates the final feature vector) is used to pool and transform the single-channel spatial weight map after spatial attention enhancement to generate the first feature vector.

[0062] The received feature map is spatially attention modulated, rich in information and focused on key regions.

[0063] The two-dimensional spatial feature map is compressed into a one-dimensional feature vector through global average pooling (or global max pooling). This step aggregates information from all spatial locations, but thanks to the preceding spatial attention, this aggregation process is weighted, with key regions contributing more. One or more fully connected layers or bottleneck layers further perform nonlinear transformations and dimensionality reduction on the feature vector, ultimately outputting a fixed-length, highly abstract first feature vector for use by subsequent classifiers.

[0064] Step 1024: For each pixel position in the aligned region of interest sequence, extract its grayscale value change in the entire region of interest sequence to generate the corresponding pixel-level temporal signal; This is a crucial step in converting spatial image sequences into spatiotemporal analysis data. After precise registration, the change in grayscale value of each pixel over time can be considered mainly caused by minute deformations due to vibration, changes in light reflection, etc., which are directly related to its dynamic mechanical response.

[0065] The pixel-level time-series signal generates a time-series curve for each physical point, which essentially records the vibration trajectory of that point under periodic excitation.

[0066] Among them, the pixel-level time-series signal is the gray value corresponding to the same pixel position in each region of interest sequence.

[0067] Step 1025: Based on the pixel-level time-series signal, extract the second feature vector characterizing the dynamic response properties of the bonding point.

[0068] This step involves feature dimensionality reduction and engineering representation of the massive temporal signals (one signal per pixel) generated in the previous step.

[0069] These features, statistically or calculated from pixel-level signals, together constitute the second feature vector describing the dynamic stiffness, damping, and adhesion state of the entire bonding point.

[0070] In the embodiments corresponding to steps 1021 to 1025, static and dynamic features are effectively extracted through precise image processing and feature extraction techniques, providing an important information foundation for subsequent defect detection. This method combines image analysis and machine learning techniques, which can improve the accuracy and efficiency of defect detection.

[0071] Specifically, step 1025 includes steps B1 to B3: Step B1: For each pixel-level timing signal, calculate its dynamic response characteristics with respect to the periodic micromechanical excitation signal; This step is crucial for establishing the quantization relationship between excitation (input) and response (output). By calculating the correlation characteristics between the temporal signal of each pixel and the known excitation signal, the local mechanical properties of that point can be directly reflected.

[0072] Specifically, step B1 includes steps B11 to B16: Step B11: Obtain the reference timing signal for the periodic micromechanical excitation applied to the chip; The reference signal serves as the baseline for the entire phase analysis. It represents the pure excitation applied to the chip and is typically recorded by a signal generator or sensor (such as an accelerometer) that controls the bonding actuator.

[0073] Step B12: Based on the acquisition timestamp corresponding to the region of interest and the generation timestamp of the reference timing signal, unify the two onto the same time axis; Timestamp alignment is a crucial yet easily overlooked preprocessing step. Image acquisition systems (cameras) and excitation generation systems (controllers) are typically two independent clock sources. Directly using the acquisition frame time to correspond to the excitation time introduces errors. By unifying both onto the same high-precision timeline using precise timestamps (e.g., based on PTP or hardware triggering), the accuracy of subsequent cross-correlation analysis and phase calculations is ensured. This step lays the foundation for high-precision measurements.

[0074] Step B13: Perform cross-correlation analysis on the pixel-level time-series signal and the reference time-series signal to find the time offset corresponding to the maximum cross-correlation value; Even with a unified timeline, due to factors such as signal transmission path, sensor response, and image exposure delay, there may still be a fixed, unknown time delay between the pixel signal and the reference signal.

[0075] Cross-correlation analysis is a classic and robust method for estimating the time delay between two signals. The optimal time offset is obtained by calculating the cross-correlation function of the two signals and finding the time shift corresponding to its peak value.

[0076] This step achieves precise temporal alignment between the response signal of each pixel and the excitation reference signal. This eliminates the phase error caused by system delay, ensuring that the phase difference calculated subsequently purely reflects the dynamic characteristics of the bonding point material / structure itself, rather than system error.

[0077] Step B14: Shift the pixel-level timing signal according to the time offset to complete time-domain synchronization; Step B15: Perform Fourier transform on the synchronized pixel-level timing signal and the reference timing signal respectively to obtain their complex spectrum values ​​at the fundamental frequency of periodic micromechanical excitation. Because the excitation is periodic, the energy of the response signal is mainly concentrated at its fundamental frequency (and harmonics). The Fourier transform converts the signal from the time domain to the frequency domain and extracts the complex spectral values ​​at the fundamental frequency of the excitation. This includes the amplitude A and phase φ of the frequency component.

[0078] Step B16: Calculate the phase angle difference between the fundamental frequency complex spectrum value of the pixel-level timing signal and the fundamental frequency complex spectrum value of the reference timing signal, and use it as the dynamic response feature of the pixel position.

[0079] For a linear vibration system, the phase difference between the response signal and the excitation signal directly reflects the system's damping characteristics and resonance state.

[0080] When the phase difference is close to 0 degrees, the system is in the stiffness control region (response and excitation are in phase). When the phase difference is close to 90 degrees, the system is near the resonance point. When the phase difference is close to 180 degrees, the system is in the mass control region (response and excitation are out of phase).

[0081] Local phase difference is an extremely sensitive indicator. A bond with microcracks or poor adhesion will experience a decrease in local stiffness and an alteration in damping characteristics. This leads to a detectable change in the phase difference between the vibration of pixels in that region and that of healthy regions, as well as with the excitation signal. Therefore, using the fundamental frequency phase difference of each pixel as its dynamic response characteristic has clear physical significance and high defect sensitivity.

[0082] In the embodiments corresponding to steps B11 to B16, dynamic response features related to periodic micromechanical excitation signals are systematically extracted through refined signal processing and analysis, providing important data support for chip defect detection.

[0083] Step B2: Rearrange the dynamic response features calculated from all pixel positions according to their spatial locations to generate a multi-channel spatiotemporal feature map; This is a bridging step from signal processing to image-based deep learning analysis. Instead of simply concatenating the features of thousands of pixels into a long vector (which would completely lose spatial relationships), it repositions the multidimensional feature vector calculated for each pixel onto its corresponding position in a two-dimensional grid, according to the pixel's two-dimensional spatial coordinates in the original image.

[0084] Assuming that K dynamic features are extracted from each pixel, the rearranged feature map will result in a multi-channel feature map of size [H, W, K]. Here, H and W are the height and width of the region of interest (ROI), and K is the number of channels (i.e., the number of dynamic features).

[0085] This spatiotemporal feature map is a powerful intermediate representation. Its spatial dimension preserves the location and morphological information of defects (e.g., a crack would appear as an anomaly in the resonant frequency of pixels along a line). Its channel dimension encodes multiple dynamic physical properties at each location. This provides a perfect input format for subsequent analysis using convolutional neural networks.

[0086] Step B3: Input the spatiotemporal feature map into the second neural network for feature encoding, and output a second feature vector representing the dynamic response characteristics of the bonding point.

[0087] The second neural network here is best suited to use a convolutional neural network. CNNs are naturally good at processing grid data (such as images) with spatial local correlation and translation invariance, and the spatiotemporal feature map that was just generated is exactly this kind of data.

[0088] The CNN (second neural network) learns spatial patterns and automatically learns how anomalous dynamic features are distributed in space (e.g., whether they are local point-like anomalous features, linear anomalous features, or regional anomalous features).

[0089] The CNN (second neural network) abstracts and fuses features, combining low-level dynamic features (such as frequency and amplitude) into high-level, discriminative spatiotemporal pattern descriptors through multi-layer convolution and pooling.

[0090] Finally, through a fully connected layer or a global pooling layer, the two-dimensional spatiotemporal feature map is encoded into a fixed-length, one-dimensional second feature vector. This vector integrates the dynamic response information of all pixels in the entire bonding point region, and is a highly abstract and powerful dynamic representation.

[0091] Specifically, the second neural network includes a spatiotemporal feature decoupling coding module, a deep spatiotemporal feature extraction module, a channel attention module, and a second feature coding layer; The spatiotemporal feature decoupling encoding module has parallel spatial relationship paths and channel relationship paths. The spatial relationship path is used to encode the spatial distribution pattern of dynamic features, and the channel relationship path is used to encode the correlation pattern between different dynamic features. The spatial relationship path consists of at least one 2D convolutional layer, and the channel relationship path consists of at least one 1x1 convolutional layer. The outputs of the spatial relationship path and the channel relationship path are fused by element-wise addition or channel concatenation. The deep spatiotemporal feature extraction module consists of multiple stacked separate convolutional blocks, each of which includes a deep convolutional layer for spatial filtering and a pointwise convolutional layer for channel fusion. The channel attention module is used to perform global average pooling on the input feature map to obtain a scalar description for each channel; The channel attention module is used to input a scalar description sequence into a neural network containing two fully connected layers and output the weight coefficients of each channel; wherein, the first fully connected layer is followed by a ReLU activation function and the second fully connected layer is followed by a Sigmoid activation function. The second feature encoding layer is used to pool and transform the features after channel attention weighting to generate the second feature vector.

[0092] The spatiotemporal feature decoupling encoding module has parallel spatial relationship paths and channel relationship paths... The spatial relationship path consists of at least one 2D convolutional layer; the channel relationship path consists of at least one 1x1 convolutional layer. The output is fused by element-wise addition or channel concatenation.

[0093] The input second feature map is a tensor of [H, W, C], where C channels represent C different dynamic features (such as phase difference, dominant frequency amplitude, etc.). These features have spatial distribution patterns (such as abnormal features at cracks connecting into lines), and also have correlation patterns between channels (such as phase anomalies may be accompanied by amplitude attenuation).

[0094] Spatial relational pathways (2D convolutions) focus on learning the correlation patterns of the same feature across different spatial locations. For example, a 3x3 2D convolution kernel can learn local spatial patterns such as a phase anomaly in the center pixel often being accompanied by anomalies in the pixels above, below, to the left, and to the right. This is crucial for detecting defects with spatial continuity, such as cracks and scratches.

[0095] The channel relationship path (1x1 convolution) is a classic operation for channel fusion. This path focuses on learning the combination and weight relationships between different dynamic feature channels at the same spatial location. For example, it may learn a specific combination pattern of phase difference channels and vibration amplitude channels, which is a strong indicator of poor adhesion.

[0096] By fusing (adding or concatenating) the feature maps processed by the two pathways, an enhanced feature map is obtained that simultaneously deepens the spatial context and optimizes the channel combination. This parallel decoupling structure is more efficient and targeted than simply stacking standard convolutional layers.

[0097] The deep spatiotemporal feature extraction module consists of multiple stacked separate convolutional blocks, each of which includes a deep convolutional layer for spatial filtering and a pointwise convolutional layer for channel fusion.

[0098] Separate convolution is the core idea of ​​modern efficient CNN architectures (such as MobileNet and Xception). Depthwise convolution (performing spatial convolution on each input channel separately) is responsible for further extracting spatial features; pointwise convolution (1x1 convolution) is responsible for cross-channel feature fusion and dimensionality transformation.

[0099] This structure significantly reduces the number of parameters and computational load while maintaining strong feature extraction capabilities. For scenarios like industrial inspection, which may require deployment on edge devices or real-time processing, efficiency is crucial. Stacking multiple such blocks allows for the construction of deep networks that learn more complex spatiotemporal feature hierarchies.

[0100] The channel attention module (the second core innovation) is used to perform global average pooling on the input feature map. It outputs the weight coefficients of each channel. After the preceding multi-layer processing, the feature map may contain dozens or even hundreds of channels. Not all channels contribute equally to the final defect category determination. The channel attention mechanism (similar to the idea of ​​SENet) allows the network to automatically evaluate and emphasize information-rich channels while suppressing less important ones.

[0101] Global average pooling compresses the HxW feature map of each channel into a scalar, which represents the global response strength of the channel's features.

[0102] Two fully connected layers form a simple bottleneck neural network that learns the nonlinear interactions between channels. ReLU provides the nonlinearity, and Sigmoid normalizes the output to the range (0,1) as weights.

[0103] Ultimately, each channel of the feature map is multiplied by a corresponding weighting factor. Important feature channels (e.g., a specific frequency response channel that clearly distinguishes between cracks and contamination) are amplified, while noisy or unimportant channels are attenuated. This achieves adaptive calibration at the feature channel level, improving the discriminative power of the features.

[0104] The second feature encoding layer is used to pool and transform the channel attention-weighted features to generate the second feature vector. This layer is the final part of the network. Global average pooling compresses the spatial dimension HxW, resulting in a C-dimensional vector (C being the number of channels). This vector aggregates information from all spatial locations. One or more fully connected layers further perform non-linear transformations and dimensionality reduction on this vector, ultimately outputting a fixed-length, highly abstract second feature vector, which is then fused with the first feature vector.

[0105] In the embodiments corresponding to steps B1 to B3, through layer-by-layer processing and encoding, an effective feature representation for defect detection is ultimately formed. The logic and systematic nature of this process ensures that the dynamic behavior of the chip under micromechanical stimulation can be accurately captured, thereby providing strong support for defect detection.

[0106] Step 103: Fuse the first feature vector and the second feature vector, and input them into the classification decision model to obtain the confidence scores corresponding to each of the multiple defect categories output by the classification decision model; A classification decision model is a generalized intelligent discrimination system, which can be a traditional machine learning model (such as SVM, random forest) or a deep learning model (such as CNN, LSTM or a combination thereof). After training, the model can establish a complex mapping relationship between static and dynamic feature combinations and defect categories.

[0107] The output is the confidence score for each of the multiple defect categories, which is the standard output format for a multi-classification problem (e.g., [No defects: 0.05, Offset: 0.80, Crack: 0.15]).

[0108] Step 104: If any of the confidence levels is greater than the threshold, output the defect anomaly information corresponding to the confidence level; wherein, the defect anomaly information includes the defect category, bonding point location and chip serial number corresponding to the confidence level.

[0109] A threshold (e.g., 0.67) is set for comparison to achieve automatic interpretation and reduce human intervention. The threshold can be adjusted according to the stringency of quality requirements.

[0110] In the embodiments corresponding to steps 101 to 104, by applying a pre-set periodic vibration mechanical excitation to the chip bonding point, the dynamic response characteristics of the bonding point can be effectively captured. This combination of dynamic and static appearance features makes defect identification more comprehensive and detailed, significantly improving detection accuracy. This method extracts feature vectors of two different dimensions—static and dynamic features. By fusing these two types of features, the classification decision model can more effectively learn and identify various defect categories. Compared to single-feature detection methods, feature fusion significantly enhances the model's classification ability, making the identification of complex defects clearer. This method, through automated image acquisition and processing, can quickly detect defects after chip bonding, providing immediate feedback. This real-time monitoring capability helps to promptly identify and adjust problems, thereby reducing production losses caused by defects and improving the overall efficiency of the production line.

[0111] like Figure 2 This invention provides a chip bonding process defect detection device. Please refer to [link / reference]. Figure 2 , Figure 2 A schematic diagram of a chip bonding process defect detection device provided by the present invention is shown, as follows: Figure 2 The chip bonding process defect detection device shown includes: The acquisition unit 21 is used to acquire the historical communication traffic corresponding to the historical sub-periods of multiple historical periods; wherein, the historical period includes 24 hours, and the historical sub-period includes 1 hour; Prediction unit 22 is used to predict the current traffic range corresponding to the communication traffic of the next sub-cycle based on the historical communication traffic. The matching unit 23 is used to match the power mode corresponding to the current traffic range and control the communication device based on the power mode; wherein the power mode includes sleep, power saving, normal and high performance.

[0112] This invention provides a chip bonding process defect detection device that effectively captures the dynamic response characteristics of the bonding points by applying pre-set periodic vibration mechanical excitation. This combination of dynamic and static appearance features makes defect identification more comprehensive and detailed, significantly improving detection accuracy. This method extracts two feature vectors of different dimensions—static and dynamic features. By fusing these two types of features, the classification decision model can more effectively learn and identify various defect categories. Compared to single-feature detection methods, feature fusion significantly enhances the model's classification ability, making the identification of complex defects clearer. Through automated image acquisition and processing, this method can rapidly detect defects after chip bonding, providing immediate feedback. This real-time monitoring capability helps to promptly identify and adjust problems, thereby reducing production losses caused by defects and improving the overall efficiency of the production line.

[0113] Figure 3 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 3 As shown, a terminal device 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a chip bonding process defect detection program. When the processor 30 executes the computer program 32, it implements the steps in the various embodiments of the chip bonding process defect detection method described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each unit in the above-described device embodiments, for example... Figure 2 The function of the unit shown.

[0114] For example, the computer program 32 can be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the terminal device 3. For example, the specific functions of each unit of the computer program 32 can be divided as follows: An acquisition unit is used to acquire historical communication traffic corresponding to each of multiple historical periods; wherein, the historical period includes 24 hours, and the historical sub-period includes 1 hour; The prediction unit is used to predict the current traffic range corresponding to the communication traffic in the next sub-cycle based on the historical communication traffic. A matching unit is used to match the power mode corresponding to the current traffic range and control the communication device based on the power mode; wherein the power mode includes sleep, power saving, normal and high performance.

[0115] The terminal device includes, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of a terminal device 3 and does not constitute a limitation on a terminal device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0116] The processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0117] The memory 31 can be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. The memory 31 can also be an external storage device of the terminal device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 3. Furthermore, the memory 31 can include both internal and external storage units of the terminal device 3. The memory 31 is used to store the computer program and other programs and data required by the roaming control device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0119] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0121] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0122] This invention provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0124] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units.

[0128] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0129] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0130] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0131] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0132] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0133] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting defects in chip bonding processes, characterized in that, The chip bonding process defect detection method includes: After the chip bonding points are formed, the bonding actuator is controlled to apply a pre-set periodic vibration mechanical excitation to the chip, and a set of image sequences of the bonding points under the periodic vibration mechanical excitation is collected. Based on the image sequence set, a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point are extracted. The first feature vector and the second feature vector are fused and input into the classification decision model to obtain the confidence scores corresponding to each of the multiple defect categories output by the classification decision model. If any of the confidence levels is greater than the threshold, the defect anomaly information corresponding to the confidence level is output; wherein, the defect anomaly information includes the defect category, the location of the bonding point and the chip serial number corresponding to the confidence level.

2. The method of claim 1, wherein the chip bonding process defect detection method is characterized by, The step of extracting a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point based on the image sequence set includes: For the first frame image in the image sequence set, identify the region of interest corresponding to the bonding point and obtain its initial bounding box; Using the initial bounding box as a reference, subpixel-level image registration is performed on each frame of the image sequence set to obtain the bond point aligned region of interest sequence; At least one frame of image is selected from the aligned region of interest sequence as structural analysis input, and input into a pre-trained first convolutional neural network for feature encoding, and outputs a first feature vector representing the static appearance features of the bonding point; For each pixel position in the aligned region of interest sequence, extract its grayscale value change in the entire region of interest sequence to generate the corresponding pixel-level temporal signal; Based on the pixel-level time-series signal, a second feature vector characterizing the dynamic response properties of the bonding point is extracted.

3. The method of claim 2, wherein the chip bonding process defect is detected by using a camera to capture an image of the chip bonding process defect. The step of performing sub-pixel-level image registration on each frame of the image sequence set based on the initial bounding box to obtain the sequence of regions of interest with precisely aligned bonding points includes: Based on the region within the first frame image of the initial bounding box, a set of feature points are extracted; Construct a Gaussian image pyramid between the first frame image and the current frame; wherein, the current frame refers to the frame image to be registered in the image sequence set; Starting from the highest layer of the Gaussian image pyramid, based on the position of the initial bounding box or the initial transformation parameters passed from the previous layer, the feature points are projected onto the current frame of the current layer, up to the original resolution layer. For each projected feature point, the optical flow method is used to calculate the local displacement with sub-pixel accuracy in its neighborhood through iterative optimization. Based on the local displacements calculated from all feature points, fit the affine transformation model of the current layer; The fitted affine transformation model is passed to the next layer as the initial transformation parameters for the next layer. Based on the affine transformation model finally optimized at the original resolution layer, the current frame is reverse-mapped and resampled to align it with the reference frame. Based on the initial bounding box, the precisely aligned region of interest of the bonding points is cropped from the aligned current frame.

4. The chip bonding process defect detection method as described in claim 2, characterized in that, The first convolutional neural network includes a multi-scale feature pyramid module, a spatial attention module, and a first feature encoding layer; The multi-scale feature pyramid module is used to extract and fuse multi-scale visual features from the input image by setting multiple dilated convolutional branches with different receptive fields in parallel. The parallel dilated convolutional branches of the multi-scale feature pyramid module include a first branch, a second branch, a third branch, and a global context branch. The first branch uses a 1x1 standard convolutional layer; the second branch uses a 3x3 convolutional layer with a dilation rate of 2; and the third branch uses a 3x3 convolutional layer with a dilation rate of 4. The global context branch includes a global average pooling layer, a 1x1 convolutional layer, and an upsampling layer. The outputs of each branch are concatenated along the channel dimension. The spatial attention module performs global max pooling and global average pooling on the input feature map, and concatenates the pooling results along the channel dimension. The spatial attention module passes the stitched feature map through a sub-network including convolutional layers and activation functions to generate a single-channel spatial weight map with the same spatial size as the input feature map; wherein, the spatial weight map is used to enhance the feature response of the bonding point region; The first feature encoding layer is used to pool and transform the single-channel spatial weight map after spatial attention enhancement to generate the first feature vector.

5. The chip bonding process defect detection method as described in claim 2, characterized in that, The step of extracting a second feature vector representing the dynamic response characteristics of the bonding point based on the pixel-level temporal signal includes: For each pixel-level timing signal, calculate its dynamic response characteristics with respect to the periodic micromechanical excitation signal; The dynamic response features calculated from all pixel positions are rearranged according to their spatial positions to generate a multi-channel spatiotemporal feature map. The spatiotemporal feature map is input into the second neural network for feature encoding, and the output is a second feature vector representing the dynamic response characteristics of the bonding point.

6. The chip bonding process defect detection method as described in claim 5, characterized in that, The step of calculating the dynamic response characteristics between each pixel-level time-series signal and the periodic micromechanical excitation signal includes: Acquire the reference timing signal for the periodic micromechanical excitation applied to the chip; Based on the acquisition timestamp corresponding to the region of interest and the generation timestamp of the reference timing signal, the two are unified to the same time axis; Perform cross-correlation analysis between pixel-level time-series signals and reference time-series signals to find the time offset corresponding to the maximum cross-correlation value; The pixel-level timing signal is shifted according to the time offset to achieve time-domain synchronization. Fourier transforms are performed on the synchronized pixel-level timing signal and the reference timing signal respectively to obtain their complex spectrum values ​​at the fundamental frequency of periodic micromechanical excitation. The phase angle difference between the fundamental frequency complex spectrum value of the pixel-level timing signal and the fundamental frequency complex spectrum value of the reference timing signal is calculated and used as the dynamic response feature of the pixel position.

7. The chip bonding process defect detection method as described in claim 5, characterized in that, The second neural network includes a spatiotemporal feature decoupling coding module, a deep spatiotemporal feature extraction module, a channel attention module, and a second feature coding layer; The spatiotemporal feature decoupling encoding module has parallel spatial relationship paths and channel relationship paths. The spatial relationship path is used to encode the spatial distribution pattern of dynamic features, and the channel relationship path is used to encode the correlation pattern between different dynamic features. The spatial relationship path consists of at least one 2D convolutional layer, and the channel relationship path consists of at least one 1x1 convolutional layer. The outputs of the spatial relationship path and the channel relationship path are fused by element-wise addition or channel concatenation. The deep spatiotemporal feature extraction module consists of multiple stacked separate convolutional blocks, each of which includes a deep convolutional layer for spatial filtering and a pointwise convolutional layer for channel fusion. The channel attention module is used to perform global average pooling on the input feature map to obtain a scalar description for each channel; The channel attention module is used to input a scalar description sequence into a neural network containing two fully connected layers and output the weight coefficients of each channel; wherein, the first fully connected layer is followed by a ReLU activation function and the second fully connected layer is followed by a Sigmoid activation function. The second feature encoding layer is used to pool and transform the features after channel attention weighting to generate the second feature vector.

8. A chip bonding process defect detection device, characterized in that, The chip bonding process defect detection device includes: The acquisition unit is used to control the bonding actuator to apply a pre-set periodic vibration mechanical excitation to the chip after the chip bonding points are formed, and to acquire a set of image sequences of the bonding points under the periodic vibration mechanical excitation. The extraction unit is used to extract a first feature vector characterizing the static appearance features of the bonding point and a second feature vector characterizing the dynamic response characteristics of the bonding point based on the image sequence set. A classification unit is used to fuse the first feature vector and the second feature vector and input them into a classification decision model to obtain the confidence scores of each of the multiple defect categories output by the classification decision model. The judgment unit is configured to output defect anomaly information corresponding to any confidence level if any of the confidence levels is greater than a threshold; wherein the defect anomaly information includes the defect category, bonding point location and chip serial number corresponding to the confidence level.

9. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a chip bonding process defect detection program stored in the memory and executable on the processor, the chip bonding process defect detection program being configured to implement the steps of the chip bonding process defect detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the chip bonding process defect detection method as described in any one of claims 1 to 7.

Images