A TVS diode detection system and method based on multi-modal vision technology

Abstract

The application discloses a kind of TVS diode detection system and method based on multi-modal vision technology, it is related to semiconductor package detection field, including: multi-modal image acquisition module, through time-sharing illumination and structure light projection synchronous acquisition device multidimensional original data;Multidimensional feature extraction module, with the same variance uncertainty weighted multi-task parallel network is registered to data fusion, the heterogeneous feature vector of surface state and pin geometric dimension is output simultaneously;Process abnormality deduction module, based on time sliding window and probability graph model dynamically calculates the abnormal posterior probability of front-end process node and outputs feedforward control signal;Welding risk prediction module, by welding health degree score and accordingly to device flexible shunt interception.The application realizes from high-precision multi-modal imaging, multi-task feature extraction, upstream process traceability to downstream welding quality prediction whole-process intelligent detection, effectively improves detection precision and production line intelligent control level.

Description

Technical Field

[0001] This invention relates to the field of semiconductor packaging inspection, specifically to a system and method for performing refined defect identification, upstream process tracing, and end-welding quality prediction of TVS diodes using multimodal computer vision, multi-task deep learning, and probabilistic graphical models. Background Technology

[0002] Transient voltage suppression (TVS) diodes are critical circuit protection devices, and their packaging quality directly affects the long-term reliability of downstream electronic products. In semiconductor packaging and testing lines, appearance defects (such as surface scratches, cracks, and stains) and lead geometry deviations (such as insufficient coplanarity, misalignment, and excessive roughness) of TVS diodes are the main causes of soldering failures such as tombstoning and cold solder joints in subsequent surface mount technology (SMT) reflow soldering processes.

[0003] Current mainstream inspection methods primarily rely on traditional two-dimensional automated optical inspection (2DAOI) technology. However, TVS diodes are typically packaged with highly reflective metal materials, and 2DAOI is prone to high-light overflow due to specular reflection during imaging, causing micron-sized bubbles and cracks to be obscured by strong light. Furthermore, it cannot acquire three-dimensional depth information of the pins, making it difficult to accurately quantify key geometric parameters such as pin coplanarity and skew angle, resulting in physical limitations to its inspection accuracy. Although some production lines have introduced discrete 3D measuring instruments, their inspection cycle is slow and cannot be matched with high-speed production lines, typically only used for offline sampling inspection.

[0004] At the data processing level, existing systems often employ multiple independent deep learning models to sequentially process segmentation (defect identification), detection (localization), and regression (size measurement) tasks, resulting in high inference latency and high hardware resource consumption. Even when attempts are made to use multi-task networks, the differences in the dimensions of the loss functions for different tasks and gradient conflicts often lead to "model collapse," where one task dominates the training, and the accuracy of other tasks drops significantly.

[0005] Furthermore, traditional inspection logic is based on "isolated static judgment": the system performs binary sorting of individual devices as "qualified" or "scrap" according to a preset fixed threshold, and a large amount of minor defect information on the edge of tolerance is directly discarded. This not only wastes valuable data containing the health status of the process, but also fails to provide early warning of the slow deterioration trend of upstream processes such as electroplating and stamping (such as mold wear and chemical concentration drift), and is often only passively discovered after a batch of scrap products has been generated.

[0006] More importantly, existing standards assume that as long as the physical dimensions of components meet the tolerances on the drawings, they can be used for surface mount technology. However, in the actual reflow soldering process, the superposition of multiple tiny deformations on the edge of compliance can produce a non-linear coupling effect, leading to the entry of potentially defective components that are "dimensionally compliant but have extremely low soldering reliability" into the market, posing a serious reliability risk to end products. Summary of the Invention

[0007] The purpose of this invention is to provide a TVS diode detection system and method based on multimodal vision technology to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a TVS diode detection system based on multimodal vision technology, comprising a multimodal image acquisition module for performing multi-dimensional optical scanning of the device to be detected to obtain a raw dataset, and further comprising:

[0009] The multidimensional feature extraction module is configured to receive the original dataset, perform alignment and dimensionality reduction extraction on the original dataset through a built-in multi-task parallel network, and simultaneously output a first feature vector characterizing the surface state of the device and a second feature vector characterizing the geometric dimension of the pins.

[0010] The process anomaly inference module is configured to receive the first feature vector and the second feature vector, collect multiple sets of continuously input feature vectors according to a preset time sliding window, dynamically calculate the posterior probability of anomalies of the front-end process node corresponding to the device under test based on a probabilistic graphical model, and output a process intervention feedback signal when the posterior probability of anomalies exceeds a preset threshold.

[0011] The welding risk prediction module is configured to receive the second feature vector, input a pre-trained kernel function regression model, nonlinearly map out the welding health score corresponding to the downstream reflow soldering process, and trigger a diversion based on the score to physically intercept or allow the device to be tested.

[0012] Preferably, the multimodal image acquisition module includes:

[0013] The time-sharing lighting unit includes a multi-quadrant programmable light source;

[0014] A structured light projection unit is used to project line laser onto the device under test;

[0015] A synchronization controller, electrically connected to the time-division illumination unit, the structured light projection unit, and the imaging unit, is configured to emit a microsecond-level pulse signal in response to a device arrival signal to hard trigger the imaging unit to synchronously complete exposure at the instant a specific light source is lit or a line laser is projected, thereby generating the original dataset in the same spatiotemporal coordinate system.

[0016] Preferably, the multidimensional feature extraction module is also configured with data registration logic and dynamic loss balancing logic;

[0017] The data registration logic operation method includes, in response to receiving the original dataset, using a spatial transformation network to perform sub-pixel-level spatial registration of the two-dimensional contour map, the three-dimensional point cloud projection map and the three-dimensional texture map reconstructed based on photometric stereo vision in the original dataset, and fusing them into a multi-channel tensor.

[0018] The dynamic loss balancing logic operation method includes inputting the multi-channel tensor into the multi-task parallel network, and dynamically adjusting the gradient conflict during the multi-task execution process by introducing homoscedastic uncertainty weights to ensure the synchronous convergence output of the first feature vector and the second feature vector.

[0019] Preferably, the data input terminal of the process anomaly inference module is also connected to a Gaussian mixture model, which is configured to cluster the feature dataset under historical normal production conditions, thereby providing a prior probability density function as a calculation benchmark for the probabilistic graphical model.

[0020] Preferably, a communication gateway is also included. The process anomaly prediction module establishes a closed-loop control link with the front-end production equipment through the communication gateway. It is configured to issue a shutdown or parameter adjustment command to the front-end production equipment through the communication gateway in response to the anomaly posterior probability exceeding the control limit set based on the process capability index for multiple consecutive time windows.

[0021] Preferably, the kernel function regression model uses a radial basis function, which is configured to perform dimensionless standardization on the pin coplanarity, skew angle and roughness in the second feature vector, and then solve the hyperplane regression equation to output the welding health score.

[0022] As a preferred embodiment, the method for physically intercepting or allowing the device under test based on score-triggered diversion includes the following steps:

[0023] If the score is higher than the first set value, keep the conveyor belt straight to allow passage to the automotive-grade production line;

[0024] If the score is between the first set value and the second set value, it will be directed to the consumer-grade production line;

[0025] If the score is lower than the second set value, rejection is triggered, and the device to be tested is pushed into the scrap storage box.

[0026] Preferably, when performing multi-dimensional optical scanning on the device under test, the multimodal image acquisition module, for highly reflective material surfaces, filters out specular overflow from the physical information by solving the normal vector and albedo in the Lambertian reflection model to reconstruct the surface texture.

[0027] Furthermore, the present invention also provides a TVS diode detection method based on multimodal vision technology, comprising the following steps:

[0028] Step S1: Synchronously trigger optical and imaging hardware to acquire multi-dimensional physical information and fuse and reconstruct it;

[0029] Step S2: Using the aforementioned reconstructed data as input, heterogeneous feature vectors are output in parallel through a multi-task network with homoscedastic uncertainty weighting;

[0030] Steps S3 and S4 are executed in parallel. Step S3 accumulates the aforementioned feature vectors using a time sliding window to deduce the posterior probability of the process and performs feedforward control. Step S4 uses the geometric feature vectors of the current device to regress and predict the welding risk score and performs end-point physical interception.

[0031] Preferably, if a batch change occurs on the production line, the probabilistic graphical model in step S3 combines a hidden Markov model to treat the process states of different batches as the hidden states of the Markov chain in order to update the prior probabilities.

[0032] In summary, the beneficial effects of this invention are:

[0033] 1. By using the Lambertian reflection model and multi-quadrant time-division illumination technology, the mirror interference of highly reflective metal surfaces is effectively eliminated, restoring the true surface texture; combined with laser triangulation and spatial transformation network (STN), sub-pixel-level registration and multi-modal fusion of 2D / 3D images are achieved, significantly improving the detection accuracy of minute defects (such as cracks and bubbles) and pin geometric parameters (such as coplanarity and skew angle).

[0034] 2. A dynamic loss balancing mechanism based on homoscedasticity uncertainty is introduced to automatically adjust the weights of segmentation, detection, and regression tasks, avoiding gradient conflicts and "model collapse". While ensuring the accuracy of multi-task parallel inference, latency is significantly reduced to meet the high-cycle detection requirements of the production line.

[0035] 3. A Gaussian Mixture Model (GMM) is used to construct the prior distribution of normal processes. Combined with a time sliding window and a Bayesian Belief Network (BBN), the posterior probability of anomalies at front-end process nodes is dynamically calculated. Early warnings or shutdown commands can be issued to production equipment before batch defects occur, achieving a leap from "passive interception" to "proactive process diagnosis and control," effectively reducing scrap rates and production costs.

[0036] 4. Based on the radial basis function (RBF) support vector regression (SVR) model, multi-dimensional pin features are reduced in dimension and mapped to a welding health score. This identifies potentially hazardous devices that are "size compliant but have high welding risks." Based on the score, flexible diversion or scrapping is achieved for automotive-grade and consumer-grade production lines, balancing the goal of "zero defects" with the economic value of marginal good products.

[0037] This not only improves the accuracy and efficiency of TVS diode appearance and size inspection, but also enables early warning of upstream process anomalies and quantitative prediction of downstream welding quality, significantly enhancing the intelligence level and quality control capabilities of semiconductor packaging and testing production lines. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the overall structure of a TVS diode detection system based on multimodal vision technology according to the present invention;

[0040] Figure 2 This is a schematic diagram of the multimodal optical acquisition and FPGA timing control principle of a TVS diode detection system based on multimodal vision technology according to the present invention.

[0041] Figure 3 This is a flowchart of a multi-task parallel feature extraction network for a TVS diode detection system based on multimodal vision technology according to the present invention.

[0042] Figure 4 This is a logic block diagram for deducing process anomalies in a TVS diode detection system based on multimodal vision technology according to the present invention.

[0043] Figure 5 This is a flowchart of the welding risk prediction and current diversion process of a TVS diode inspection system based on multimodal vision technology according to the present invention;

[0044] Figure 6 This is a schematic diagram illustrating the feature detection of a TVS diode detection system based on multimodal vision technology according to the present invention. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0046] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.

[0047] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0048] Any feature disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by other equivalent or similar features for a similar purpose, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0049] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; they can refer to the internal communication of at least two elements or the interaction relationship of at least two elements, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0050] Please see Figures 1-6 The present invention provides an embodiment of a TVS diode detection system based on multimodal vision technology, as shown in the reference. Figure 1 Through multimodal optical acquisition at the hardware level, high-light interference is eliminated mainly by photometric stereo vision and laser triangulation, and high-precision 2D / 3D features are extracted. The algorithm layer adopts multi-task CNN with dynamic weighting of homoscedastic uncertainty to output heterogeneous features in parallel. The logic layer introduces Bayesian Belief Network (BBN) with time series constraints to back-infer the posterior probability of anomalies in hidden process nodes. Finally, the multidimensional visual features are reduced in dimensionality and mapped to welding health scores through support vector regression (SVR) based on RBF kernel, realizing the leap from passive visual interception to active process diagnosis.

[0051] At the physical and logical levels, it mainly includes the following four core modules:

[0052] Multimodal image acquisition module: Used to perform multi-dimensional optical scanning of the device under test to obtain raw data. This module includes an FPGA hard trigger controller, a four-quadrant programmable LED ring light source, a bottom high-brightness parallel backlight, an industrial-grade high frame rate area array camera, and a lateral high-precision line laser displacement sensor. The specific structure includes a time-division illumination unit containing a multi-quadrant programmable light source, a structured light projection unit for projecting line laser light onto the device under test, and a synchronization controller electrically connected to the time-division illumination unit, the structured light projection unit, and the imaging unit.

[0053] Multi-dimensional feature extraction module: configured to receive the original dataset, perform alignment and dimensionality reduction extraction on the original dataset through a built-in multi-task parallel network, and simultaneously output a first feature vector representing the surface state of the device and a second feature vector representing the geometric dimension of the pins. This module is deployed on an edge computing terminal equipped with GPU computing power and has a built-in multi-task feature extraction engine based on the ConvNeXt backbone network and combined with the Spatial Transformation Network (STN).

[0054] Process anomaly inference module: It is configured to receive a first feature vector and a second feature vector, and collect multiple sets of continuously input feature vectors according to a preset time sliding window. This module has a built-in graph theory-based Bayesian belief network inference engine and a historical process distribution Gaussian mixture model (GMM), and dynamically calculates the posterior probability of anomalies of the front-end process nodes corresponding to the device under test based on the probabilistic graphical model. When the posterior probability of anomalies exceeds a preset threshold, it outputs a process intervention feedback signal.

[0055] Welding risk prediction module: It is configured to receive a second feature vector, input a pre-trained kernel function regression model, nonlinearly map out the welding health score corresponding to the downstream reflow soldering process, and trigger a diversion based on the score to physically intercept or allow the device to be tested. This module has a built-in pre-trained support vector regression (SVR) model and an automated diversion communication gateway.

[0056] The overall execution includes the following steps:

[0057] Step S1: Microsecond-level multimodal data acquisition and physical reconstruction based on FPGA hard triggering, refer to... Figure 2 ;

[0058] The synchronization controller is configured to emit microsecond-level pulse signals in response to a device positioning signal to hard trigger the imaging unit to synchronously complete exposure at the instant a specific light source is illuminated or a line laser is projected, thereby generating a raw dataset in the same spatiotemporal coordinate system. Based on the Lambertian reflection model, the processor receives four image grayscale matrices I from different illumination directions. Given the light source direction matrix L, the surface normal vector n and albedo P of each pixel are solved using the least squares method.

[0059] ;

[0060] When performing multi-dimensional optical scanning on the device under test, the multimodal image acquisition module, for highly reflective surfaces, filters out specular highlights from physical information by solving for the normal vector and albedo in the Lambertian reflection model to reconstruct the surface texture. The microscopic 3D texture map of the surface after highlight removal is reconstructed through normal vector integration. Subsequently, a bottom parallel backlight is activated to extract the high-contrast 2D outer contour. Simultaneously, a side-mounted line laser emitter projects at a 30° angle. Utilizing the principle of laser triangulation and combining it with the Scherm's law calibration matrix, the image pixel coordinates are converted into real-world 3D point cloud coordinates (X, Y, Z) to extract pin coplanarity.

[0061] Traditional 2DAOI systems typically use coaxial light or ordinary ring light. When facing the highly reflective metal surface of a TVS diode, they are prone to high light overflow (causing tiny bubbles to be obscured) or shadows (causing the loss of crack edge features), and cannot obtain depth information of the pins.

[0062] This invention, by introducing the Lambertian reflection model and time-division stroboscopic light source, directly solves the pixel normal vector at the physical optics level. Compared with existing technologies, this method completely eliminates the interference of specular reflection, improving the contrast of tiny defects on the metal surface by several times; simultaneously, combined with laser triangulation to synchronously obtain 3D coplanarity, it achieves high-precision topography restoration with a single station and no blind spots, significantly reducing the front-end false negative rate.

[0063] Step S2: Parallel feature extraction using multi-task CNNs weighted by homoscedasticity uncertainty, refer to... Figure 3 ;

[0064] Data alignment: The affine transformation matrix is ​​calculated using a spatial transformation network, and the 3D point cloud projection map, 2D contour map and surface texture map obtained by S1 are registered at the subpixel level and stitched into a multi-channel tensor.

[0065] Feature Extraction and Dynamic Loss Balancing: Tensor Input Multi-Task Network (Backbone+FPN). To address the gradient dominance problem caused by different scales in segmentation, detection, and regression tasks, this system introduces homoscedastic uncertainty to automatically learn the weights for each task. To ensure the synchronized convergence of the first and second feature vectors, the total loss function is defined as follows:

[0066]

[0067] Output layer: The surface branch outputs a mask calculation vector, which is the first feature vector.

[0068] ;

[0069] The pin branch performs continuous value regression and outputs a vector, which is the second feature vector:

[0070] .

[0071] If multiple independent CNN models are called sequentially for segmentation (surface defects), detection, and regression (pin deformation), it will result in extremely high inference latency and memory usage, which cannot meet the high-speed requirements of the production line. If they are simply merged into a multi-task network, gradient conflicts often occur because the loss function dimensions of the regression task and the segmentation task are too different. One task dominates the network update, and other tasks experience "model collapse".

[0072] This step innovatively introduces the automatic calculation of task weights based on homoscedastic uncertainty. Compared to the traditional static weight allocation through manual parameter tuning, this method allows the network to dynamically balance the gradient magnitudes of segmentation, detection, and regression tasks during backpropagation. Its benefits include: compressing end-to-end inference latency while maintaining high accuracy, significantly reducing reliance on edge computing hardware for computational power and energy consumption.

[0073] Step S3: Bayesian process anomaly deduction using a time-sliding window, refer to... Figure 4 ;

[0074] Parameter initialization: The historical normal production data is clustered using a Gaussian mixture model (GMM) to establish the prior probability density functions of surface micro-defects S and pin micro-defects P under normal process conditions.

[0075] Dynamic joint probability calculation: A sliding time window of size N is set (e.g., the most recent 1000 devices). The system calculates the joint probability distribution of S and P within the window in real time. .

[0076] Posterior anomaly inference: Based on Bayes' theorem, calculate the posterior probability of anomalies occurring at hidden nodes (such as excessive impurities in electroplating solution or wear of stamping dies) when a specific combination of defects is observed.

[0077]

[0078] when If the control limit is exceeded for three consecutive time windows based on the process capability index Cpk, the system also includes a communication gateway. The process anomaly prediction module establishes a closed-loop control link with the front-end production equipment through the communication gateway. It is configured to issue shutdown or parameter adjustment commands to the front-end production equipment via the communication gateway in response to the anomaly posterior probability exceeding the control limit set based on the process capability index for multiple consecutive time windows.

[0079] Traditional inspection systems are "isolated" and "static," performing a binary "pass / fail" judgment based solely on fixed size thresholds. A large number of tiny defects that are on the edge of tolerance (not reaching the scrap line) are discarded as invalid data, making it impossible to provide early warning of the slow deterioration trend (such as chemical concentration drift) in upstream processes such as electroplating and stamping.

[0080] This step transforms discrete visual inspection results into continuous process diagnostic data by constructing a dynamic Bayesian belief network (BBN). Compared to traditional post-inspection sampling, this solution, by solving for the posterior probability, can issue early warnings to the PLC before a batch of defective products is generated, i.e., when process parameters are in a sub-optimal state. This achieves a fundamental technological leap in quality control from "passively intercepting terminal defects" to "actively predicting and feedforward closed-loop control."

[0081] Step S4: Dimensionality reduction mapping of SVR welding reliability based on RBF kernel function, refer to... Figure 5 ;

[0082] Data standardization: This involves standardizing continuous features detected in real-time, including coplanarity variance. , deflection angle Roughness Z-score standardization is performed to eliminate the influence of dimensions. The hyperplane regression equation is then solved to output the welding health score, and the input vector is constructed as follows:

[0083]

[0084] Kernel function mapping: Input vector X into the support vector regression model that has been calibrated using the "soldering balance test". Due to the complex nonlinear relationship between features and solder wetting force, a radial basis kernel function is used.

[0085]

[0086] Risk Output and Control: The model solves the hyperplane regression equation and outputs a... The "welding health score" is referenced. Figure 6 :

[0087] The method for physically intercepting or allowing the device under test based on score-triggered traffic control includes the following steps:

[0088] If the score is higher than the first set value (e.g., greater than 90), keep the conveyor belt straight to allow it to be released to the automotive-grade production line;

[0089] If the score is between the first and second set values ​​(e.g., 70-90), it will be directed to the consumer-grade production line.

[0090] If the score is lower than the second set value (e.g., less than 70), the rejection mechanism is triggered to intercept and push the device to be tested into the scrap collection box.

[0091] Traditional standards state that "as long as the physical dimensions of components (such as coplanarity and skew angle) meet the tolerances of the drawings, they can be placed normally." However, in the actual SMT reflow soldering process, the superposition of multiple tiny deformations on the edge of the qualified state can easily cause nonlinear uneven stress, leading to "tombstone" or "cold solder joint" phenomena, which pose serious hidden dangers to the end product.

[0092] This invention breaks through the single-dimensional size verification logic. Through the SVR model of radial basis kernel function, it reduces the dimensionality of multi-dimensional spatial geometric features and maps them to a "soldering health score" that is directly related to the actual solder wetting force. Compared with the existing technology, its core beneficial effect is that it intercepts those hidden devices that are "size compliant but have extremely high overall soldering risk" and realizes flexible graded allocation of devices (automotive grade / consumer grade). While ensuring zero defects in the core supply chain, it maximizes the economic value of marginal good products.

[0093] It is also worth mentioning that in this embodiment, in step S3, if there are multiple batch small-batch switching on the production line, the Bayesian inference model can be combined with the Hidden Markov Model (HMM) to treat the process states of different batches as the hidden states of the Markov chain, so as to improve the inference accuracy in the early stage of line changeover.

[0094] In actual operation, the following example is used.

[0095] Inspection Item: One TVS diode from batch number B2401, model SMBJ5.0A, part number D1234. This device has just completed the packaging process and has entered the online visual inspection station.

[0096] 1. Multimodal image acquisition (step S1)

[0097] The device arrival signal triggers the synchronization controller (FPGA hard trigger), which sequentially completes the following acquisitions within a microsecond time:

[0098] Photometric stereoscopic imaging: A four-quadrant programmable LED ring light source flashes sequentially in the directions of 0°, 90°, 180°, and 270°, and an industrial camera simultaneously acquires four grayscale images (resolution 2048×2048) from different lighting directions.

[0099] Bottom-backlit imaging: A parallel backlight source is illuminated at the bottom to obtain a high-contrast image of the device's outer contour.

[0100] Laser triangulation: A lateral line laser projector scans the pin area at a 30° angle and generates 3D point cloud data by calibrating the matrix using Schahm's law.

[0101] The collected raw dataset includes: 4 photometric stereo images, 1 backlit image, and 1 set of pin 3D point clouds.

[0102] High reflectivity treatment: The pin surface is made of tin-plated copper, exhibiting strong specular reflection. The system solves the Lambertian reflection model, calculating the normal vector and albedo pixel-by-pixel on four stereoscopic photometric images, successfully filtering out highlight overflow and reconstructing a clear surface micro-texture map. Figure 6 (Top left). At the same time, laser triangulation obtains the three-dimensional coordinates of the pin and calculates preliminary geometric quantities such as coplanarity and skew angle.

[0103] 2. Multidimensional feature extraction (step S2)

[0104] Raw data is input into the multi-dimensional feature extraction module (edge ​​computing terminal, GPU computing power supported):

[0105] Spatial registration: The 2D contour map, 3D point cloud projection map and surface texture map are aligned at the subpixel level using the Spatial Transformation Network (STN) and fused into a 6-channel tensor, which is RGB texture + depth + gradient + albedo.

[0106] Multi-task CNN inference: Tensors are fed into a multi-task network based on the ConvNeXt backbone. The loss for segmenting surface defects, detecting pin positions, and regressing geometry is dynamically weighted using homoscedastic uncertainty. Network output:

[0107] First feature vector (surface condition): [Defect type = 1 (scratch), defect area = 0.02mm] 2 [Defect depth = 5μm] A tiny scratch was detected at the root of the pin, which traditional AOI cannot identify due to reflection;

[0108] Second eigenvector (pin geometry): [Coplanarity variance = 0.03mm] 2 [Oblique angle = 1.2°, roughness Ra = 0.8μm], pin coplanarity is near the upper limit of specification, but does not exceed tolerance.

[0109] 3. Process Anomaly Prediction (Step S3)

[0110] The system maintains a time sliding window of size N=1000 and statistically analyzes the feature vector distribution of all devices within the window in real time. Historical normal process data has been clustered using a Gaussian mixture model (GMM) to establish a joint prior probability density function for surface defects (S) and pin defects (P).

[0111] After the current device is added to the window, the joint defect distribution P(S,P) of the most recent 1000 devices is statistically obtained. Based on the Bayesian belief network, the posterior probability of hidden process nodes, such as the impurity concentration in the electroplating tank and the abnormal wear of the stamping die, is calculated. The calculated posterior probability is 96.5%, which has exceeded the control limit (95%) set based on the process capability index (Cpk) for three consecutive time windows. The system sends a command to the electroplating equipment through the communication gateway: "Increase the electroplating current by 2% and check the impurity concentration in the tank solution", realizing feedforward closed-loop control and avoiding batch scrap.

[0112] 4. Welding risk prediction and diversion (step S4)

[0113] The second feature vector [0.03, 1.2, 0.8], after Z-score normalization, is input into a pre-trained RBF kernel support vector regression model, which has been calibrated using soldering balance test data. The model outputs a soldering health score of 88 (out of 100).

[0114] According to the preset traffic splitting rules:

[0115] Score > 90 → Released to automotive-grade production line;

[0116] 70≤score≤90→guide to consumer-grade production line;

[0117] A score of <70 triggers the removal and disqualification of the organization.

[0118] Since the score of 88 falls within the 70-90 range, the system controls the sorting forks on the conveyor belt to guide the device to the consumer-grade production line hopper, while simultaneously recording its quality data for subsequent traceability.

[0119] The above description is merely a specific embodiment of the invention, but the scope of protection of the invention is not limited thereto. Any variations or substitutions conceived without inventive effort should be included within the scope of protection of the invention. Therefore, the scope of protection of the invention should be determined by the scope defined in the claims.

Claims

1. A TVS diode detection system based on multimodal vision technology, comprising a multimodal image acquisition module for performing multi-dimensional optical scanning of the device under test to obtain a raw dataset, characterized in that: Also includes: The multidimensional feature extraction module is configured to receive the original dataset, perform alignment and dimensionality reduction extraction on the original dataset through a built-in multi-task parallel network, and simultaneously output a first feature vector characterizing the surface state of the device and a second feature vector characterizing the geometric dimension of the pins. The process anomaly inference module is configured to receive the first feature vector and the second feature vector, collect multiple sets of continuously input feature vectors according to a preset time sliding window, dynamically calculate the posterior probability of anomalies of the front-end process node corresponding to the device under test based on a probabilistic graphical model, and output a process intervention feedback signal when the posterior probability of anomalies exceeds a preset threshold. The welding risk prediction module is configured to receive the second feature vector, input a pre-trained kernel function regression model, nonlinearly map out the welding health score corresponding to the downstream reflow soldering process, and trigger a diversion based on the score to physically intercept or allow the device to be tested.

2. The TVS diode detection system based on multimodal vision technology according to claim 1, characterized in that: The multimodal image acquisition module includes: The time-sharing lighting unit includes a multi-quadrant programmable light source; A structured light projection unit is used to project line laser onto the device under test; A synchronization controller, electrically connected to the time-division illumination unit, the structured light projection unit, and the imaging unit, is configured to emit a microsecond-level pulse signal in response to a device arrival signal to hard trigger the imaging unit to synchronously complete exposure at the instant a specific light source is lit or a line laser is projected, thereby generating the original dataset in the same spatiotemporal coordinate system.

3. The TVS diode detection system based on multimodal vision technology according to claim 2, characterized in that: The multidimensional feature extraction module is also configured with data registration logic and dynamic loss balancing logic; The data registration logic operation method includes, in response to receiving the original dataset, using a spatial transformation network to perform sub-pixel-level spatial registration of the two-dimensional contour map, the three-dimensional point cloud projection map and the three-dimensional texture map reconstructed based on photometric stereo vision in the original dataset, and fusing them into a multi-channel tensor. The dynamic loss balancing logic operation method includes inputting the multi-channel tensor into the multi-task parallel network, and dynamically adjusting the gradient conflict during the multi-task execution process by introducing homoscedastic uncertainty weights to ensure the synchronous convergence output of the first feature vector and the second feature vector.

4. The TVS diode detection system based on multimodal vision technology according to claim 1, characterized in that: The data input terminal of the process anomaly inference module is also connected to a Gaussian mixture model. The Gaussian mixture model is configured to cluster the feature dataset under historical normal production conditions, thereby providing a prior probability density function as a calculation benchmark for the probabilistic graphical model.

5. The TVS diode detection system based on multimodal vision technology according to claim 4, characterized in that: It also includes a communication gateway. The process anomaly prediction module establishes a closed-loop control link with the front-end production equipment through the communication gateway. It is configured to issue a shutdown or parameter adjustment command to the front-end production equipment through the communication gateway in response to the anomaly posterior probability exceeding the control limit set based on the process capability index for multiple consecutive time windows.

6. The TVS diode detection system based on multimodal vision technology according to claim 1, characterized in that: The kernel function regression model uses a radial basis function, which is configured to standardize the pin coplanarity, skew angle and roughness in the second feature vector by eliminating dimensions, and then solve the hyperplane regression equation to output the welding health score.

7. The TVS diode detection system based on multimodal vision technology according to claim 6, characterized in that: The method for physically intercepting or allowing the device under test based on score-triggered traffic diversion includes the following steps: If the score is higher than the first set value, keep the conveyor belt straight to allow passage to the automotive-grade production line; If the score is between the first set value and the second set value, it will be directed to the consumer-grade production line; If the score is lower than the second set value, rejection is triggered, and the device to be tested is pushed into the scrap storage box.

8. The TVS diode detection system based on multimodal vision technology according to claim 2, characterized in that: When performing multi-dimensional optical scanning on the device under test, the multimodal image acquisition module filters out specular overflow from the physical information to reconstruct the surface texture by solving the normal vector and albedo in the Lambertian reflection model for highly reflective material surfaces.

9. A TVS diode detection method based on multimodal vision technology, using the system described in any one of claims 1-8, characterized in that: Includes the following steps: Step S1: Synchronously trigger optical and imaging hardware to acquire multi-dimensional physical information and fuse and reconstruct it; Step S2: Using the aforementioned reconstructed data as input, heterogeneous feature vectors are output in parallel through a multi-task network with homoscedastic uncertainty weighting; Steps S3 and S4 are executed in parallel. Step S3 accumulates the aforementioned feature vectors using a time sliding window to deduce the posterior probability of the process and performs feedforward control. Step S4 uses the geometric feature vectors of the current device to regress and predict the welding risk score and performs end-point physical interception.

10. The TVS diode detection method based on multimodal vision technology according to claim 9, characterized in that: If a batch change occurs on the production line, the probabilistic graphical model in step S3 combines a hidden Markov model to treat the process states of different batches as the hidden states of a Markov chain in order to update the prior probabilities.

Images