Advanced packaging defect online detection and compensation method and system based on machine vision

Abstract

The application discloses an advanced packaging defect online detection and compensation method and system based on machine vision, relates to the chip packaging technical field, and comprises the following steps: acquiring multi-modal visual information of a packaging structure in an advanced packaging process; inputting the multi-modal visual information into an intelligent defect analysis engine to perform layered defect detection and identification; generating a targeted compensation decision by a real-time compensation control module; based on the compensation decision, controlling a compensation execution mechanism to perform online dynamic adjustment on current or subsequent packaging process parameters; after the compensation action is executed, image information of a packaging object is collected again, the compensation effect is verified, and the whole-process data of the current detection, compensation and verification is synchronously updated to a data management and visualization platform. The application has the advantages that by constructing a closed-loop control system integrating online detection, intelligent analysis and real-time compensation, the lagging defect of traditional offline detection and manual intervention is effectively overcome.

Description

Technical Field

[0001] This invention relates to the field of chip packaging technology, specifically to an advanced online detection and compensation method and system for packaging defects based on machine vision. Background Technology

[0002] Advanced packaging technology is a key path to continuing Moore's Law and achieving high performance and high integration in electronic systems. As chip feature sizes continue to shrink and three-dimensional integrated structures become increasingly complex, defect control in the packaging process faces unprecedented challenges. Defects such as via (TSV) holes, bridging and breakage in redistribution layers (RDLs), uneven height of microbumps (μ-bumps), and alignment deviations in chip stacking directly affect the electrical performance, reliability, and yield of the final product. Currently, the industry mainly relies on manual sampling or offline automated optical inspection (AOI) for quality control. Manual inspection is inefficient, subjective, and prone to fatigue; while offline AOI can achieve a certain degree of automation, it typically only provides a "yes / no" defect judgment and rejects defective products, failing to provide real-time feedback to the production line for process adjustments. This results in the inability to correct the causes of defects in a timely manner, leading to waste of materials and time, and making it difficult to meet the stringent requirements of high yield and low cost for advanced packaging mass production.

[0003] Existing online inspection solutions typically separate the inspection system from the production process equipment, resulting in limited functionality. Most are limited to using machine vision for defect identification and classification, reporting results as alarms. Subsequent process compensation heavily relies on engineers' experience for manual parameter adjustments, leading to significant lag in this open-loop control model. More importantly, current technology lacks a closed-loop system that deeply integrates "high-precision defect perception," "intelligent root cause analysis," and "real-time proactive compensation." The disconnect between defect detection, analysis, and compensation prevents immediate and accurate process parameter corrections at the moment a defect occurs or in subsequent processing within the same batch, thus failing to prevent defect recurrence. Therefore, developing an integrated system capable of online real-time detection, intelligent decision-making, and automatic compensation is an urgent industrial need for achieving intelligent manufacturing and zero-defect production in advanced packaging. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an advanced online detection and compensation method and system for packaging defects based on machine vision. This technical solution solves at least one of the technical problems mentioned in the background section.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] Advanced online detection and compensation methods for packaging defects based on machine vision include:

[0007] In advanced packaging processes, multispectral vision acquisition modules are used to acquire images of intermediate or finished packaging products online to obtain multimodal visual information of the packaging structure.

[0008] The multimodal visual information is input into the intelligent defect analysis engine to perform hierarchical defect detection and identification, including locating, classifying and assessing the severity of geometric defects, material defects or potential electrical performance defects in the packaging structure.

[0009] Based on the location, classification, and severity assessment results of the defects, the real-time compensation control module generates targeted compensation decisions, which include at least the types and amounts of process parameters that need to be adjusted.

[0010] Based on the compensation decision, the control compensation execution mechanism dynamically adjusts the current or subsequent packaging process parameters online to compensate for identified defects or prevent similar defects.

[0011] After the compensation action is performed, the image information of the encapsulated object is collected again to verify the compensation effect, and the entire process data of this detection, compensation and verification is synchronously updated to the data management and visualization platform.

[0012] Preferably, in the advanced packaging process, the use of a multispectral vision acquisition module to perform online image acquisition of intermediate or finished packaging products to obtain multimodal visual information of the packaging structure specifically includes:

[0013] Using at least two of the illumination modes of bright field illumination, dark field illumination and polarized light illumination, alternate or synchronous image acquisition is performed on the same package area to obtain images that reflect different features of surface morphology, edge contour and internal structure.

[0014] While acquiring two-dimensional images, three-dimensional topographic information of the key encapsulation area is obtained through laser triangulation, structured light or binocular vision methods.

[0015] For different advanced packaging process types, including through-silicon vias, redistribution layers, or microbumps, the illumination angle, light source wavelength, and camera parameters are adaptively configured to optimize the imaging quality of target features.

[0016] Preferably, the step of inputting the multimodal visual information into the intelligent defect analysis engine to perform hierarchical defect detection and recognition specifically includes:

[0017] Using traditional image processing algorithms based on threshold segmentation, edge detection, or template matching, the acquired images are analyzed quickly in real time to filter out areas with obvious anomalies;

[0018] An image containing anomalies is input into a deep learning-based defect detection neural network model. The model employs a multi-scale feature pyramid structure that incorporates an attention mechanism to accurately identify and locate defect types.

[0019] The identified defects are reconstructed in three dimensions and quantitatively analyzed. The severity of the defects is assessed by combining historical process data, and the evolution trend of the defects in subsequent processes is predicted by using a sequence model.

[0020] Preferably, the training and optimization process of the defect detection neural network model includes:

[0021] We constructed a labeled image dataset containing various advanced encapsulation defect types and performed data augmentation operations on the dataset, including rotation, scaling, brightness adjustment, and noise addition.

[0022] A transfer learning strategy is adopted, using a model pre-trained on a large-scale general image dataset as the base network, and fine-tuning the training using the encapsulation defect dataset.

[0023] During the model inference stage, knowledge distillation or model pruning techniques are used to lightweight the network to meet the real-time requirements of online detection, and the optimized model parameters are deployed to edge computing devices through an online update mechanism.

[0024] Preferably, the step of generating targeted compensation decisions by the real-time compensation control module based on the location, classification, and severity assessment results of the defects specifically includes:

[0025] Based on the type, size, location, and severity of the defect, a matching query is performed in the preset "defect feature-process parameter mapping relationship database" to obtain preliminary compensation strategy suggestions;

[0026] A compensation decision model is constructed using reinforcement learning algorithms. The model takes the current process state and defect information as state input, process parameter adjustment as action space, and the degree of quality improvement after compensation as reward. The compensation strategy is iteratively optimized through interaction with the environment.

[0027] Before implementing physical compensation, the compensation decision is input into the digital twin model of the packaging process for virtual verification, the compensation effect is predicted, and the compensation parameters are fine-tuned based on the simulation results.

[0028] Preferably, the step of controlling the compensation execution mechanism to dynamically adjust the current or subsequent packaging process parameters online based on the compensation decision, so as to compensate for the identified defects or prevent similar defects, specifically includes:

[0029] For geometric defects such as bonding position misalignment or misalignment, compensation is made by controlling a multi-axis precision motion platform to adjust the position and attitude of the bonding head, mounting head or lithography machine stage in real time.

[0030] For material defects such as insufficient solder amount and uneven bump height, compensation can be made by controlling the dispensing valve and the output of the inkjet head, or by adjusting the temperature profile and pressure parameters of the reflow soldering.

[0031] For potential electrical defects caused by contamination or oxidation, in-situ plasma cleaning or local flux spraying processes are triggered, and after compensation, rapid verification is performed using electrical test probes integrated into the production line.

[0032] Furthermore, an advanced online detection and compensation system for packaging defects based on machine vision is proposed to implement the aforementioned advanced online detection and compensation method for packaging defects based on machine vision, including:

[0033] A multispectral vision acquisition module, comprising a high-resolution industrial camera array, a programmable multi-angle ring light source system, and beam-splitting optical components, is used for online acquisition of multimodal images of packaged objects;

[0034] The intelligent defect analysis engine is connected to the visual acquisition module and integrates a traditional image processing algorithm library and a deep learning-based defect detection model to perform layered defect analysis and identification on the acquired images.

[0035] The real-time compensation control module is communicatively connected to the intelligent defect analysis engine. It includes a process parameter optimization calculation unit, a motion control card, and a compensation execution mechanism. It is used to generate compensation instructions based on the defect analysis results and drive the execution mechanism to move.

[0036] The data management and visualization platform communicates and connects with the aforementioned modules to store production process data, defect images, compensation records, and provides a human-machine interface, data reports, and real-time monitoring dashboards.

[0037] Optionally, the multispectral visual acquisition module specifically includes:

[0038] The optical imaging unit consists of at least one high-resolution area scan camera and one high-precision line scan camera, which are used for global scanning and high-speed, high-resolution imaging of key areas, respectively.

[0039] The adaptive lighting unit consists of multiple light sources whose brightness, angle, and polarization state can be independently controlled, and the emission spectrum of the light sources covers the visible to near-infrared band;

[0040] The optical adjustment unit, including a motorized zoom lens, polarizer, and filter wheel, can automatically switch imaging configurations according to the material and characteristics of the packaged structure being tested.

[0041] Optionally, the intelligent defect analysis engine specifically includes:

[0042] The preprocessing unit is used to perform denoising, enhancement, registration, and ROI region extraction operations on the input image;

[0043] The defect identification unit deploys a lightweight optimized deep learning model for performing defect localization and classification. The model integrates a convolutional neural network and a Transformer module.

[0044] The feature quantification and root cause analysis unit is used to calculate the quantitative features of defects such as size, location, and contrast, and to trace the potential process steps that caused the defects based on the causal inference model and historical process data.

[0045] Optionally, the real-time compensation control module specifically includes:

[0046] The compensation decision unit has a built-in hybrid decision model based on rules and reinforcement learning, which can select the optimal adjustment scheme from the process parameter library based on defect information.

[0047] A high-speed multi-axis motion control unit, integrated with the main control system of a bonding machine, pick-and-place machine, or dispensing machine, can achieve real-time position compensation at the micron level;

[0048] Miniature process execution units, including micro-dispensing valves, local laser heaters, or plasma spray guns, are used to add material or perform surface treatment on defective areas at specific points and in specific quantities without shutting down the system.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention effectively overcomes the lag inherent in traditional offline detection and manual intervention by constructing a closed-loop control system integrating "online detection, intelligent analysis, and real-time compensation." Its beneficial effects are as follows: First, through adaptive multi-mode visual fusion and hierarchical intelligent recognition, it significantly improves the detection rate and classification accuracy of complex and minute defects, achieving full coverage and high-sensitivity perception of various defects in advanced packaging processes. Second, utilizing a decision model based on digital twins and reinforcement learning, the system can generate and verify optimal compensation strategies in real time based on defect characteristics, driving the actuator to dynamically adjust process parameters online. This allows for precise compensation at the initial stage of defect occurrence or within the same batch, shifting from "post-event rejection" to "in-process correction and prevention," significantly reducing material scrap and rework costs. Finally, through a closed-loop data system and knowledge accumulation throughout the entire process, the system not only achieves a substantial improvement in production yield and intelligent optimization of process windows but also provides an intelligent quality assurance system for continuous self-improvement in advanced packaging manufacturing, which is of key value in promoting the industry towards "zero-defect" intelligent manufacturing. Attached Figure Description

[0051] Figure 1 This is a flowchart of the advanced online detection and compensation method for packaging defects based on machine vision proposed in this solution. Detailed Implementation

[0052] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0053] Reference Figure 1 As shown, an advanced online detection and compensation method for packaging defects based on machine vision includes:

[0054] In advanced packaging processes, multispectral vision acquisition modules are used to acquire images of intermediate or finished packaging products online, obtaining multimodal visual information about the packaging structure. By integrating multiple illumination modes such as bright field, dark field, ring light, and coaxial light, and combining them with imaging technologies of different wavelengths, these modules can simultaneously acquire multimodal visual information, including surface texture, edge contours, height differences, and specific material features, targeting complex features such as highly reflective metal surfaces, transparent dielectric layers, and minute three-dimensional structures in advanced packaging. This simultaneous acquisition of multiple information sources overcomes the limitations of incomplete information from a single imaging mode, providing a rich and complementary data foundation for subsequent high-precision, high-reliability defect analysis.

[0055] Multimodal visual information is input into the intelligent defect analysis engine to perform layered defect detection and identification. This includes locating, classifying, and assessing the severity of geometric defects, material defects, or potential electrical performance defects in the packaging structure. The layered detection strategy specifically includes: first, using traditional image algorithms for rapid initial screening to efficiently filter out defect-free areas; second, employing a deep learning-based object detection network to accurately locate and classify candidate areas, identifying defects such as missing bumps, bridging, and scratches; and finally, performing 3D morphological reconstruction and quantitative analysis on the identified defects to evaluate their key parameters such as size and depth, and determining their severity level in conjunction with process specifications and historical data. This "coarse-to-fine" layered architecture greatly improves overall processing efficiency while ensuring detection accuracy.

[0056] Based on the location, classification, and severity assessment of defects, the real-time compensation control module generates targeted compensation decisions. These decisions include at least the types and amounts of process parameters that need adjustment. The generation of these decisions relies on a pre-built, continuously evolving "defect-process parameter" knowledge base. The system matches the characteristics of the current defect with the knowledge base and uses rule-based reasoning or machine learning models (such as decision trees or neural networks) to quickly calculate the optimal compensation scheme. For example, for a "non-uniform solder ball height" defect, the decision might be "adjust the peak temperature of reflow soldering zone Z by +5°C" or "adjust the solder paste printing pressure by -0.1N." This process models and automates expert experience.

[0057] Based on compensation decisions, the system controls the compensation execution mechanism to dynamically adjust current or subsequent packaging process parameters online. This compensates for identified defects or prevents similar defects. The compensation execution mechanism is deeply integrated with key process equipment on the production line, such as die bonders, wire bonders, reflow ovens, and dispensing machines, or directly controls some of their execution units. The system translates compensation decisions into equipment-readable instructions, driving the equipment to adjust parameters in real time. For example, if a chip placement misalignment is detected, the visual alignment parameters for the next placement cycle can be fine-tuned immediately; if insufficient solder is detected, the dispensing machine can be controlled to precisely replenish solder in subsequent units. This achieves a shift from passive rejection to proactive correction.

[0058] After the compensation action is executed, image information of the packaged object is collected again to verify the compensation effect. The entire process of detection, compensation, and verification data is then synchronously updated to the data management and visualization platform. Post-compensation verification is a crucial step in quality assurance. By comparing the defect status before and after compensation, the effectiveness of the compensation action can be quantitatively evaluated. Regardless of whether the compensation is successful or not, all process data, including original images, defect features, compensation decisions, execution results, and verification images, is stored in a structured manner. This data is not only used for production traceability and quality analysis but can also be fed back to the intelligent defect analysis engine and compensation decision model as training samples to continuously optimize the algorithm, enabling the system to evolve and adapt to new processes and new defect types.

[0059] In some preferred embodiments, during the advanced packaging process, the use of a multispectral vision acquisition module to perform online image acquisition of intermediate or finished packaging products to obtain multimodal visual information of the packaging structure specifically includes:

[0060] Using at least two of the illumination modes of bright field illumination, dark field illumination and polarized light illumination, alternate or synchronous image acquisition is performed on the same package area to obtain images that reflect different features of surface morphology, edge contour and internal structure.

[0061] While acquiring two-dimensional images, three-dimensional topographic information of the key encapsulation area is obtained through laser triangulation, structured light or binocular vision methods.

[0062] For different advanced packaging process types, including through-silicon vias, redistribution layers, or microbumps, the illumination angle, light source wavelength, and camera parameters are adaptively configured to optimize the imaging quality of target features.

[0063] Specifically, the implementation method for using a multispectral vision acquisition module to perform online image acquisition of packaging intermediates or finished products to obtain multimodal visual information of the packaging structure is as follows:

[0064] The system drives the multispectral vision acquisition module to start working. The module integrates a programmable multi-angle ring LED light source, a coaxial light source, and a rotatable linear polarizer. For the same package area under test (e.g., a wafer with completed bump fabrication or a chip stack structure), the system control unit triggers bright-field illumination, dark-field illumination, and polarized illumination modes sequentially or in combination according to a preset sequence. Under bright-field illumination, the system primarily acquires macroscopic surface texture and color contrast information; after switching to dark-field illumination, minute scratches, dents, or contaminants on the surface become more prominent due to scattered light; simultaneously enabling polarized illumination effectively suppresses high-gloss reflections on metal surfaces and enhances the visibility of features such as dielectric layer edges or internal interfaces. Through the coordinated or alternating acquisition of these three modes, the system obtains an image set of the same area under various physical and optical properties, thus comprehensively covering visual features of different dimensions, including surface morphology, edge contours, and even subsurface interfaces.

[0065] While acquiring the aforementioned multimodal 2D images, the system activates an integrated 3D topography measurement unit. This unit typically employs laser triangulation or structured light projection. For example, a line laser beam is precisely projected onto a key feature of the package (such as a micro-bump array). Due to variations in surface height, the laser line observed by the camera will deform. By calculating this deformation, 3D topography information such as the height and coplanarity of the bumps can be reconstructed in real time. These 3D point cloud data are spatially rigorously registered with the high-resolution 2D images, together forming multimodal visual information describing the package structure in "2.5D" or even complete 3D.

[0066] To ensure optimal imaging quality, the system features adaptive parameter configuration. Before the inspection process begins or when changing product types, the system retrieves the corresponding imaging scheme from a pre-stored database of empirical parameters based on the process type (e.g., "Through Silicon Via (TSV) Sidewall Inspection" or "Redistribution Layer (RDL) Linewidth Measurement") issued by the host computer. For example, for TSV hole inspection, low-angle dark-field illumination may be prioritized to enhance the contrast of the hole wall edges; for micro-bump coplanarity inspection, it mainly relies on three-dimensional topography measurement, supplemented by coaxial light of a specific wavelength for surface condition monitoring. The system automatically adjusts parameters such as light source brightness, illumination angle combination, polarizer angle, camera exposure time, and gain to achieve the most significant contrast and clarity for the target defect or feature in the image, laying a high-quality image data foundation for subsequent intelligent analysis.

[0067] In some preferred embodiments, inputting multimodal visual information into the intelligent defect analysis engine to perform hierarchical defect detection and identification specifically includes:

[0068] Using traditional image processing algorithms based on threshold segmentation, edge detection, or template matching, the acquired images are analyzed quickly in real time to filter out areas with obvious anomalies;

[0069] Images containing abnormal regions are input into a deep learning-based defect detection neural network model. The model employs a multi-scale feature pyramid structure that incorporates an attention mechanism to accurately identify and locate defect types.

[0070] The identified defects are reconstructed in three dimensions and quantitatively analyzed. The severity of the defects is assessed by combining historical process data, and the evolution trend of the defects in subsequent processes is predicted by using a sequence model.

[0071] The specific implementation methods for layered defect detection and identification are as follows:

[0072] Rapid Initial Screening: After receiving the raw image from the multispectral vision acquisition module, the intelligent defect analysis engine first initiates a preprocessing and rapid analysis thread. This thread uses highly computationally efficient traditional digital image processing algorithms to analyze the image in real time. For example, an adaptive threshold segmentation algorithm is used for bright-field images to quickly separate the foreground (such as bumps or wires) from the background; the Canny edge detection operator is applied to dark-field images to capture abrupt edge changes or minor scratches with high sensitivity; simultaneously, the current image is compared with a pre-stored, rigorously calibrated "Golden Template" image using normalized cross-correlation matching to quickly calculate the difference regions. The design goal of this layer is "high throughput and low false negatives," enabling it to scan the entire field of view within milliseconds and filter out all regions of interest (ROIs) with obvious grayscale, texture, or shape anomalies. Image blocks containing these ROIs and their location coordinates are output to the next layer, while normal regions without anomalies are directly marked as qualified, thus greatly reducing the computational load of subsequent complex algorithms.

[0073] Precise Localization and Classification: Suspicious ROI image patches output from the first layer are fed into a deep learning-based defect detection neural network model for detailed analysis. This model employs an improved multi-scale feature pyramid network structure that incorporates an attention mechanism. Its backbone network (such as ResNet) extracts multi-level features from the input ROI image; the feature pyramid structure can simultaneously utilize shallow high-resolution features (facilitating the localization of small defects) and high-level abstract semantic features (facilitating classification). Embedded attention modules (such as SE-Net or CBAM) adaptively enhance the weights of defect feature channels and suppress complex background noise. Trained on a massive, annotated, advanced encapsulated defect dataset, this model can accurately output the defect's bounding box, category (such as "solder ball bridging," "RDL open circuit," "contaminant," etc.), and confidence level. This layer ensures high detection accuracy and low false positives, and is the core of qualitative and quantitative defect identification.

[0074] In-depth analysis and evaluation: For each defect instance identified in the second layer, the engine invokes the 3D topography reconstruction module. This module utilizes laser triangulation point cloud data synchronously acquired by the multispectral vision acquisition module to perform local 3D surface reconstruction of the defect area, accurately calculating the physical dimensions (such as length, width, depth, and volume), location coordinates, and deviations from design values. Subsequently, the defect quantification and evaluation unit compares these quantified data with preset, gradable standards (such as Class A: minor, Class B: requiring attention, Class C: severe) to automatically determine the severity level of the defect. Furthermore, the system can invoke a time-series prediction model (such as an LSTM network) to predict the evolution trend (such as void expansion and crack propagation) of the defect after undergoing subsequent high-temperature and high-pressure processes such as reflow soldering and molding, combining the characteristics of the current defect, historical data of this batch, and parameters of subsequent predetermined process steps, providing a forward-looking basis for compensation decisions. Thus, the system completes a full, layered intelligent analysis loop from "rapid anomaly detection" to "precise location and classification" and then to "in-depth quantification evaluation and prediction."

[0075] In some preferred embodiments, the training and optimization process of the defect detection neural network model includes:

[0076] We constructed a labeled image dataset containing various advanced encapsulation defect types and performed data augmentation operations on the dataset, including rotation, scaling, brightness adjustment, and noise addition.

[0077] A transfer learning strategy is adopted, using a model pre-trained on a large-scale general image dataset as the base network, and fine-tuning the training using a packaged defect dataset.

[0078] During the model inference stage, knowledge distillation or model pruning techniques are used to lightweight the network to meet the real-time requirements of online detection, and the optimized model parameters are deployed to edge computing devices through an online update mechanism.

[0079] The training and optimization process is the core technology ensuring that the intelligent detection system possesses high accuracy, strong generalization, and real-time processing capabilities. Constructing high-quality, diverse labeled datasets is the cornerstone of model performance. Since advanced encapsulation of defect samples is costly and varied in form in real production lines, systematic data augmentation (such as simulating brightness adjustments under different lighting conditions, adding blur and noise to simulate mechanical vibrations, and simulating rotation and scaling from different perspectives) can significantly expand the scale and diversity of the dataset, effectively simulating the complex and varied imaging conditions in production lines, thereby improving the model's adaptability to unknown samples and preventing overfitting. Employing transfer learning strategies has significant engineering value: directly and randomly initializing and training deep networks requires massive amounts of data and a long training period, while initializing with model parameters pre-trained on general datasets such as ImageNet is equivalent to giving the model basic general feature extraction capabilities. Based on this "knowledge," supervised fine-tuning using a relatively small-scale dedicated defect dataset allows the model to quickly and efficiently "transfer" its general feature recognition capabilities and "focus" on defect features, significantly reducing training data requirements and training time, and accelerating model deployment. Lightweighting and online update mechanisms for models are crucial for industrial deployment. Complex detection models may run well on standard servers, but when directly deployed to edge devices on the production line (such as industrial control computers and intelligent cameras with computing power), they face bottlenecks in computing power and real-time performance. Through knowledge distillation (using a large model to guide the training of a smaller model) or structured pruning (removing redundant neural connections in the network), the model size and computational load can be compressed by several to tens of times with minimal loss of accuracy (typically <1%), meeting the production line's cycle time requirements. Simultaneously, establishing an online model update mechanism allows for the secure and convenient pushing of newly trained and optimized model parameters from the cloud to the edge for updates, enabling the entire detection system to continuously learn and evolve, responding to new defect types and process changes. This complete technical chain, from "data construction" and "efficient training" to "lightweight deployment" and "continuous evolution," collectively ensures the practicality, reliability, and forward-looking nature of the defect detection engine in real industrial environments.

[0080] In some preferred embodiments, based on the location, classification, and severity assessment results of the defects, the real-time compensation control module generates targeted compensation decisions, specifically including:

[0081] Based on the type, size, location, and severity of the defect, a matching query is performed in the preset "defect feature-process parameter mapping relationship database" to obtain preliminary compensation strategy suggestions;

[0082] A compensation decision model is constructed using reinforcement learning algorithms. The model takes the current process state and defect information as state input, process parameter adjustment as action space, and the degree of quality improvement after compensation as reward. The compensation strategy is iteratively optimized through interaction with the environment.

[0083] Before implementing physical compensation, the compensation decision is input into the digital twin model of the packaging process for virtual verification, the compensation effect is predicted, and the compensation parameters are fine-tuned based on the simulation results.

[0084] The compensation decision generation mechanism is the core of this system's intelligent closed loop from "perception" to "execution." Its design transcends simple rule matching, constructing a progressive intelligent decision-making system that progresses from experience-based learning to autonomous optimization and then to virtual verification. First, the pre-set "defect feature-process parameter mapping database" serves as a static carrier of system knowledge and experience, constructed from the historical experience of process experts, experimental data, and extensive production records. When a defect is detected, the system performs a rapid matching query in this database, immediately providing a relatively reliable preliminary compensation strategy suggestion based on historical successful cases. This reflects case-based reasoning, ensuring the basic rationality and response speed of the decision, especially for common defects.

[0085] However, advanced packaging processes are complex and ever-changing, and static databases cannot cover all new situations or complex coupling defects. Therefore, the system introduces a dynamic decision-making model constructed using reinforcement learning algorithms. This model treats the production environment (including equipment status, material batches, current process parameters, and defect information) as the "environment," and various process parameter adjustment schemes (such as temperature ±ΔT, pressure ±ΔP, and position offset ±ΔX / Y) as "actions." By continuously "trial and error" in real production lines or high-fidelity simulation environments and receiving "reward" signals for quality improvement after compensation, the model autonomously explores and learns which process states, which defects, and which adjustments can achieve long-term optimal yield. This enables the system not only to handle known defects but also to adapt to new operating conditions and discover better process windows that human experts have not summarized, achieving self-evolution of decision-making capabilities.

[0086] To strike a balance between exploring optimization and ensuring current production safety and stability, the digital twin virtual verification process is crucial. Before any compensation decision generated by static rules or reinforcement learning models is sent to physical equipment for execution, the system inputs it into a high-fidelity digital twin model of the packaging process. This model integrates material properties, physicochemical equations, and equipment dynamics, enabling high-precision simulation of the instantaneous and subsequent changes in product state, defect morphology, and key quality indicators after compensation actions are executed. By predicting the compensation effect through simulation, the decision can be pre-assessed for risk assessment and fine-tuned, thus completing "trial and error" in virtual space to ensure that the compensation instructions ultimately sent to the actual production line are safe, effective, and optimal. The integration of this "triple decision-making mechanism" jointly guarantees the scientific nature, adaptability, and high reliability of the compensation decisions.

[0087] In some preferred embodiments, based on compensation decisions, the control compensation execution mechanism dynamically adjusts current or subsequent packaging process parameters online to compensate for identified defects or prevent similar defects. Specifically, this includes:

[0088] For geometric defects such as bonding position misalignment or misalignment, compensation is made by controlling a multi-axis precision motion platform to adjust the position and attitude of the bonding head, mounting head or lithography machine stage in real time.

[0089] For material defects such as insufficient solder amount and uneven bump height, compensation can be made by controlling the dispensing valve and the output of the inkjet head, or by adjusting the temperature profile and pressure parameters of the reflow soldering.

[0090] For potential electrical defects caused by contamination or oxidation, in-situ plasma cleaning or local flux spraying processes are triggered, and after compensation, rapid verification is performed using electrical test probes integrated into the production line.

[0091] The compensation execution process is the physical endpoint of this system's "detection-decision-execution" closed loop. Its core lies in precise, executable, and dynamic intervention that minimizes production disruption for different defect mechanisms. Firstly, from a technical logic perspective, compensation measures strictly correspond to the physical causes of defects: for geometric deviations, the system directly sends position offset commands to the high-precision motion control system, performing "feedforward compensation" in the next work cycle or for the next unit to correct alignment errors at the source; for material defects, it controls fluid drive or thermal parameters to adjust the physical conditions of material distribution or forming processes in real time, achieving precise correction of forming results (such as solder volume and bump shape); for potential failures, it proactively intervenes in auxiliary processes such as cleaning or activation to eliminate reliability risks, and combines in-situ electrical testing for immediate effect verification. This "targeted" execution logic ensures the direct effectiveness of the compensation measures. Secondly, from a control architecture perspective, compensation execution is not an isolated action, but a modular, configurable collection of microservices. The system deeply integrates with controllers of various process equipment (motion platforms, dispensing machines, reflow ovens, plasma cleaners) through standard industrial communication protocols (such as EtherCAT and PROFINET), translating abstract compensation decisions (such as "X-axis positive compensation +10μm") into specific instructions recognizable by the equipment. This architecture allows the system to flexibly adapt to diverse equipment combinations on different production lines. Furthermore, considering production cycle time, all compensation designs prioritize high speed and minimal intervention. For example, position compensation is completed in milliseconds, temperature profile adjustments take effect before adjacent products enter the temperature zone, and dispensing compensation is only applied to specific units, thus avoiding production line downtime and achieving true "online" dynamic adjustment. Ultimately, compensation results (whether it's the feedback position of the motion platform or the on-resistance of electrical tests) are collected in real time and fed back to the system, forming a complete "defect-compensation-result" data pair together with defect detection results. This continuously enriches the system's knowledge base, drives the continuous optimization of decision models and compensation parameters, and enables the system to evolve from "executing preset rules" to "learning the optimal process."

[0092] This solution also proposes an embodiment of an advanced online packaging defect detection and compensation system based on machine vision, including:

[0093] A multispectral vision acquisition module, comprising a high-resolution industrial camera array, a programmable multi-angle ring light source system, and beam-splitting optical components, is used for online acquisition of multimodal images of packaged objects;

[0094] The intelligent defect analysis engine communicates with the visual acquisition module and integrates a traditional image processing algorithm library and a deep learning-based defect detection model to perform layered defect analysis and recognition on the acquired images.

[0095] The real-time compensation control module, which communicates with the intelligent defect analysis engine, includes a process parameter optimization calculation unit, a motion control card, and a compensation actuator. It is used to generate compensation commands based on the defect analysis results and drive the actuator to move.

[0096] The data management and visualization platform communicates and connects with the aforementioned modules to store production process data, defect images, compensation records, and provides a human-machine interface, data reports, and real-time monitoring dashboards.

[0097] The multispectral vision acquisition module specifically includes:

[0098] The optical imaging unit consists of at least one high-resolution area scan camera and one high-precision line scan camera, which are used for global scanning and high-speed, high-resolution imaging of key areas, respectively.

[0099] The adaptive lighting unit consists of multiple light sources whose brightness, angle, and polarization state can be controlled independently, and the emission spectrum of the light sources covers the visible to near-infrared band;

[0100] The optical adjustment unit, including a motorized zoom lens, polarizer, and filter wheel, can automatically switch imaging configurations according to the material and characteristics of the packaged structure being tested.

[0101] The intelligent defect analysis engine specifically includes:

[0102] The preprocessing unit is used to perform denoising, enhancement, registration, and ROI region extraction operations on the input image;

[0103] The defect identification unit deploys a lightweight optimized deep learning model to perform defect localization and classification. The model integrates convolutional neural networks and Transformer modules.

[0104] The feature quantification and root cause analysis unit is used to calculate the quantitative features of defects such as size, location, and contrast, and to trace the potential process steps that caused the defects based on the causal inference model and historical process data.

[0105] The real-time compensation control module specifically includes:

[0106] The compensation decision unit has a built-in hybrid decision model based on rules and reinforcement learning, which can select the optimal adjustment scheme from the process parameter library based on defect information.

[0107] A high-speed multi-axis motion control unit, integrated with the main control system of a bonding machine, pick-and-place machine, or dispensing machine, can achieve real-time position compensation at the micron level;

[0108] Miniature process execution units, including micro-dispensing valves, local laser heaters, or plasma spray guns, are used to add material or perform surface treatment on defective areas at specific points and in specific quantities without shutting down the system.

[0109] In summary, the advantages of this invention are: by constructing a closed-loop control system integrating "online detection, intelligent analysis, and real-time compensation", it effectively overcomes the lag defects of traditional offline detection and manual intervention.

[0110] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. An advanced online detection and compensation method for packaging defects based on machine vision, characterized in that, include: In advanced packaging processes, multispectral vision acquisition modules are used to acquire images of intermediate or finished packaging products online to obtain multimodal visual information of the packaging structure. The multimodal visual information is input into the intelligent defect analysis engine to perform hierarchical defect detection and identification, including locating, classifying and assessing the severity of geometric defects, material defects or potential electrical performance defects in the packaging structure. Based on the location, classification, and severity assessment results of the defects, the real-time compensation control module generates targeted compensation decisions, which include at least the types and amounts of process parameters that need to be adjusted. Based on the compensation decision, the control compensation execution mechanism dynamically adjusts the current or subsequent packaging process parameters online to compensate for identified defects or prevent similar defects. After the compensation action is performed, the image information of the encapsulated object is collected again to verify the compensation effect, and the entire process data of this detection, compensation and verification is synchronously updated to the data management and visualization platform.

2. The advanced online detection and compensation method for packaging defects based on machine vision according to claim 1, characterized in that, In the advanced packaging process, the use of a multispectral vision acquisition module to perform online image acquisition of intermediate or finished packaging products to obtain multimodal visual information of the packaging structure specifically includes: Using at least two of the illumination modes of bright field illumination, dark field illumination and polarized light illumination, alternate or synchronous image acquisition is performed on the same package area to obtain images that reflect different features of surface morphology, edge contour and internal structure. While acquiring two-dimensional images, three-dimensional topographic information of the key encapsulation area is obtained through laser triangulation, structured light, or binocular vision methods. For different advanced packaging process types, including through-silicon vias, redistribution layers, or microbumps, the illumination angle, light source wavelength, and camera parameters are adaptively configured to optimize the imaging quality of target features.

3. The advanced online detection and compensation method for packaging defects based on machine vision according to claim 2, characterized in that, The step of inputting the multimodal visual information into the intelligent defect analysis engine to perform hierarchical defect detection and recognition specifically includes: Using traditional image processing algorithms based on threshold segmentation, edge detection, or template matching, the acquired images are analyzed quickly in real time to filter out areas with obvious anomalies; An image containing anomalies is input into a deep learning-based defect detection neural network model. The model employs a multi-scale feature pyramid structure that incorporates an attention mechanism to accurately identify and locate defect types. The identified defects are reconstructed in three dimensions and quantitatively analyzed. The severity of the defects is assessed by combining historical process data, and the evolution trend of the defects in subsequent processes is predicted by using a sequence model.

4. The advanced online detection and compensation method for packaging defects based on machine vision according to claim 3, characterized in that, The training and optimization process of the defect detection neural network model includes: We constructed a labeled image dataset containing various advanced encapsulation defect types and performed data augmentation operations on the dataset, including rotation, scaling, brightness adjustment, and noise addition. A transfer learning strategy is adopted, using a model pre-trained on a large-scale general image dataset as the base network, and fine-tuning the training using the encapsulation defect dataset. During the model inference stage, knowledge distillation or model pruning techniques are used to lightweight the network to meet the real-time requirements of online detection, and the optimized model parameters are deployed to edge computing devices through an online update mechanism.

5. The advanced online detection and compensation method for packaging defects based on machine vision according to claim 4, characterized in that, The process of generating targeted compensation decisions by the real-time compensation control module based on the location, classification, and severity assessment results of defects specifically includes: Based on the type, size, location, and severity of the defect, a matching query is performed in the preset "defect feature-process parameter mapping relationship database" to obtain preliminary compensation strategy suggestions; A compensation decision model is constructed using reinforcement learning algorithms. The model takes the current process state and defect information as state input, process parameter adjustment as action space, and the degree of quality improvement after compensation as reward. The compensation strategy is iteratively optimized through interaction with the environment. Before implementing physical compensation, the compensation decision is input into the digital twin model of the packaging process for virtual verification, the compensation effect is predicted, and the compensation parameters are fine-tuned based on the simulation results.

6. The advanced online detection and compensation method for packaging defects based on machine vision according to claim 5, characterized in that, Based on the compensation decision, controlling the compensation execution mechanism to dynamically adjust the current or subsequent packaging process parameters online, in order to compensate for identified defects or prevent similar defects, specifically includes: For geometric defects such as bonding position misalignment or misalignment, compensation is made by controlling a multi-axis precision motion platform to adjust the position and attitude of the bonding head, mounting head or lithography machine stage in real time. For material defects such as insufficient solder amount and uneven bump height, compensation can be made by controlling the dispensing valve and the output of the inkjet head, or by adjusting the temperature profile and pressure parameters of the reflow soldering. For potential electrical defects caused by contamination or oxidation, in-situ plasma cleaning or local flux spraying processes are triggered, and after compensation, rapid verification is performed using electrical test probes integrated into the production line.

7. An advanced packaging defect online detection and compensation system based on machine vision, used to implement the advanced packaging defect online detection and compensation method based on machine vision as described in any one of claims 1-6, comprising: A multispectral vision acquisition module, comprising a high-resolution industrial camera array, a programmable multi-angle ring light source system, and beam-splitting optical components, is used for online acquisition of multimodal images of packaged objects; The intelligent defect analysis engine is connected to the visual acquisition module and integrates a traditional image processing algorithm library and a deep learning-based defect detection model for performing layered defect analysis and identification on the acquired images. The real-time compensation control module, which is communicatively connected to the intelligent defect analysis engine, includes a process parameter optimization calculation unit, a motion control card, and a compensation execution mechanism. It is used to generate compensation instructions based on the defect analysis results and drive the execution mechanism to move. The data management and visualization platform communicates and connects with the aforementioned modules to store production process data, defect images, compensation records, and provides a human-machine interface, data reports, and real-time monitoring dashboards.

8. The advanced online detection and compensation system for packaging defects based on machine vision according to claim 7, characterized in that, The multispectral vision acquisition module specifically includes: The optical imaging unit consists of at least one high-resolution area scan camera and one high-precision line scan camera, which are used for global scanning and high-speed, high-resolution imaging of key areas, respectively. The adaptive lighting unit consists of multiple light sources whose brightness, angle, and polarization state can be independently controlled, and the emission spectrum of the light sources covers the visible to near-infrared band; The optical adjustment unit, including a motorized zoom lens, polarizer, and filter wheel, can automatically switch imaging configurations according to the material and characteristics of the packaged structure being tested.

9. The advanced online detection and compensation system for packaging defects based on machine vision according to claim 7, characterized in that, The intelligent defect analysis engine specifically includes: The preprocessing unit is used to perform denoising, enhancement, registration, and ROI region extraction operations on the input image; The defect identification unit deploys a lightweight optimized deep learning model for performing defect localization and classification. The model integrates a convolutional neural network and a Transformer module. The feature quantification and root cause analysis unit is used to calculate the quantitative features of defects such as size, location, and contrast, and to trace the potential process steps that caused the defects based on the causal inference model and historical process data.

10. The advanced online detection and compensation system for packaging defects based on machine vision according to claim 7, characterized in that, The real-time compensation control module specifically includes: The compensation decision unit has a built-in hybrid decision model based on rules and reinforcement learning, which can select the optimal adjustment scheme from the process parameter library based on defect information. A high-speed multi-axis motion control unit, integrated with the main control system of a bonding machine, pick-and-place machine, or dispensing machine, can achieve real-time position compensation at the micron level; Miniature process execution units, including micro-dispensing valves, local laser heaters, or plasma spray guns, are used to add material or perform surface treatment on defective areas at specific points and in specific quantities without shutting down the system.

Images