Chip advanced packaging process quality online detection method and system based on multi-modal data fusion

Abstract

The application discloses a multi-modal data fusion chip advanced packaging process quality online detection method and system, relates to the chip detection technical field, and comprises the following steps: constructing a mode-defect confidence matrix based on historical multi-modal detection data; synchronously or quasi-synchronously collecting detection data of multiple modes; obtaining preliminary defect detection results output by each mode; for each suspected defect type, calculating a comprehensive confidence score of the defect type according to the mode-defect confidence matrix; and comparing the comprehensive confidence score with a preset threshold, and outputting a final defect determination result according to the comparison result. The application has the advantages that by constructing and applying the mode-defect confidence matrix, different fusion weights are dynamically allocated to different modes for specific defect types in multi-modal fusion decision, and the comprehensive performance of the detection system can be significantly improved.

Description

Technical Field

[0001] This invention relates to the field of chip testing technology, specifically to an online testing method and system for advanced chip packaging process quality based on multimodal data fusion. Background Technology

[0002] Advanced chip packaging technology is crucial for realizing high-density, high-performance integrated circuits, and its process quality directly determines the yield and reliability of the final product. Current quality inspection relies heavily on single or a few inspection technologies, such as X-rays and optical imaging. However, single-modal inspection methods often have limitations when dealing with complex and hidden defect types, leading to missed detections or misjudgments. Even with multimodal inspection, existing methods are typically limited to simply presenting multi-source data in parallel or performing basic fusion based on fixed rules. They fail to deeply evaluate and utilize the differentiated detection capabilities of different inspection technologies (modals) for various specific defects, making it difficult for the overall accuracy, stability, and intelligence level of the inspection system to meet the requirements of modern precision manufacturing.

[0003] More specifically, existing multimodal inspection schemes typically treat all modal data equally or employ static, empirical weights during fusion decision-making. This approach ignores the different physical principles underlying various inspection modalities, leading to inherent differences in their sensitivity and reliability to different defects. For example, X-rays are sensitive to internal cavities, while optical imaging is more effective at detecting surface scratches. Fixed fusion strategies cannot adaptively adjust the contribution of each modal data according to specific defect types. This not only fails to maximize the efficiency of the multimodal system but also directly reduces the robustness of the entire system when a particular modality is disturbed or experiences performance fluctuations. Therefore, there is an urgent need for an intelligent fusion method that can dynamically evaluate and utilize the specific detection confidence of each modality to achieve more accurate and reliable online quality assessment. Summary of the Invention

[0004] To address the aforementioned technical problems, this technical solution provides an online inspection method and system for advanced chip packaging process quality based on multimodal data fusion. This solution addresses at least one of the technical problems mentioned in the background section.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] Online quality inspection methods for advanced chip packaging processes based on multimodal data fusion include:

[0007] Based on historical multimodal detection data and their corresponding confirmed real defect type labels, the detection confidence of each detection mode for each defect type is calculated, and a modality-defect confidence matrix is ​​constructed.

[0008] For the chip package currently being inspected, multiple modal inspection data are collected simultaneously or quasi-synchronously;

[0009] The detection data of each mode are input into the corresponding single-mode defect detection model to obtain the preliminary defect detection results output by each mode. The preliminary defect detection results include at least the suspected defect type and its corresponding preliminary confidence score.

[0010] For each suspected defect type, based on the modality-defect confidence matrix, corresponding fusion weights are assigned to each modality that reported the suspected defect type, and the preliminary confidence scores given by each modality are weighted and fused to calculate the comprehensive confidence score of the defect type.

[0011] The overall confidence score is compared with a preset threshold, and the final defect determination result is output based on the comparison result.

[0012] Preferably, the step of calculating the detection confidence of each detection modality for each defect type based on historical multimodal detection data and its corresponding confirmed true defect type labels, and constructing the modality-defect confidence matrix specifically includes:

[0013] Collect historical datasets, which contain multiple sets of samples. Each set of samples includes: data of the same chip package under different detection modes, as well as annotation information of the actual defect type and location that has been finally confirmed.

[0014] For each defect type, the historical detection performance indicators of each modality on that defect type are statistically analyzed, and the performance indicators include at least the detection rate and the accuracy rate.

[0015] Based on the performance metrics, a numerical confidence level is assigned to each "detection mode-defect type" pair through quantitative calculation. All confidence levels form a two-dimensional matrix, where one-dimensional index represents different detection modes and the other-dimensional index represents different defect types.

[0016] Preferably, the step of assigning a numerical confidence level to each "detection modality-defect type" pair based on the performance index through quantitative calculation specifically includes:

[0017] For a given detection modality i and defect type j, select a subset of all samples with true defects j from the historical dataset;

[0018] Calculate the detection rate of mode i in this subset of samples, that is, the proportion of samples in which mode i correctly reports defect j out of the total number of samples;

[0019] Calculate the accuracy of mode i on this subset of samples, that is, the proportion of results in mode i that are actually defect j among all results reported as defect j;

[0020] The detection rate and accuracy are combined to calculate the initial confidence level of mode i for defect j. The combination method includes, but is not limited to, taking the average value, taking the weighted sum, or calculating based on the F1 score.

[0021] Preferably, for each suspected defect type, based on the modality-defect confidence matrix, corresponding fusion weights are assigned to each modality that reported the suspected defect type, and the preliminary confidence scores given by each modality are weighted and fused to calculate the comprehensive confidence score for the defect type. Specifically:

[0022] For the current object being inspected, identify the set of modes that report defect type k in all preliminary inspection results. ;

[0023] For sets For each mode i, query its offline confidence score for defect type k from the mode-defect confidence matrix. ;

[0024] Based on the offline confidence level Determine the dynamic weight of mode i in this fusion. ,in yes A monotonically increasing function;

[0025] Calculate the overall confidence score for defect type k. ,in The initial confidence score for defect k is given for mode i.

[0026] Preferably, the offline confidence level is used as the basis for... Determine the dynamic weight of mode i in this fusion. The methods include one of the following:

[0027] Method 1: Directly use offline confidence scores as dynamic weights, i.e. ;

[0028] Method 2 involves scaling and shifting the offline confidence scores to create dynamic weights. , where a and b are preset coefficients;

[0029] Method 3 involves inputting the offline confidence score into a preset nonlinear mapping function, and using its output as dynamic weights.

[0030] Preferably, the method further includes a model update phase:

[0031] The final defect judgment results output during the online detection phase are added to the historical dataset as new labeled samples after being manually reviewed or confirmed by higher precision equipment.

[0032] Periodically or when a certain number of new samples are accumulated, based on the updated historical dataset, the detection confidence of each detection modality for each defect type is recalculated based on the historical multimodal detection data and its corresponding confirmed real defect type labels, so as to update the modality-defect confidence matrix.

[0033] Furthermore, a multimodal data fusion-based online quality inspection system for advanced chip packaging processes is proposed to implement the aforementioned multimodal data fusion-based online quality inspection method for advanced chip packaging processes, including:

[0034] The data acquisition module is configured to synchronously or quasi-synchronously trigger and acquire data from multiple detection modes at the production line inspection station.

[0035] An edge computing unit is connected to the data acquisition module and is configured to preprocess the raw data of each modality, run each single-modality defect detection model, obtain and upload preliminary defect detection results;

[0036] A converged analytics server, communicatively connected to the edge computing unit, includes:

[0037] A confidence model library is used to store and maintain the modality-defect confidence matrix.

[0038] The multimodal fusion decision engine is configured to receive the preliminary defect detection results of each modality, query the confidence model library, execute the weighted fusion algorithm, calculate the comprehensive confidence score, and make the final defect judgment.

[0039] The human-computer interaction and application module is connected to the fusion analysis server and is configured to display the final detection results, comprehensive confidence level, trigger alarms, and generate quality reports.

[0040] Optionally, the detection modes integrated by the data acquisition module include at least the following three: X-ray imaging, optical appearance imaging, and infrared thermal imaging; the edge computing unit includes multiple parallel processing sub-units, each of which is dedicated to processing data of one mode and running the corresponding single-mode detection model.

[0041] Optionally, the fusion analysis server further includes an online model update unit, configured for:

[0042] Receive confirmed online detection results and corresponding multimodal raw data, and store them as new samples in the historical database;

[0043] Monitor the number of new samples. When the number reaches a preset threshold, automatically trigger or be manually triggered by the administrator to recalculate the confidence model.

[0044] The offline modeling program is invoked to recalculate and generate a new modality-defect confidence matrix based on the updated historical database, which replaces the old matrix in the confidence model library.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] This invention, by constructing and applying a modality-defect confidence matrix, dynamically assigns differentiated fusion weights to different modalities for specific defect types in multimodal fusion decision-making, which can significantly improve the overall performance of the detection system. Its beneficial effects are specifically reflected in:

[0047] By giving higher decision weight to modalities with more reliable historical performance in the determination of corresponding defects, the overall accuracy and reliability of the final defect identification are greatly improved, and false alarms and false negatives are effectively reduced.

[0048] The system has excellent fault tolerance and robustness. When the performance of a certain mode temporarily declines due to its own fluctuations or environmental interference, its decision weight will be automatically reduced, thereby ensuring the continuous and stable operation of the detection process.

[0049] This dynamic weighting mechanism makes the final decision highly interpretable and helps in quality root cause analysis;

[0050] The confidence model supports online updates, enabling the system to have self-learning and self-adaptive capabilities. It can continuously track and adapt to process changes, maintain optimal detection performance over a long period of time, and thus provide a high-precision, highly stable, and intelligent online detection solution for quality control of advanced chip packaging processes. Attached Figure Description

[0051] Figure 1 This is a flowchart of the online quality inspection method for advanced chip packaging processes based on multimodal data fusion proposed in this invention. Detailed Implementation

[0052] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0053] Reference Figure 1 As shown, the online quality inspection method for advanced chip packaging processes based on multimodal data fusion includes:

[0054] Based on historical multimodal detection data and their corresponding confirmed true defect type labels, the detection confidence of each detection modality for each defect type is calculated, constructing a modality-defect confidence matrix. Through offline analysis of historical data, the inherent detection capabilities and reliability differences of different detection technologies (i.e., "modalities," such as X-rays and infrared) for different types of defects (such as voids and bridging) are quantitatively evaluated. This transforms expert experience or data patterns into a computable, structured knowledge model (i.e., the confidence matrix). This matrix enables the system to avoid treating all modal signals equally or indiscriminately during subsequent online fusion. Instead, it allows for differentiated reliance on evidence provided by different modalities based on the specific suspected defect type, providing an objective, data-driven basis for dynamic weight allocation.

[0055] For the chip package currently being inspected, multiple modal detection data are acquired synchronously or quasi-synchronously. Synchronous or quasi-synchronous acquisition is a key technical prerequisite for ensuring the spatiotemporal consistency of data. It avoids errors introduced by changes in the object's state (such as temperature drift and positional changes) due to asynchronous detection time. The types of acquired modalities are usually complementary, such as X-rays reflecting internal structure, optical imaging reflecting surface features, and infrared imaging reflecting heat distribution, to comprehensively cover various physical signals that may cause defects, providing raw materials for subsequent multi-angle and complementary analysis.

[0056] The detection data from each modality are input into the corresponding single-modal defect detection model to obtain preliminary defect detection results for each modality. These preliminary results include at least the suspected defect type and its corresponding preliminary confidence score. Each single-modal detection model (such as a trained deep learning network or a dedicated image processing algorithm) specializes in identifying defect patterns it excels at from specific types of data and outputs a preliminary judgment with a "preliminary confidence score." This score reflects the modality model's own confidence in its judgment. This step transforms the raw data into structured, fusionable preliminary semantic information, providing standardized input for subsequent cross-modal decision fusion. The independent operation of each modality model also reflects the modular design of this scheme, facilitating updates and maintenance.

[0057] For each suspected defect type, based on the modality-defect confidence matrix, corresponding fusion weights are assigned to each modality that reported the suspected defect type. The preliminary confidence scores given by each modality are then weighted and fused to calculate the comprehensive confidence score for that defect type. For a specific suspected defect type K, the system queries the confidence matrix to find the modality that has historically been most reliable in detecting K types of defects and assigns it a higher fusion weight. For example, if an internal cavity is suspected, the weight of the X-ray modality is automatically increased; if a surface scratch is suspected, the weight of the optical modality is dominant. Subsequently, the preliminary scores of each modality are weighted and summed to obtain a "comprehensive confidence score." This process simulates an expert team, where the most experienced expert on different issues has greater authority, making the final comprehensive judgment more accurate and reliable than any single modality's judgment.

[0058] The comprehensive confidence score is compared with a preset threshold, and the final defect judgment result is output based on the comparison result. By comparing the calculated "comprehensive confidence score," which reflects multimodal weighted consensus, with a preset judgment threshold, a binary decision of "qualified" or "a certain type of defect exists" is made. This threshold can be adjusted according to the stringency of quality control (such as AQL standards). This step transforms the complex multimodal information fusion process into a clear, explicit, and executable quality inspection instruction, which can be directly linked with the production line control system (such as alarms, sorting, and marking), realizing an online, automatic, and intelligent quality inspection closed loop.

[0059] In some preferred embodiments, the step of calculating the detection confidence of each detection modality for each defect type based on historical multimodal detection data and its corresponding confirmed true defect type labels, and constructing a modality-defect confidence matrix specifically includes:

[0060] Collect historical datasets, which contain multiple sets of samples. Each set of samples includes: data of the same chip package under different detection modes, as well as annotation information of the actual defect type and location that has been finally confirmed.

[0061] For each defect type, the historical detection performance indicators of each modality on that defect type are statistically analyzed, and the performance indicators include at least the detection rate and the accuracy rate.

[0062] Based on the performance metrics, a numerical confidence level is assigned to each "detection mode-defect type" pair through quantitative calculation. All confidence levels form a two-dimensional matrix, where one-dimensional index represents different detection modes and the other-dimensional index represents different defect types.

[0063] This process achieves a refined transformation from raw historical data to a computable knowledge model. First, collecting multimodal aligned data labeled with "real defects" is fundamental to ensuring the validity of subsequent statistics, providing a gold standard for evaluating the "right" or "wrong" performance of each modality. Second, "detection rate" and "accuracy" are selected as core performance indicators, comprehensively and objectively quantifying the detection effectiveness of each modality for specific defects from two complementary dimensions: "ability to detect defects" (avoiding missed detections) and "accuracy in judging correctness" (avoiding false positives). Finally, these statistical indicators are transformed into single confidence scores using formulas (such as weighted average, F1 score calculation, or other synthesis methods), ultimately constructing a structured "modality-defect confidence matrix." This matrix is ​​essentially the "experience knowledge base" of the entire system. In a data-driven manner, it digitizes and makes explicit the strengths and weaknesses of different detection methods, providing a fundamental basis for intelligent and differentiated integration in the online phase.

[0064] The method for constructing this confidence matrix represents a leap from qualitative experience to quantitative evidence. It abandons the traditional approach of relying on subjective expert experience to assign fixed weights, instead using objective historical data statistics to assign precise and quantifiable confidence levels to each "modality-defect" pair, making the foundation of fusion decisions more scientific and reliable. It provides multi-dimensional, fine-grained evaluation. By comprehensively considering "detection rate" and "accuracy rate," this method can effectively distinguish whether a modality is "detectable but frequently misjudged" or "accurately judged but prone to missed detections," thus deriving a more comprehensive confidence evaluation and guiding a more balanced fusion strategy. It lays the foundation for system self-optimization and interpretability. Each value in the matrix has a clear historical statistical meaning, making the final fusion decision traceable and analyzable. Furthermore, the matrix can be periodically updated with the accumulation of new data, enabling the system to continuously learn and adapt to process changes, ensuring its long-term advanced nature and accuracy.

[0065] In some preferred embodiments, the step of assigning a numerical confidence level to each "detection modality-defect type" pair based on the performance index through quantitative calculation specifically includes:

[0066] For a given detection modality i and defect type j, select a subset of all samples with true defects j from the historical dataset;

[0067] Calculate the detection rate of mode i in this subset of samples, that is, the proportion of samples in which mode i correctly reports defect j out of the total number of samples;

[0068] Calculate the accuracy of mode i on this subset of samples, that is, the proportion of results in mode i that are actually defect j among all results reported as defect j;

[0069] The detection rate and accuracy are combined to calculate the initial confidence level of mode i for defect j. The combination method includes, but is not limited to, taking the average value, taking the weighted sum, or calculating based on the F1 score.

[0070] Through rigorous mathematical definitions, the performance of a specific "modality-defect" pair is profiled from two orthogonal dimensions: "the ability to detect defects" (detection rate) and "the accuracy of correct judgments" (accuracy rate). Detection rate focuses on "no missed detections," assessing the modality's sensitivity and coverage of real-world defects; accuracy focuses on "no false positives," assessing the reliability of the modality's positive judgments. Combining these two metrics (e.g., by calculating their harmonic mean F1 score) yields a balanced, single confidence score. This approach effectively avoids biases that may arise from relying on a single indicator. For example, a modality might prioritize a high detection rate over accurate detection, leading to extremely low accuracy. This ensures that the final confidence score comprehensively and objectively reflects the modality's overall ability to identify the defect, providing a stable and balanced basis for subsequent weighted fusion.

[0071] This method, which comprehensively quantifies detection rate and accuracy, offers significant advantages. It ensures the objectivity and scientific rigor of confidence assessment. Statistical calculations based entirely on historical data completely eliminate the arbitrariness and uncertainty arising from subjective experience-based weighting, making the constructed confidence matrix robust and reliable. It achieves completeness and balance in the assessment dimensions. By simultaneously considering the two most critical risks in quality control—"missed detections" and "false detections"—and using comprehensive indicators such as the F1 score, it naturally assigns higher evaluations to modalities that perform well in both aspects, guiding the fusion system towards an overall optimal direction of high precision and high recall. It greatly enhances the system's interpretability and optimizability. Since each confidence value has a clear statistical source, engineers can clearly analyze the shortcomings of a particular modality in judging a certain type of defect, thereby enabling targeted improvements to the detection algorithm or parameters of that modality, achieving a closed loop from problem diagnosis to performance improvement.

[0072] In some preferred embodiments, for each suspected defect type, according to the modality-defect confidence matrix, assigning corresponding fusion weights to each modality that reported the suspected defect type, and weighting and fusing the preliminary confidence scores given by each modality to calculate the comprehensive confidence score for the defect type, specifically is as follows:

[0073] For the current object being inspected, identify the set of modes that report defect type k in all preliminary inspection results. ;

[0074] For sets For each mode i, query its offline confidence score for defect type k from the mode-defect confidence matrix. ;

[0075] Based on the offline confidence level Determine the dynamic weight of mode i in this fusion. ,in yes A monotonically increasing function;

[0076] Calculate the overall confidence score for defect type k. ,in The initial confidence score for defect k is given for mode i.

[0077] It completes the transformation from static knowledge (confidence matrix) to dynamic decision-making (weight allocation and fusion). Its innovation lies in the fact that the weight allocation is highly "contextualized": the system does not use the same set of fixed weights for all defect types, but for the specific defect type k that is currently suspected, it "retrieves" the historical performance of each modality specifically for this type of defect from the confidence matrix, and dynamically generates weights based on this. yes The function is a monotonically increasing function, which ensures that modalities with more reliable historical performance have greater influence in current decision-making. Finally, a comprehensive confidence score is calculated by weighting the initial confidence scores Si,k given by each modality with dynamic weights. This score not only integrates the "instantaneous opinions" of each modality's current observations but also incorporates their long-term accumulated "professional reputation," thus simulating an expert consultation process led by domain experts for a specific problem, making the final decision more authoritative and accurate.

[0078] This dynamic weighted fusion mechanism brings several significant benefits. It greatly improves the accuracy and reliability of the final decision. By giving higher weight to modalities with better historical performance in the determination of corresponding defects, the system can effectively suppress interference from modalities that are insensitive to or prone to errors in determining such defects. This makes the fusion result more inclined to adopt the judgment of the most reliable evidence source, thus achieving overall detection performance surpassing any single modality. It endows the system with excellent fault tolerance and adaptive robustness. When the performance of a modal temporarily declines due to equipment status, environmental interference, etc., its offline confidence for a specific defect is naturally lower, resulting in a lower dynamic weight in this fusion. This automatically weakens the impact of its erroneous judgment on the final result, ensuring the stable operation of the system as a whole. It effectively avoids the decision-making rigidity of static or average fusion strategies, allowing the fusion strategy to flexibly respond to specific defect situations and achieve optimal resource allocation. This mechanism, based on explicit numerical values... The weighting method makes the final "overall confidence score" highly interpretable, allowing quality inspectors to clearly understand the decision-making basis and facilitating root cause analysis and process traceability.

[0079] In some preferred embodiments, the offline confidence level is used as the basis for... Determine the dynamic weight of mode i in this fusion. The methods include one of the following:

[0080] Method 1: Directly use offline confidence scores as dynamic weights, i.e. ;

[0081] Method 2 involves scaling and shifting the offline confidence scores to create dynamic weights. , where a and b are preset coefficients;

[0082] Method 3 involves inputting the offline confidence score into a preset nonlinear mapping function, and using its output as dynamic weights.

[0083] In some preferred embodiments, the method further includes a model update phase:

[0084] The final defect judgment results output during the online detection phase are added to the historical dataset as new labeled samples after being manually reviewed or confirmed by higher precision equipment.

[0085] Periodically or when a certain number of new samples are accumulated, based on the updated historical dataset, the detection confidence of each detection modality for each defect type is recalculated based on the historical multimodal detection data and its corresponding confirmed real defect type labels, so as to update the modality-defect confidence matrix.

[0086] The model update phase is the core mechanism for achieving continuous self-optimization and long-term adaptability of the detection system, forming a complete iterative closed loop of "offline modeling → online application → data accumulation → model update". The core value of this mechanism lies in transforming high-value data generated by online detection into fuel for system evolution. By continuously incorporating this data, labeled with "new truths," into historical datasets, the system's knowledge base is continuously expanded and revised. Subsequently, through periodic or triggered recalculations, the "modality-defect confidence matrix" can be dynamically refreshed. This process enables the system to: track and adapt to slow drifts in production processes or batch changes in raw materials; promptly adjust the weight of a detection modality in fusion decisions after hardware performance degradation or maintenance calibration; and continuously improve its knowledge graph as new defect types emerge or accumulate. Therefore, this update phase ensures that the entire detection system is not a static, "dead" system whose performance begins to decline after deployment, but a "living" intelligent system that can continuously learn from production practice, keep pace with the times, and maintain optimal detection efficiency, thereby guaranteeing the reliability and stability of detection accuracy throughout the product's entire lifecycle.

[0087] This solution also proposes an online quality inspection system for advanced chip packaging processes based on multimodal data fusion, used to implement the online quality inspection method for advanced chip packaging processes based on multimodal data fusion as described above, including:

[0088] The data acquisition module is configured to synchronously or quasi-synchronously trigger and acquire data from multiple detection modes at the production line inspection station.

[0089] An edge computing unit is connected to the data acquisition module and is configured to preprocess the raw data of each modality, run each single-modality defect detection model, obtain and upload preliminary defect detection results;

[0090] A converged analytics server, communicatively connected to the edge computing unit, includes:

[0091] A confidence model library is used to store and maintain the modality-defect confidence matrix.

[0092] The multimodal fusion decision engine is configured to receive the preliminary defect detection results of each modality, query the confidence model library, execute the weighted fusion algorithm, calculate the comprehensive confidence score, and make the final defect judgment.

[0093] The human-computer interaction and application module is connected to the fusion analysis server and is configured to display the final detection results, comprehensive confidence level, trigger alarms, and generate quality reports.

[0094] The data acquisition module integrates at least three detection modes: X-ray imaging, optical appearance imaging, and infrared thermal imaging; the edge computing unit includes multiple parallel processing sub-units, each of which is dedicated to processing data of one mode and running the corresponding single-mode detection model.

[0095] The fusion analysis server also includes an online model update unit, configured for:

[0096] Receive confirmed online detection results and corresponding multimodal raw data, and store them as new samples in the historical database;

[0097] Monitor the number of new samples. When the number reaches a preset threshold, automatically trigger or be manually triggered by the administrator to recalculate the confidence model.

[0098] The offline modeling program is invoked to recalculate and generate a new modality-defect confidence matrix based on the updated historical database, which replaces the old matrix in the confidence model library.

[0099] In summary, the advantages of this invention are: by constructing and applying a modality-defect confidence matrix, different fusion weights are dynamically assigned to different modalities for specific defect types in multimodal fusion decision-making, which can significantly improve the overall performance of the detection system.

[0100] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for online quality inspection of advanced chip packaging processes based on multimodal data fusion, characterized in that, include: Based on historical multimodal detection data and their corresponding confirmed real defect type labels, the detection confidence of each detection mode for each defect type is calculated, and a modality-defect confidence matrix is ​​constructed. For the chip package currently being inspected, multiple modal inspection data are collected simultaneously or quasi-synchronously; The detection data of each mode are input into the corresponding single-mode defect detection model to obtain the preliminary defect detection results output by each mode. The preliminary defect detection results include at least the suspected defect type and its corresponding preliminary confidence score. For each suspected defect type, based on the modality-defect confidence matrix, corresponding fusion weights are assigned to each modality that reported the suspected defect type, and the preliminary confidence scores given by each modality are weighted and fused to calculate the comprehensive confidence score of the defect type. The overall confidence score is compared with a preset threshold, and the final defect determination result is output based on the comparison result.

2. The online quality inspection method for advanced chip packaging processes based on multimodal data fusion according to claim 1, characterized in that, The process of calculating the detection confidence level of each detection modality for each defect type based on historical multimodal detection data and its corresponding confirmed true defect type labels, and constructing the modality-defect confidence matrix specifically includes: Collect historical datasets, which contain multiple sets of samples. Each set of samples includes: data of the same chip package under different detection modes, as well as annotation information of the actual defect type and location that has been finally confirmed. For each defect type, the historical detection performance indicators of each modality on that defect type are statistically analyzed, and the performance indicators include at least the detection rate and the accuracy rate. Based on the performance metrics, a numerical confidence level is assigned to each "detection mode-defect type" pair through quantitative calculation. All confidence levels form a two-dimensional matrix, where one-dimensional index represents different detection modes and the other-dimensional index represents different defect types.

3. The online quality inspection method for advanced chip packaging processes based on multimodal data fusion according to claim 2, characterized in that, The step of assigning a numerical confidence level to each "detection modality-defect type" pair based on the performance index through quantitative calculation specifically includes: For a given detection modality i and defect type j, select a subset of all samples with true defects j from the historical dataset; Calculate the detection rate of mode i in this subset of samples, that is, the proportion of samples in which mode i correctly reports defect j out of the total number of samples; Calculate the accuracy of mode i on this subset of samples, that is, the proportion of results in mode i that are actually defect j among all results reported as defect j; The detection rate and accuracy are combined to calculate the initial confidence level of mode i for defect j. The combination method includes, but is not limited to, taking the average value, taking the weighted sum, or calculating based on the F1 score.

4. The online quality inspection method for advanced chip packaging processes based on multimodal data fusion according to claim 3, characterized in that, For each suspected defect type, based on the modality-defect confidence matrix, corresponding fusion weights are assigned to each modality that reported the suspected defect type, and the preliminary confidence scores given by each modality are weighted and fused to calculate the comprehensive confidence score for the defect type. For the current object being inspected, identify the set of modes that report defect type k in all preliminary inspection results. ; For sets For each mode i, query its offline confidence score for defect type k from the mode-defect confidence matrix. ; Based on the offline confidence level Determine the dynamic weight of mode i in this fusion. ,in yes A monotonically increasing function; Calculate the overall confidence score for defect type k. ,in The initial confidence score for defect k is given for mode i.

5. The online quality inspection method for advanced chip packaging processes based on multimodal data fusion according to claim 4, characterized in that, The offline confidence level Determine the dynamic weight of mode i in this fusion. The methods include one of the following: Method 1: Directly use offline confidence scores as dynamic weights, i.e. ; Method 2 involves scaling and shifting the offline confidence scores to create dynamic weights. , where a and b are preset coefficients; Method 3 involves inputting the offline confidence score into a preset nonlinear mapping function, and using its output as dynamic weights.

6. The online quality inspection method for advanced chip packaging processes based on multimodal data fusion according to claim 5, characterized in that, It also includes the model update phase: The final defect judgment results output during the online detection phase are added to the historical dataset as new labeled samples after being manually reviewed or confirmed by higher precision equipment. Periodically or when a certain number of new samples are accumulated, based on the updated historical dataset, the detection confidence of each detection modality for each defect type is recalculated based on the historical multimodal detection data and its corresponding confirmed real defect type labels, so as to update the modality-defect confidence matrix.

7. A multimodal data fusion-based online quality inspection system for advanced chip packaging processes, used to implement the multimodal data fusion-based online quality inspection method for advanced chip packaging processes as described in any one of claims 1-6, characterized in that, include: The data acquisition module is configured to synchronously or quasi-synchronously trigger and acquire data from multiple detection modes at the production line inspection station. An edge computing unit is connected to the data acquisition module and is configured to preprocess the raw data of each modality, run each single-modality defect detection model, obtain and upload preliminary defect detection results; A converged analytics server, communicatively connected to the edge computing unit, includes: A confidence model library is used to store and maintain the modality-defect confidence matrix. The multimodal fusion decision engine is configured to receive the preliminary defect detection results of each modality, query the confidence model library, execute the weighted fusion algorithm, calculate the comprehensive confidence score, and make the final defect judgment. The human-computer interaction and application module is connected to the fusion analysis server and is configured to display the final detection results, comprehensive confidence level, trigger alarms, and generate quality reports.

8. The online quality inspection system for advanced chip packaging processes based on multimodal data fusion according to claim 7, characterized in that, The data acquisition module integrates at least three detection modes: X-ray imaging, optical appearance imaging, and infrared thermal imaging; the edge computing unit includes multiple parallel processing sub-units, each of which is dedicated to processing data of one mode and running the corresponding single-mode detection model.

9. The online quality inspection system for advanced chip packaging processes based on multimodal data fusion according to claim 7, characterized in that, The fusion analysis server also includes an online model update unit, configured for: Receive confirmed online detection results and corresponding multimodal raw data, and store them as new samples in the historical database; Monitor the number of new samples. When the number reaches a preset threshold, automatically trigger or be manually triggered by the administrator to recalculate the confidence model. The offline modeling program is invoked to recalculate and generate a new modality-defect confidence matrix based on the updated historical database, which replaces the old matrix in the confidence model library.

Images