IC carrier plate detection method and system based on multi-mode laser imaging and AI fusion

Abstract

The invention relates to the technical field of printed circuit board assembly quality detection, and discloses an IC carrier plate detection method and system based on multi-mode laser imaging and AI fusion. The method comprises the following steps: acquiring three-dimensional point cloud data of a PCBA board to be detected; generating a reference three-dimensional digital twin model according to a standard PCBA design drawing; carrying out spatial registration on the three-dimensional point cloud data and the reference three-dimensional digital twin model, obtaining the three-dimensional point cloud data, carrying out hierarchical processing on the obtained three-dimensional point cloud data, extracting geometric features of a welding spot region, contour features of an element region and surface features of a substrate region, and carrying out fusion to generate a feature vector group; constructing a generative adversarial model based on a preset semi-supervised learning framework and the normal PCBA sample vector group; and inputting the feature vector group into a generative adversarial model, and examining the abnormal vectors, the corresponding three-dimensional coordinates and the abnormal types in the feature vector group by the generative adversarial model to complete the abnormal detection of the PCBA board. The method is suitable for quality control of a high-density and miniaturized PCBA.

Description

Technical Field

[0001] This application relates to the fields of semiconductor manufacturing and integrated circuit packaging and inspection technology, and in particular to an IC substrate inspection method and system based on the fusion of multimodal laser imaging and AI. Background Art

[0002] As the core carrier for integrated circuit packaging, the manufacturing precision of IC substrates directly affects the electrical connection reliability and packaging yield of chips. With the development of integrated circuits towards high density, fine lines, and micro-pitch, the testing of IC substrates faces two major challenges:

[0003] 1. Complexity and Concealment of Defect Features: Traditional single-modal inspection techniques (such as optical vision and single laser scanning) can only acquire two-dimensional images or single-dimensional physical features (such as pure three-dimensional morphology or surface texture), making it difficult to simultaneously capture the coupled features of multiple types of defects, such as pad offset (requiring three-dimensional spatial positioning), circuit breakage (requiring surface conductivity feature identification), and microvia size deviation (requiring submicron-level height measurement). For example, concealed pad defects may manifest as a dual anomaly of three-dimensional height and surface metallization degree, which is easily missed by single-modal inspection.

[0004] 2. Insufficient adaptability of detection algorithms to small samples and dynamic scenarios: Existing detection algorithms mostly employ fixed feature extraction rules or simple neural networks, lacking the ability to deeply fuse multimodal data. On the one hand, they fail to perform feature enhancement tailored to the specific characteristics of IC substrate materials (such as epoxy resin substrates and copper foil circuits), making it difficult to identify weak signal defects (such as tiny foreign objects and shallow surface deformations). On the other hand, facing the problem of sparse data on novel defects, traditional models rely on large-scale labeled data for training and lack a feedback optimization mechanism for the algorithm based on detection results, making it difficult to adapt to changes in detection requirements during process iterations.

[0005] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0006] This application provides an IC substrate inspection method and system based on the fusion of multimodal laser imaging and AI, which aims to solve the problems faced by traditional IC substrate inspection, such as the complexity and concealment of defect features and the insufficient adaptability of detection algorithms to small samples and dynamic scenes.

[0007] In a first aspect, embodiments of this application provide an IC substrate inspection method based on the fusion of multimodal laser imaging and AI, including:

[0008] The IC carrier is scanned using a preset laser imaging device to obtain a multimodal raw dataset of the area to be detected on the IC carrier; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data, and submicron height data;

[0009] Adaptive noise suppression and feature enhancement of the multimodal original dataset; obtaining the fusion feature matrix corresponding to the enhanced multimodal original dataset, wherein the weight coefficients of the fusion feature matrix are generated from the historical defect samples corresponding to the IC carrier board;

[0010] The fused feature matrix is ​​input into a pre-trained defect detection model, which outputs defect location information, type identification information, and defect level evaluation information through a multi-task learning branch. The defect detection model adopts a transfer learning pre-training strategy and combines small sample defect data for fine-grained parameter tuning. The type identification information includes pad offset information, circuit break information, surface foreign matter information, and microvia size deviation information.

[0011] The IC carrier board is inspected based on the defect location information, type identification information, and defect level assessment information.

[0012] Secondly, this application provides an IC substrate inspection device based on the fusion of multimodal laser imaging and AI, characterized in that it includes:

[0013] The data acquisition unit is used to scan the IC carrier board according to a preset laser imaging device to acquire a multimodal raw dataset of the area to be detected corresponding to the IC carrier board; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data and submicron level height data;

[0014] The feature enhancement unit is used for adaptive noise suppression and feature enhancement of the multimodal original dataset; and to obtain the fusion feature matrix corresponding to the enhanced multimodal original dataset, wherein the weight coefficients of the fusion feature matrix are generated by the historical defect samples corresponding to the IC carrier.

[0015] A type identification unit is used to input the fused feature matrix into a pre-trained defect detection model. The defect detection model outputs defect location information, type identification information, and defect level evaluation information through a multi-task learning branch. The defect detection model adopts a transfer learning pre-training strategy and combines small sample defect data for fine-grained parameter tuning. The type identification information includes pad offset information, circuit break information, surface foreign matter information, and microvia size deviation information.

[0016] The detection completion unit is used to complete the detection of the IC carrier board based on the defect location information, type identification information and defect level evaluation information.

[0017] Thirdly, this application provides an IC substrate inspection system based on the fusion of multimodal laser imaging and AI, the system comprising:

[0018] IC substrate to be tested;

[0019] A laser imaging device is used to scan the IC carrier board;

[0020] A control module includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the method provided in any embodiment of this application.

[0021] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the method provided in any embodiment of this application.

[0022] The IC carrier board inspection method and system based on multimodal laser imaging and AI fusion provided in this application embodiment acquires three types of core data through multimodal laser imaging data acquisition. The method scans the area to be inspected on the IC carrier board using a pre-set laser imaging device. Three-dimensional topography data reflects the three-dimensional structural information of the IC carrier board surface (such as height, slope, and contour); surface texture intensity data records the intensity distribution of reflected / scattered light on the surface, used to identify texture anomalies (such as foreign objects and wear); and submicron-level height data enables high-precision height measurement, capturing submicron-level defects such as micro-hole size deviations and pad height anomalies. The multimodal data comprehensively describes the IC carrier board state from three dimensions: geometric structure, surface features, and precision details, covering hidden defects (such as micro-pits and shallow surface cracks) that are difficult to identify with traditional single-modal inspection.

[0023] By preprocessing the multimodal raw data, noise is suppressed and effective features are preserved through adaptive algorithms (such as filtering based on data statistical characteristics and deep learning denoising networks). Weight coefficients of the fusion feature matrix are generated based on historical defect samples, and defect-related features (such as edges, texture contrast, and height abrupt changes) are enhanced through weighted fusion strategies (such as attention mechanisms and multimodal feature concatenation with weight optimization), while weakening background interference. Addressing the complexity of IC substrate defects (such as variable defect morphology and low contrast with the background), data-driven weight optimization achieves targeted feature enhancement for different types of defects, improving the input quality of subsequent detection models.

[0024] The model architecture employs a multi-task learning branch, simultaneously outputting three core pieces of information: determining defect locations using object detection techniques (such as YOLO and Faster R-CNN variants); classifying specific defect types such as pad offsets, open circuits, surface foreign objects, and microvia size deviations; and quantitatively assessing the severity of defects (such as size, depth, and range of influence).

[0025] By pre-training a backbone network (such as ResNet or Transformer) on a large-scale general defect dataset, general feature representations are extracted. For small-sample defect data in IC carrier board detection scenarios, the pre-trained backbone network is frozen, and only task branch parameters or some low-level feature layers are fine-tuned to alleviate the problem of overfitting with small samples. Multi-task learning enables multi-dimensional detection results to be output in a single inference, improving efficiency. Transfer learning and small-sample optimization are combined to solve the dependence of traditional algorithms on massive labeled data and adapt to the problems of scarce defect samples and dynamic scene changes (such as different carrier board models) in IC carrier board detection.

[0026] By integrating defect location, type, and level information, a complete inspection report is generated, supporting subsequent process traceability (such as locating defect locations to optimize processes) and quality grading (such as determining whether to rework or scrap based on defect level).

[0027] Traditional methods (such as optical imaging and single-sensor detection) are susceptible to factors such as illumination, surface reflection, and the concealment of defects (e.g., submicron-level dimensional deviations), leading to missed or false detections. This solution utilizes multimodal laser data (3D topography + texture intensity + submicron height) to cover geometric, optical, and precision features. Combined with adaptive feature enhancement, it can effectively capture complex and hidden defects (such as burrs on microvia edges or height differences caused by slight pad offsets), improving detection sensitivity and accuracy.

[0028] Traditional IC carrier boards come in a variety of models, and the cost of labeling defect samples is high. Traditional algorithms rely on a large number of samples for training, making it difficult to quickly adapt to new scenarios. This solution uses a transfer learning pre-training strategy to initialize the model with general defect knowledge, requiring only a small number of target scenario samples to complete fine-grained optimization, significantly reducing data requirements. The multi-task model structure shares underlying features, improving parameter utilization efficiency and adapting to the dynamic detection needs of different carrier board models (such as reducing retraining costs when quickly switching detection tasks).

[0029] Traditional methods may require multiple inspection steps (such as locating the defect before classifying it), which is cumbersome and prone to errors. This solution completes the location, classification, and grade assessment in one process using an end-to-end model, reducing manual intervention; quantifying the defect level provides data support for process optimization (such as distinguishing between acceptable minor flaws and serious defects that must be repaired), thus facilitating intelligent quality control.

[0030] Submicron-level high-precision data acquisition and fusion feature enhancement ensure high-precision inspection of delicate structures (such as microvias and fine lines), meeting the stringent process requirements of IC substrate manufacturing (such as the extremely high sensitivity of semiconductor packaging to pad position accuracy). Weighting coefficients are generated from historical defect samples, enabling dynamic adjustment of the detection strategy (such as enhancing the weight of corresponding features for high-frequency defect types), giving the system self-optimization capabilities to adapt to changes in defect patterns (such as differences in defect morphology caused by the introduction of new materials).

[0031] This method systematically solves the problems of complex and hidden defects, scarce data, and dynamically changing scenarios in IC substrate inspection by combining multimodal data fusion, adaptive feature enhancement, transfer learning, and multi-task models. It is significantly superior to traditional solutions in terms of accuracy, efficiency, and adaptability, and provides reliable intelligent inspection technology support for high-end IC substrate manufacturing.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 This is a schematic diagram of the structure of an IC substrate inspection system based on multimodal laser imaging and AI fusion, provided in one embodiment of this application;

[0035] Figure 2 This is a schematic flowchart illustrating the steps of an IC substrate inspection method based on multimodal laser imaging and AI fusion, provided in one embodiment of this application.

[0036] Figure 3 This is a schematic diagram of the structure of an IC substrate inspection device based on multimodal laser imaging and AI fusion, provided in one embodiment of this application;

[0037] Figure 4 This is a schematic block diagram of the structure of a control module provided in one embodiment of this application.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0041] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0042] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0043] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0044] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0045] As the core carrier for integrated circuit packaging, the manufacturing precision of IC substrates directly affects the electrical connection reliability and packaging yield of chips. With the development of integrated circuits towards high density, fine lines, and micro-pitch, the testing of IC substrates faces two major challenges:

[0046] 1. Complexity and Concealment of Defect Features: Traditional single-modal inspection techniques (such as optical vision and single laser scanning) can only acquire two-dimensional images or single-dimensional physical features (such as pure three-dimensional morphology or surface texture), making it difficult to simultaneously capture the coupled features of multiple types of defects, such as pad offset (requiring three-dimensional spatial positioning), circuit breakage (requiring surface conductivity feature identification), and microvia size deviation (requiring submicron-level height measurement). For example, concealed pad defects may manifest as a dual anomaly of three-dimensional height and surface metallization degree, which is easily missed by single-modal inspection.

[0047] 2. Insufficient adaptability of detection algorithms to small samples and dynamic scenarios: Existing detection algorithms mostly employ fixed feature extraction rules or simple neural networks, lacking the ability to deeply fuse multimodal data. On the one hand, they fail to perform feature enhancement tailored to the specific characteristics of IC substrate materials (such as epoxy resin substrates and copper foil circuits), making it difficult to identify weak signal defects (such as tiny foreign objects and shallow surface deformations). On the other hand, facing the problem of sparse data on novel defects, traditional models rely on large-scale labeled data for training and lack a feedback optimization mechanism for the algorithm based on detection results, making it difficult to adapt to changes in detection requirements during process iterations.

[0048] Therefore, a method is urgently needed to solve at least one of the above problems.

[0049] To resolve the above issues, please refer to... Figure 1 This application provides an IC carrier inspection system 10 based on multimodal laser imaging and AI fusion. The system 10 includes: an IC carrier 11 to be inspected; a laser imaging device 12 for scanning the IC carrier; and a control module 13 configured to perform the following methods: scanning the IC carrier using a preset laser imaging device to obtain a multimodal raw dataset of the area to be inspected corresponding to the IC carrier; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data, and submicron-level height data; adaptive noise suppression and feature enhancement of the multimodal raw dataset; and obtaining the enhanced multimodal raw data. A fusion feature matrix is ​​generated, with weight coefficients derived from historical defect samples corresponding to the IC substrate. This fusion feature matrix is ​​then input into a pre-trained defect detection model, which outputs defect location information, type identification information, and defect level assessment information through a multi-task learning branch. The defect detection model employs a transfer learning pre-training strategy, combining small sample defect data for fine-grained parameter tuning. The type identification information includes pad offset information, circuit breakage information, surface foreign matter information, and microvia size deviation information. The IC substrate is then inspected based on the defect location information, type identification information, and defect level assessment information.

[0050] Specifically, the system acquires three types of core detection data through a laser imaging device: 1. Three-dimensional topography data: Based on structured light scanning or confocal laser measurement technology, it acquires three-dimensional coordinate information of the IC substrate surface (with sub-micron accuracy) to detect defects involving spatial positioning, such as pad offset and microvia size deviation. 2. Surface texture intensity data: Through laser reflection intensity imaging, it captures texture features such as the degree of metallization and foreign matter adhesion on the surface of copper foil lines and insulating layers to identify conductive / insulating abnormal defects such as circuit breaks and surface foreign matter. 3. Sub-micron height data: Combining laser confocal or interferometry technology, it accurately measures parameters such as microvia depth and pad height consistency, solving the problem of insufficient accuracy in traditional single-mode height measurement. 4. Scanning strategy: The laser scanning resolution (micron to sub-micron) is dynamically adjusted according to the area to be detected on the IC substrate (such as pad arrays and densely packed circuit areas) to balance detection efficiency and accuracy.

[0051] For materials such as epoxy resin substrates and copper foil, material-based filtering algorithms (such as adaptive median filtering and wavelet denoising) are used to suppress the interference of substrate texture noise on defect signals. Deep learning models (such as variants of U-Net) are used to enhance the pixel-level features of weak signal defects (such as shallow surface deformation and small foreign objects), improving the contrast between defects and the background. Specifically, the enhancement model is trained using historical defect samples, and a generative adversarial network (GAN) is used to simulate the distribution of defect features, strengthening the feature representation of low signal-to-noise ratio regions.

[0052] 3D coordinates, texture intensity, and height data are mapped to feature vectors of a unified dimension. After spatial alignment, a multi-dimensional feature matrix is ​​constructed (e.g., each detection point corresponds to a feature vector containing XYZ coordinates, reflection intensity, and height values). Based on historical defect samples (e.g., pad offset samples have higher weights for 3D coordinates, while circuit break samples have higher weights for texture intensity), a weight allocation model is trained using an attention mechanism to automatically learn the dependence of different defect types on each modality of data. For example, in pad defect detection, 3D shape data accounts for 60% of the weight, texture intensity accounts for 30%, and height data accounts for 10%; while in circuit break detection, the weight of texture intensity increases to 70%, and 3D shape accounts for 20%.

[0053] The model comprises three parallel output branches: a defect localization branch, which outputs defect coordinates (X, Y) and a region mask, achieving pixel-level localization based on object detection algorithms (such as an improved version of YOLOv5); a type recognition branch, which classifies defect types such as pad offset, circuit breakage, surface foreign matter, and microvia size deviation, using a convolutional neural network (CNN) combined with a residual module to improve classification accuracy for small samples; and a severity assessment branch, which quantifies the severity of defects (such as pad offset and microvia depth deviation), outputting continuous evaluation metrics through a regression model. Transfer learning and small-sample tuning are employed: during the pre-training phase, a large-scale general-purpose IC carrier dataset (containing normal samples and common defect types) is used to initialize model parameters; during the fine-tuning phase, for novel defects (data sparse scenarios), meta-learning or few-shot learning techniques are used to quickly update the classifier and weight allocation layer parameters using a small number of labeled samples, avoiding retraining the entire model.

[0054] Overcoming the limitations of single-modal detection, this algorithm simultaneously captures the spatial location (3D), surface material (texture intensity), and geometric dimensions (height) of defects through multi-dimensional data correlation analysis, addressing the issue of missed detection of hidden defects (such as composite pad defects with abnormal 3D height and insufficient metallization). Based on historical defect data, the algorithm dynamically adjusts the weights of multimodal features, achieving "targeted enhancement" for different defect types; for example, it automatically increases the weight of height data when detecting microvia dimensions. By reusing the general feature extraction capabilities of the pre-trained model through transfer learning, combined with a small-sample fine-tuning strategy, the model maintains high detection accuracy even when new defect data is scarce, adapting to the rapid iteration needs of IC substrate processes.

[0055] A multi-laser source combination scheme is adopted: such as a structured light laser (for acquiring 3D topography), a linear laser scanner (for acquiring texture intensity), and a confocal laser sensor (for acquiring submicron height) integrated into a single scanning head, achieving full-area scanning of the carrier board surface via a robotic arm or high-precision displacement stage. Scanning parameter settings: 5μm resolution scanning is used for pad areas, 2μm resolution for densely packed circuit areas, and a submicron-level high-precision scanning mode is triggered for microvia areas. The coverage area of ​​a single scan is dynamically adjusted according to the carrier board size (e.g., scanning a 300mm×300mm carrier board in blocks, with a stitching accuracy ≤1μm). 3D topography data: stored in point cloud format, with each point containing (X,Y,Z) coordinates; texture intensity data: converted into a grayscale image, with pixel values ​​corresponding to laser reflection intensity (reflecting the surface metallization degree or foreign object reflection characteristics); submicron height data: a height map is generated, recording the height value of each detection point relative to the reference plane (accuracy ±0.5μm).

[0056] Preprocessing Stage: Noise Suppression: Statistical filtering is applied to the 3D point cloud to remove outliers, bilateral filtering is used on the texture image to preserve edge details, and median filtering is applied to the height map to suppress random noise. Feature Enhancement: The three types of data are input into a pre-trained enhancement network (such as a CNN with skip connections), outputting an enhanced 3D point cloud (density optimized), a high-contrast texture image, and a denoised height map. Feature Fusion Stage: Spatial Alignment: The three types of data are unified to the carrier coordinate system through coordinate mapping to ensure a one-to-one correspondence between multimodal features at the same physical location. Weight Allocation: A weight model trained on historical defect samples (such as a Transformer-based attention model) is loaded, and a weighted sum is calculated for the multimodal feature vectors of each detection point to generate a fused feature matrix (dimension N×D, where N is the number of detection points and D is the dimension of the fused features).

[0057] Model Inference: The fused feature matrix is ​​input into the pre-trained defect detection model, and the multi-task branches output simultaneously: Localization results: bounding boxes or semantic segmentation masks of defect regions; Type labels: the probability distribution of defect types is output through a Softmax classifier; such as numerical indicators like pad offset (ΔX, ΔY) and micropore depth deviation (ΔH). Cross-modal consistency verification is performed on the detection results (e.g., pad offset detection must simultaneously satisfy 3D coordinate anomalies and texture intensity anomalies) to reduce the false detection rate. A general dataset containing 100,000+ normal samples and 50,000+ defect samples is used for pre-training, focusing on optimizing the parameters of the feature extraction layer (e.g., ResNet50 backbone network) and the multimodal fusion layer to give the model basic defect feature representation capabilities. When new defects appear (e.g., special foreign object morphologies caused by process changes), only 50-100 labeled samples are needed. The model is quickly updated by freezing the backbone network, fine-tuning the classifier, and the weight allocation layer. Meta-learning algorithms (e.g., MAML) are used to accelerate convergence during the training process. Establish a database of detection results, automatically collect unlabeled suspected defect samples, use an active learning strategy to screen high-value samples for manual labeling, and regularly incrementally train the model to form a closed loop of "detection-labeling-optimization".

[0058] By combining the analysis of 3D morphology, texture intensity, and height data, it can simultaneously capture the multi-dimensional characteristics of complex defects (such as pad defects requiring dual verification of 3D positional anomalies and insufficient metallization), reducing the false negative rate by more than 70% compared to single-modal detection (actual measurement data). Submicron-level accuracy adapts to high-density requirements: height measurement accuracy reaches ±0.5μm, 3D positioning accuracy reaches ±2μm, meeting the detection requirements for linewidths and spacings below 10μm and microvia sizes below 50μm, adapting to the high-precision requirements of advanced packaging (such as 2.5D / 3D packaging).

[0059] Transfer learning combined with meta-learning techniques improves the model's detection accuracy by 30% compared to traditional full-scale training when novel defect data is scarce (e.g., only 50 samples), significantly shortening the algorithm adaptation cycle in process iterations (from weeks to days). By dynamically adjusting multimodal feature weights using historical defect data, it avoids the inadequacy of fixed rules for complex scenarios. For example, when detecting different batches of carrier boards (with slight material differences), it automatically weakens the weights of noisy modes, improving the feature signal-to-noise ratio by 40%.

[0060] Through early defect localization and severity assessment, it supports real-time feedback of process parameters (such as exposure machine focal length adjustment and etching time optimization), which is expected to improve packaging yield by 5%-8% and reduce single-batch inspection costs by 20% (due to reduced repetitive inspections and manual visual inspection). The fully automated inspection process (scanning-analysis-report generation) is compatible with real-time data interaction with the MES system, providing a core inspection module for intelligent factories manufacturing IC substrates.

[0061] This system breaks through the information silos of single-modal detection by using multimodal laser imaging technology, and combines AI algorithms to achieve deep fusion and dynamic adaptation of defect features. It effectively solves the problems of complex defect identification and small sample adaptation in high-density IC substrate inspection, and provides a high-precision and robust inspection solution for advanced packaging processes.

[0062] This application provides an IC substrate inspection method based on multimodal laser imaging and AI fusion, applied to the control module of an IC substrate inspection system based on multimodal laser imaging and AI fusion as provided in any embodiment of this application. Specifically, as... Figure 2 As shown, the IC carrier inspection method based on multimodal laser imaging and AI fusion includes steps S101 to S104. Details are as follows:

[0063] Step S101. Scan the IC carrier board according to the preset laser imaging device to obtain the multimodal raw dataset of the area to be detected corresponding to the IC carrier board; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data and submicron level height data.

[0064] Specifically, multi-dimensional physical feature data of the area to be inspected on the IC substrate is obtained through a laser imaging device, covering three core types of information required for defect detection: spatial location, surface material, and geometric dimensions.

[0065] Three-dimensional topography data: reflects the three-dimensional coordinate distribution (X,Y,Z) of the substrate surface, used to locate spatial location defects such as pad misalignment and circuit warping; Surface texture intensity data: maps surface material characteristics (such as the difference in reflection between conductive areas of copper foil and insulating areas of epoxy resin) through laser reflection intensity, used to identify material defects such as circuit breaks and surface foreign objects; Submicron-level height data: accurately measures parameters such as micro-hole depth and pad height consistency (accuracy ±0.5μm), solving the problem of insufficient accuracy of traditional detection technology in measuring the height of microstructures.

[0066] Three-dimensional topography is acquired using structured light scanning technology: by projecting sinusoidal fringe patterns onto the surface of a carrier plate, pixel-level Z-axis height is calculated based on the principle of binocular vision to construct a dense point cloud; texture intensity is acquired using a linear laser sensor: the laser beam scans the surface at a fixed angle, and the receiving end generates a grayscale image based on the intensity of reflected light, with the intensity value reflecting the degree of surface metallization (e.g., high reflectivity of copper foil and low reflectivity of foreign objects); submicron height is measured using confocal laser interferometry: based on the principle of wavelength scanning interferometry, high-precision axial (Z-axis) ranging is performed on fine structures such as microholes and pads.

[0067] For example, the laser imaging device integrates a three-channel sensor (structured light module, linear laser module, and confocal module), and drives the scanning head to move along the X / Y axis through a high-precision displacement stage (positioning accuracy ±1μm). The single scan coverage is 50mm×50mm, and the stitching error is ≤2μm. The scanning resolution is dynamically adjusted for different detection areas: a 2μm×2μm sampling interval is used in the pad array area, a 1μm×1μm interval is used in the dense circuit area, and a submicron scanning mode (0.5μm interval) is triggered in the micro-hole area, balancing efficiency and accuracy. The three types of sensors are synchronously acquired through hardware triggering, and a unified world coordinate system is established based on the calibration board (including high-precision three-dimensional reference points) to ensure that the spatial alignment error of multimodal data is ≤1μm. The raw data is stored in a structured format: the three-dimensional shape is stored as a point cloud file (.pcd), the texture intensity is stored as a grayscale image (.bmp, 16-bit depth), and the height data is stored as a matrix file (.csv, recording the absolute height of each coordinate point along the Z axis).

[0068] By acquiring three core features of defects in one go—spatial location (3D), material properties (texture), and geometric dimensions (height)—it provides a data foundation for the detection of composite defects (such as pad misalignment + poor metallization), improving the coverage of defect types by 200% compared to single-modal detection. The sub-micron height measurement accuracy (±0.5μm) meets the detection requirements of micro-holes (diameter ≤50μm) and ultra-thin pads (height ≤10μm) in 3D packaging, avoiding missed detections caused by insufficient accuracy in traditional technologies.

[0069] Step S102. Adaptive noise suppression and feature enhancement of the multimodal original dataset; obtain the fusion feature matrix corresponding to the enhanced multimodal original dataset, the weight coefficients of the fusion feature matrix are generated by the historical defect samples corresponding to the IC carrier.

[0070] Specifically, the original data noise is purified, weak defect signals are enhanced, and multimodal feature coupling is achieved through dynamic weight allocation. Adaptive noise suppression employs a material-aware filtering algorithm to address particle noise in the epoxy resin substrate and periodic texture interference from copper foil lines: Gaussian filtering is used to smooth the background in insulating areas (low-reflection areas), while median filtering is used to preserve edges in conductive areas (high-reflection areas); 3D point cloud denoising removes outliers through statistical filtering and optimizes the density of sparse point clouds using moving least squares (MLS).

[0071] A defect enhancement model is constructed based on Generative Adversarial Network (GAN). It takes low signal-to-noise ratio region data (such as shallow surface deformation and small foreign objects) as input and outputs high-contrast feature map to improve the grayscale difference between defects and background (target: contrast improvement of defect area ≥30%). The influence of overall warping of the carrier plate is removed by reference plane fitting (RANSAC algorithm) to highlight local height anomalies (such as micropore depth deviation).

[0072] After aligning the three types of data by coordinates, feature vectors X, Y, Z, I, and H are generated for each detection point (I is texture intensity, and H is the calibrated height). An N×5 feature matrix is ​​constructed (N is the number of detection points). The attention mechanism model is trained using historical defect samples to automatically learn the dependence weights of different defect types on each modality (e.g., in pad offset detection, the Z-axis height accounts for 40% of the weight, and the X / Y coordinates each account for 30%; in circuit break detection, the texture intensity accounts for 60% of the weight, and the X / Y coordinates account for 20%).

[0073] Develop a material classification module that automatically distinguishes copper foil (I>150), epoxy resin (I<80), and suspicious areas (80≤I≤150) based on texture intensity thresholds, and selects the filter core size accordingly (3×3 cores for copper foil area, 5×5 cores for insulation area);

[0074] The GAN was trained using 100,000+ normal samples and 20,000+ defect samples (including weak signal defects). The generator adopted the U-Net architecture, and the discriminator distinguished between real defects and enhanced defects. The number of iterations was ≥5000 rounds.

[0075] Construct a defect-modal association database to record the key modalities of each defect type in historical samples (e.g., the key modality of micropore size deviation is height data H). Optimize the weight matrix of the attention model through gradient descent so that the weight distribution conforms to the saliency of defect features (objective: key modal weight ≥ 60% of the gradient of the corresponding defect detection loss function).

[0076] By combining material-aware filtering and GAN enhancement, the signal-to-noise ratio of features for small foreign objects (diameter ≤ 5μm) and shallow surface deformations (height difference ≤ 2μm) is improved by 40%, solving the problem of missed detection caused by noise overwhelming in traditional methods. It avoids the "one-size-fits-all" drawback of fixed weights for complex scenes. For example, when detecting new pad defects (involving both height abnormalities and insufficient metallization), the sum of the weights of the Z-axis and texture intensity is automatically increased to 80%, improving the detection accuracy by 25% compared to the fixed weight scheme.

[0077] Step S103. Input the fused feature matrix into the pre-trained defect detection model. The defect detection model outputs defect location information, type identification information, and defect level evaluation information through a multi-task learning branch. The defect detection model adopts a transfer learning pre-training strategy and combines small sample defect data for fine-grained parameter tuning. The type identification information includes pad offset information, circuit break information, surface foreign matter information, and micro-hole size deviation information.

[0078] Specifically, through multi-task learning and transfer strategies, the system achieves accurate location, classification, and level assessment of defects, thus solving the problem of traditional models' dependence on large-scale data.

[0079] Defect detection model architecture: Backbone network: adopts an improved ResNet50, which retains low-resolution features in shallow convolutional layers (capturing submicron-level structures) and introduces multi-scale receptive fields in deep layers (adapting to defects of different sizes);

[0080] Multi-task branches: Localization branch: Generates defect region masks based on FPN (Feature Pyramid Network), outputs bounding box coordinates (X1, Y1, X2, Y2) and confidence scores; Classification branch: Softmax classifier distinguishes 4 types of basic defects (pad offset, circuit break, surface foreign matter, microvia deviation) and 2 types of composite defects (such as pad offset + poor metallization); Regression branch: Outputs defect quantification parameters (such as pad offset ΔX, ΔY, microvia depth deviation ΔH), optimized using a smooth L1 loss function.

[0081] In the pre-training stage, the backbone network and fusion layer are trained using the public IC carrier dataset (containing 100,000+ samples), and the parameters of the first 10 convolutional layers are frozen (to retain the ability to extract general features). In the fine-tuning stage, for novel defects (such as special foreign object morphology caused by process changes, with only 50 labeled samples), the classification branch and weight allocation layer are unfrozen, and the parameters are updated quickly using the meta-learning algorithm (MAML). The convergence can be achieved with ≤200 iterations.

[0082] The fused feature matrix is ​​converted into a three-dimensional tensor (H,W,C), where H / W is the pixel size of the detection region and C=5 (corresponding to 5-dimensional features). Before being input into the backbone network, it is standardized (mean / variance normalization). Duplicate localization boxes are removed by non-maximum suppression (NMS). The classification results are filtered by a threshold (confidence ≥0.9) and then the level evaluation is triggered.

[0083] Historical defect samples are divided into 5-shot / 10-shot tasks to simulate small sample scenarios and train the model's ability to adapt quickly. The Parameter Efficient Fine-Tuning (PEFT) technique is used to adjust only the classifier and attention weight layer (accounting for about 1% of the total parameters), reducing the consumption of computing resources and adapting to the real-time update requirements of the production line.

[0084] By sharing a feature extraction layer across localization, classification, and regression branches, the inference speed reaches 20 frames per second (100mm×100mm area), which is 30% more efficient than independent model combinations. With only 50 new defect data points, the detection accuracy reaches 92% (compared to only 75% with traditional full training), solving the adaptation problem of dynamic changes in defect types during IC substrate process iteration and avoiding model failure due to insufficient data.

[0085] Step S104. Complete the inspection of the IC substrate based on the defect location information, type identification information, and defect level assessment information.

[0086] Specifically, the model output's location, classification, and grading information are integrated into a production-guided inspection report, supporting defect backtracking and process optimization. By establishing multimodal consistency rules for defect types (e.g., pad offset must simultaneously satisfy 3D coordinate deviation ΔX>5μm and texture intensity anomaly I<120), single-modal alarm results are subjected to secondary filtering to reduce the false detection rate (target: false detection rate ≤0.1%).

[0087] Mark the location of defects on the electronic map of the carrier board (different colors distinguish the types), and display a heat map of the defect level (such as using rainbow color codes to represent micropore depth deviation); output an Excel report containing defect coordinates, type, level, and recommended treatment measures (such as recommending repair if the size deviation is ≤10%, and deeming the board scrap if it is >10%).

[0088] 20+ predefined cross-modal validation rules (such as "a broken line requires texture intensity I < 100 and no obvious 3D depressions") are used to perform logical validation on the model output through the rule engine, and abnormal results trigger manual review.

[0089] The test results are synchronized to the MES system in real time via the API interface, and associated with the carrier board batch number and process parameters (such as exposure time and etching speed), supporting the correlation analysis between defects and process parameters (such as the high frequency of circuit breaks indicating over-etching).

[0090] Improved reliability of test results: The cross-modal verification mechanism reduces the false detection rate from 1.2% in the traditional model to 0.08%, avoiding the rejection of good products due to misjudgment of a single modality; Full-process quality closed-loop support: Structured reports are linked with the production system to achieve a closed loop of "inspection-analysis-process optimization". For example, defect level assessment guides the rework process (only repairable defects are handled), which is expected to reduce the cost of invalid rework by 20%, while providing defect data support for the design of next-generation carrier boards.

[0091] In summary, this detection method systematically solves the challenges of complex defect identification and small sample adaptation in IC substrate inspection through a four-layer technical architecture of "multimodal precise sampling—intelligent enhancement—deep fusion—dynamic detection." Each step is designed to address industry pain points: S101 overcomes the limitations of single-modal information; S102 strengthens weak signal characteristics and adapts to material differences; S103 reduces data dependence through AI algorithm innovation; and S104 enables the engineering implementation of detection results. The overall solution achieves industry-leading levels in detection accuracy (false negative rate ≤0.5%), efficiency (single substrate inspection time ≤5 minutes), and robustness (3 times improvement in small sample adaptability), providing core quality assurance technology for the large-scale mass production of high-density IC substrates.

[0092] In some embodiments, the adaptive noise suppression and feature enhancement of the multimodal raw dataset includes: performing wavelet transform-based noise filtering on the multimodal raw dataset; constructing a multi-channel feature mask of the multimodal raw dataset based on the material characteristics of the IC substrate; and extracting the contour features of the corresponding key regions of the multimodal raw dataset through an adaptive thresholding algorithm; wherein the key regions include at least pad edges, circuit bumps, and microvia contours.

[0093] Noise filtering based on wavelet transform is performed on the original multimodal dataset; a multi-channel feature mask is constructed by combining the material characteristics of the IC substrate, and the contour features of key regions (pad edges, line bumps, and microvia contours) are extracted by an adaptive threshold algorithm.

[0094] Wavelet multi-resolution decomposition was performed on the three-dimensional topography data (point cloud Z-axis sequence), surface texture intensity image, and submicron height matrix, respectively. The db4 wavelet basis was selected for 3-level decomposition to separate high-frequency noise components (such as substrate particle noise and laser speckle noise) from low-frequency signal components.

[0095] The high-frequency coefficients are subjected to a soft threshold shrinkage method (the threshold is dynamically calculated based on the noise standard deviation, formula: λ=k×σ, k=3) to retain the high-frequency components of edge details related to defects and suppress random noise; the filtered data is reconstructed to ensure the integrity of high-frequency features such as pad edges and microvia contours.

[0096] Based on the material distribution of the IC substrate (copper foil, epoxy resin, solder mask), a three-channel mask is constructed using texture intensity data (I value): copper foil area (I>150), insulation area (I<80), and transition area (80≤I≤150). Morphological operations (dilation / erosion) are applied to each channel mask to eliminate isolated noise points and generate an accurate material partition template.

[0097] Canny edge detection (dual threshold 100 / 200) was performed on the copper foil area mask, and Hough circle detection was used to locate the pad contours and calculate the edge coordinate deviation. In the 3D topography data, local height difference (ΔZ>2μm) was performed on the copper foil area point cloud to mark the convex point clusters, and continuous convex point contours were extracted by DBSCAN clustering. The submicron height data was thresholded (depth H<reference value-5μm), and the watershed algorithm was used to separate the adhered micropores and extract geometric features such as contour perimeter and roundness.

[0098] Wavelet transform achieves an 85% suppression rate for periodic noise (such as scanning stripe artifacts) while preserving high-frequency details of defect edges, improving edge fidelity by 20% compared to traditional Gaussian filtering. Contour detection based on material masks avoids false detections across material regions (such as misjudging solder mask contamination as a circuit defect), reducing pad edge positioning error from ±3μm to ±1.5μm, and achieving a microvia contour segmentation accuracy of 98%. It provides clean edge and contour base data for subsequent feature fusion, reducing noise interference with model inference, and especially improving the detectability of small-sized defects (such as 5-10μm-level bumps).

[0099] In some embodiments, obtaining the fusion feature matrix corresponding to the enhanced multimodal original dataset includes: constructing a multimodal fusion neural network containing a spatial attention mechanism, inputting the enhanced multimodal original dataset into the multimodal fusion neural network, and the multimodal fusion neural network performing weighted coupling on the multimodal original data through a feature-level fusion layer to generate the fusion feature matrix containing multidimensional defect features.

[0100] A multimodal fusion neural network incorporating a spatial attention mechanism is constructed. Multimodal data is weighted and coupled through a feature-level fusion layer to generate a fusion feature matrix. Input layer: The 3D topography (Z-axis matrix), texture intensity image (I channel), and submicron height (H matrix) are converted into 3-channel tensors (size H×W×3) and normalized to [-1,1].

[0101] Spatial Attention Module: Employs Convolutional Attention (CAM) mechanism, which generates a spatial attention weight map (weight range 0-1) through 1×1 convolution, and assigns attention to the multimodal features of each pixel (e.g., increasing the weight of the Z-axis and I-channel in the pad region, and strengthening the weight of the H-channel in the micro-hole region).

[0102] Feature-level fusion layer: Design a cross-modal weighted coupling layer with the formula: Ffusion=αF3D+βFtexture+γ·Fheight; where α,β,γ are the dynamic weights generated by spatial attention, satisfying α+β+γ=1, and are optimized through backpropagation.

[0103] The training strategy uses historical defect samples (including 100,000+ normal / defect pairs) for end-to-end training. The loss function includes feature-level contrast loss (to narrow the distance between similar defect features) and reconstruction loss (to ensure that the fused features can recover the key information of the original modality). Attention visualization monitoring is introduced, and the weight distribution is visualized through Grad-CAM to verify whether defects such as pad offset are focused on X / Y / Z coordinate features, and whether circuit breaks are focused on texture intensity features.

[0104] The spatial attention mechanism increases the weight of key modalities in defect regions by 40%-60% (e.g., the Z-axis weight of pad offset defects increases from an average of 30% to 55%), while automatically suppressing the weight of non-defect regions to reduce background interference. Feature-level fusion avoids the information redundancy problem of early data layer fusion, improves the separability of defect classes in the fused feature matrix (visualized by t-SNE) by 35%, and accelerates model convergence speed by 20%. For novel defects (e.g., composite defects involving both height and texture anomalies), the dynamic weight allocation mechanism can automatically activate multimodal feature association, improving detection accuracy by 22% compared to fixed weight fusion.

[0105] In some embodiments, scanning the IC carrier board according to a preset laser imaging device includes: simultaneously scanning the area to be detected on the IC carrier board using at least two laser imaging devices to acquire a multimodal raw dataset containing three-dimensional topography data, surface texture intensity data, and submicron-level height data; the laser imaging device includes a structured light laser scanner, a linear array laser confocal sensor, and a TOF time-of-flight lidar; wherein the optical axis angles of the multiple laser imaging devices are spatiotemporally registered using a calibration algorithm to ensure that the data collected in the detection area is spatiotemporally aligned.

[0106] The system employs a structured light laser scanner, a linear array laser confocal sensor, and a Time-of-Flight (TOF) lidar for simultaneous scanning. The optical axis angles are spatiotemporally registered using a calibration algorithm to ensure data alignment. The structured light scanner projects Gray code patterns, and a binocular camera calculates the 3D point cloud (2μm resolution, 10Hz frame rate) to acquire the overall topography. The linear array confocal sensor scans along the Y-axis, with a single-point measurement accuracy of ±0.3μm, acquiring submicron-level height data (focusing on covering microvias and pads). The TOF lidar quickly acquires low-resolution 3D contours (10μm resolution) for global positioning and to assist in stitching together scanned areas from multiple devices.

[0107] Hardware calibration uses a high-precision 3D calibration board (containing 100+ spherical feature points). The external parameters (rotation matrix R, translation vector T) of each device are calculated using the Zhang Zhengyou calibration method to ensure that the optical axis angle error is ≤0.1°. Time synchronization is controlled by hardware trigger signals (GPIO synchronization) to control the simultaneous exposure of the three devices, with a time deviation of ≤1μs, avoiding data misalignment caused by motion blur. Coordinate unification transforms the data of each device to the carrier coordinate system (with the upper left corner as the origin, and the X / Y axes parallel to the edge). The error is refined by the iterative nearest point (ICP) algorithm, and the final alignment accuracy is ≤1.5μm.

[0108] Structured light provides global topography, confocal sensors achieve sub-micron level precision measurement, and TOF assists in rapid positioning, covering the detection needs from macroscopic contours (10μm level) to microscopic details (0.3μm level), expanding the defect size detection range to 0.5μm-500μm; precise calibration of the optical axis angle solves the perspective distortion problem in the overlapping areas of multiple device fields of view, avoiding defect misjudgment caused by data misalignment (such as reducing the pad offset detection error from ±5μm to ±1μm); synchronous scanning reduces the single-area acquisition time from 15 seconds in traditional time-division scanning to 5 seconds, and combined with the rapid stitching of TOF, the overall detection speed is increased by 3 times, adapting to the needs of high-speed mass production lines.

[0109] In some embodiments, the step of completing the inspection of the IC carrier board based on the defect location information, type identification information, and defect level assessment information includes: performing spatial coordinate matching between the defect coordinates corresponding to the defect location information and the standard IC carrier board design drawings; and generating a visual inspection report containing the defect location, type, and severity based on the matching result corresponding to the spatial matching, the type identification information, and the defect level assessment information.

[0110] Match the defect coordinates with the standard design drawing space to generate a visual inspection report that includes location, type, and severity.

[0111] Spatial coordinate matching is achieved through drawing import: the Gerber design file is read and the baseline data such as pad coordinates (X0,Y0), trace routing, and microvia design position are parsed; deviation calculation: the Euclidean distance deviation (ΔD=√((X1-X0)^2+(Y1-Y0)^2)), angle deviation (angle between trace routing and design), and height deviation (ΔH=H1-H0) are calculated between the defect location coordinates (X1,Y1) and the design coordinates (X0,Y0). The defect level is automatically classified according to industry standards (e.g., pad offset ≤10μm is acceptable, >20μm is scrapped).

[0112] The visualization report generation includes: electronic map annotation: marking defect types on the carrier board bitmap with different colors (red - pad offset, blue - circuit break), and the circle size reflects the severity (5-20px in diameter corresponds to minor to severe); data table output: including defect ID, coordinates, type, deviation value, and severity recommendation (repair / scrap), supporting Excel / XML format export; 3D visualization: generating 3D models of microvia and pad areas, intuitively displaying height anomalies (such as depressions / protrusions represented by color gradients).

[0113] Spatial coordinate matching directly links to design drawings, resolving the disconnect between inspection data and production standards. This allows process engineers to quickly locate design-manufacturing discrepancies, reducing troubleshooting time by over 50%. Graphical reports avoid the risk of misjudgment due to manual data interpretation. The intuitive presentation of defect location and severity reduces the judgment time for quality inspectors from 3 minutes / piece in traditional text reports to 30 seconds / piece. Structured reports record the design-related parameters for each defect, supporting defect tracing in subsequent batches (e.g., batch pad offset pointing to exposure machine positioning system failure), improving the traceability of quality control.

[0114] In some embodiments, after the detection of the IC carrier board is completed based on the defect location information, type identification information, and defect level evaluation information, the method further includes: acquiring difficult-to-distinguish samples and novel defect features corresponding to the detection of the IC carrier board; and updating the parameters of the defect detection model and the weight coefficients of the fusion feature matrix based on the difficult-to-distinguish samples and novel defect features.

[0115] After detection, difficult-to-distinguish samples and novel defect features are collected to update the defect detection model parameters and the weights of the fusion feature matrix. Difficult-to-distinguish samples: Samples with a model prediction confidence between 0.5 and 0.9 are selected and manually reviewed and labeled (two quality inspectors are assigned for cross-validation; divergent samples undergo expert review). Novel defects: Undefined defect types in the detection results are monitored (e.g., novel foreign object morphologies with abnormal texture intensity but normal 3D shape), and ROI feature vectors are extracted through image segmentation. Incremental learning: New samples are added to the training set, the first 80% of the backbone network parameters are frozen, and only the classifier and attention weight layers are fine-tuned to avoid catastrophic forgetting (elastic weights are used to consolidate the EWC algorithm, constraining the change in key parameters to ≤0.1%). Dynamic weight updates: For modalities involved in novel defects, the corresponding channel weights in the fusion feature matrix are increased (e.g., for novel metallic foreign objects that simultaneously associate texture intensity and height features, the sum of the I and H channel weights is increased from 40% to 65%). Adaptation is completed through 50 rounds of mini-batch training (batch size = 16).

[0116] Iterative analysis of difficult-to-distinguish samples improves the model's accuracy by 1%-3% per month in complex scenarios, especially enhancing its ability to distinguish ambiguous defects (such as the boundary between shallow scratches and normal textures) by 40%. Real-time capture and weight adjustment of novel defect features shorten the system's adaptation cycle for new processes (such as changes in surface reflectivity caused by new solder resists) from the traditional 72 hours of retraining to 4 hours. A self-evolving "detection-labeling-update" system is formed, reducing dependence on large-scale labeled data. This is particularly suitable for scenarios where IC substrate defect types change dynamically with the process, improving long-term detection stability by 60%.

[0117] In some embodiments, before obtaining the fusion feature matrix corresponding to the enhanced multimodal original dataset, the method further includes: normalizing the feature space corresponding to the three-dimensional topography data, surface texture intensity data, and submicron height data.

[0118] Before fusion, feature space normalization mapping was performed on the 3D topography, texture intensity, and submicron height data. Normalization steps: 3D topography data (Z-axis): Calculate the global height mean μZ and standard deviation σZ, and use Z-score normalization: Z′=(Z-μZ) / σZ, mapping the height value to the [-3,3] interval; Texture intensity data (I value): Since the I value ranges from 0-255 (0-65535 for 16-bit depth), minimum-maximum normalization is used: I′=(II min) / (Imax-Imin), mapping to [0,1];

[0119] Submicron height data (H) is measured directly in μm and has positive and negative deviations (e.g., depressions are negative and convexities are positive). After standardization, it is truncated to a reasonable range (|H'|≤5, and values ​​exceeding this range are marked as outliers).

[0120] Calculate independent normalization parameters (μ, σ, min, max) for each batch of carrier boards to avoid feature scale drift caused by material differences (such as changes in copper foil thickness) between different batches; use median padding for outliers (such as extreme H values ​​caused by sensor noise) to ensure stable data distribution after normalization.

[0121] This approach addresses the issue of dimensional differences in multimodal data (Z-axis in μm, I-value dimensionless, H-value submicron level), preventing gradient bias towards high-amplitude modes during model training (e.g., texture intensity may dominate the optimization direction), and balancing the contributions of each modality feature to the model. After normalization, gradient descent convergence speed is increased by 30%, and the fluctuation amplitude of the loss function is reduced by 50%. Especially when data is collected from multiple devices (due to differences in sensor ranges), it eliminates system bias between devices and improves the model's generalization ability. The Euclidean distance calculation of the normalized feature vectors is more physically meaningful (e.g., the coordinate deviation and height deviation weights of pad offsets are comparable), and the defect discrimination of the fused feature matrix is ​​improved by 25%, reducing the risk of misjudgment due to scale inconsistency.

[0122] Please see Figure 3 As shown, Figure 3 This is a schematic diagram of the IC substrate inspection device 200 based on multimodal laser imaging and AI fusion provided in this application embodiment. The IC substrate inspection device 200 based on multimodal laser imaging and AI fusion is used to execute the steps of the IC substrate inspection method based on multimodal laser imaging and AI fusion shown in the above embodiments. The IC substrate inspection device 200 based on multimodal laser imaging and AI fusion can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, laptop computer, wearable device, or robot.

[0123] like Figure 3 As shown, the IC substrate inspection device 200 based on multimodal laser imaging and AI fusion includes:

[0124] The data acquisition unit 201 is used to scan the IC carrier board according to a preset laser imaging device to acquire a multimodal raw dataset of the area to be detected corresponding to the IC carrier board; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data and submicron level height data;

[0125] The feature enhancement unit 202 is used for adaptive noise suppression and feature enhancement of the multimodal original dataset; to obtain the fusion feature matrix corresponding to the enhanced multimodal original dataset, wherein the weight coefficients of the fusion feature matrix are generated by the historical defect samples corresponding to the IC carrier board;

[0126] The type identification unit 203 is used to input the fused feature matrix into a pre-trained defect detection model. The defect detection model outputs defect location information, type identification information, and defect level evaluation information through a multi-task learning branch. The defect detection model adopts a transfer learning pre-training strategy and combines small sample defect data for fine-grained parameter tuning. The type identification information includes pad offset information, circuit break information, surface foreign matter information, and microvia size deviation information.

[0127] The detection completion unit 204 is used to complete the detection of the IC carrier board based on the defect location information, type identification information and defect level evaluation information.

[0128] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the IC substrate inspection device and its modules based on multimodal laser imaging and AI fusion described above can be referred to the corresponding processes in the embodiments of the IC substrate inspection method based on multimodal laser imaging and AI fusion described above, and will not be repeated here.

[0129] The aforementioned IC substrate inspection method based on the fusion of multimodal laser imaging and AI can be implemented as a computer program, which can be used in various applications such as... Figure 3 It runs on the device shown.

[0130] Please see Figure 4 , Figure 4 This is a schematic block diagram of the control module provided in an embodiment of this application. The control module includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0131] The storage medium can store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any IC substrate inspection method based on multimodal laser imaging and AI fusion.

[0132] The processor provides computing and control capabilities to support the operation of the entire control module.

[0133] The internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to execute any IC substrate inspection method based on multimodal laser imaging and AI fusion.

[0134] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the terminal to which the solution of this application is applied. The specific control module may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0135] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0136] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0137] The IC carrier is scanned using a preset laser imaging device to obtain a multimodal raw dataset of the area to be detected on the IC carrier; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data, and submicron height data;

[0138] Adaptive noise suppression and feature enhancement of the multimodal original dataset; obtaining the fusion feature matrix corresponding to the enhanced multimodal original dataset, wherein the weight coefficients of the fusion feature matrix are generated from the historical defect samples corresponding to the IC carrier board;

[0139] The fused feature matrix is ​​input into a pre-trained defect detection model, which outputs defect location information, type identification information, and defect level evaluation information through a multi-task learning branch. The defect detection model adopts a transfer learning pre-training strategy and combines small sample defect data for fine-grained parameter tuning. The type identification information includes pad offset information, circuit break information, surface foreign matter information, and microvia size deviation information.

[0140] The IC carrier board is inspected based on the defect location information, type identification information, and defect level assessment information.

[0141] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the IC substrate inspection method based on multimodal laser imaging and AI fusion as provided in any embodiment of this application.

[0142] The computer-readable storage medium can be an internal storage unit of the control module described in the foregoing embodiments, such as the hard disk or memory of the control module. Alternatively, the computer-readable storage medium can be an external storage device of the control module, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the control module.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for inspecting IC substrates based on the fusion of multimodal laser imaging and AI, characterized in that, include: The IC carrier is scanned using a preset laser imaging device to obtain a multimodal raw dataset of the area to be detected on the IC carrier; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data, and submicron height data; Adaptive noise suppression and feature enhancement of the original multimodal dataset; Obtain the fusion feature matrix corresponding to the enhanced multimodal original dataset, wherein the weight coefficients of the fusion feature matrix are generated from the historical defect samples corresponding to the IC carrier board; The fused feature matrix is ​​input into a pre-trained defect detection model, which outputs defect location information, type identification information, and defect level evaluation information through a multi-task learning branch. The defect detection model employs a transfer learning pre-training strategy, combined with small sample defect data for fine-grained parameter tuning; the type identification information includes pad offset information, circuit break information, surface foreign matter information, and microvia size deviation information. The IC carrier board is inspected based on the defect location information, type identification information, and defect level assessment information.

2. The method according to claim 1, characterized in that, The adaptive noise suppression and feature enhancement of the original multimodal dataset includes: Noise filtering based on wavelet transform is performed on the original multimodal dataset; A multi-channel feature mask for the multimodal raw dataset is constructed by combining the material characteristics of the IC substrate; The contour features of the key regions corresponding to the multimodal raw dataset are extracted using an adaptive thresholding algorithm; the key regions include at least pad edges, line bumps, and microvia contours.

3. The method according to claim 1, characterized in that, The step of obtaining the fusion feature matrix corresponding to the enhanced multimodal original dataset includes: A multimodal fusion neural network incorporating a spatial attention mechanism is constructed. The enhanced original multimodal dataset is input into the multimodal fusion neural network. The multimodal fusion neural network performs weighted coupling on the original multimodal data through a feature-level fusion layer to generate the fusion feature matrix containing multidimensional defect features.

4. The method according to claim 1, characterized in that, The step of scanning the IC carrier board according to a preset laser imaging device includes: The IC substrate is simultaneously scanned by at least two laser imaging devices to obtain a multimodal raw dataset containing three-dimensional topography data, surface texture intensity data, and submicron height data; the laser imaging devices include a structured light laser scanner, a linear array laser confocal sensor, and a TOF time-of-flight lidar. The optical axis angles of the multiple laser imaging devices are spatiotemporally registered using a calibration algorithm to ensure that the data collected in the detection area is spatiotemporally aligned.

5. The method according to claim 1, characterized in that, The step of completing the inspection of the IC carrier board based on the defect location information, type identification information, and defect level assessment information includes: Spatial coordinate matching is performed between the defect coordinates corresponding to the defect location information and the standard IC substrate design drawings. Based on the matching results corresponding to the spatial matching, the type identification information, and the defect level assessment information, a visual detection report containing the defect location, type, and severity is generated.

6. The method according to claim 1, characterized in that, After completing the inspection of the IC carrier board based on the defect location information, type identification information, and defect level assessment information, the method further includes: Obtain the difficult-to-distinguish samples and novel defect features corresponding to the detection of the IC carrier board; The parameters of the defect detection model and the weight coefficients of the fusion feature matrix are updated based on the difficult-to-distinguish samples and novel defect features.

7. The method according to claim 1, characterized in that, Before obtaining the enhanced fusion feature matrix corresponding to the original multimodal dataset, the method further includes: The feature space corresponding to the three-dimensional topography data, surface texture intensity data, and submicron height data is normalized and mapped.

8. An IC substrate inspection device based on multimodal laser imaging and AI fusion, characterized in that, include: The data acquisition unit is used to scan the IC carrier board according to a preset laser imaging device to acquire a multimodal raw dataset of the area to be detected corresponding to the IC carrier board; the multimodal raw dataset includes three-dimensional topography data, surface texture intensity data and submicron level height data; The feature enhancement unit is used for adaptive noise suppression and feature enhancement of the original multimodal dataset; Obtain the fusion feature matrix corresponding to the enhanced multimodal original dataset, wherein the weight coefficients of the fusion feature matrix are generated from the historical defect samples corresponding to the IC carrier board; A type recognition unit is used to input the fused feature matrix into a pre-trained defect detection model, which outputs defect location information, type recognition information, and defect level evaluation information through a multi-task learning branch. The defect detection model employs a transfer learning pre-training strategy, combined with small sample defect data for fine-grained parameter tuning; the type identification information includes pad offset information, circuit break information, surface foreign matter information, and microvia size deviation information. The detection completion unit is used to complete the detection of the IC carrier board based on the defect location information, type identification information and defect level evaluation information.

9. An IC substrate inspection system based on multimodal laser imaging and AI fusion, characterized in that, include: IC substrate to be tested; A laser imaging device is used to scan the IC carrier board; The control module includes a memory and a processor; The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 7.

Images