Chip defect diagnosis method based on computer vision and detection system thereof

Abstract

The invention discloses a chip defect diagnosis method based on computer vision and a detection system thereof. Comprising the following steps that S1, first modal data from a first sensing unit and second modal data from a second sensing unit are synchronously or sequentially collected for the same chip to be detected, and the first sensing unit and the second sensing unit work based on different physical principles; and S2, inputting the paired first modal data and second modal data into a pre-trained feature fusion neural network model. According to the chip defect diagnosis method based on computer vision and the detection system thereof provided by the embodiment of the invention, cross validation and comprehensive analysis can be carried out by comprehensively utilizing complementary information of different modal data, so that the inherent misjudgment rate and omission ratio of a single detection mode are remarkably reduced, and the overall accuracy and reliability of defect diagnosis are improved.

Description

Technical Field

[0001] This application relates to the technical field of measurement and detection, and in particular to a chip defect diagnosis method and detection system based on computer vision. Background Technology

[0002] Currently, chip defect diagnosis and detection mainly rely on the following non-destructive testing methods based on different physical principles, including X-ray inspection, optical inspection, and ultrasonic inspection.

[0003] X-ray and optical inspection methods each have their own areas of potential for misjudgment. X-ray inspection may misjudge image artifacts caused by overlapping internal structures as real defects, or miss tiny foreign objects with a density similar to that of the substrate. Optical inspection, on the other hand, is prone to misjudging environmental interference as physical damage due to surface reflections or cleanliness issues, and is completely unable to detect internal defects. This difference in the direction of misjudgment makes the results of a single inspection method uncertain. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, one objective of this application is to provide a chip defect diagnosis method and detection system based on computer vision, which can comprehensively utilize complementary information from different modal data for cross-validation and comprehensive analysis, thereby significantly reducing the inherent false positive and false negative rates of a single detection method and improving the overall accuracy and reliability of defect diagnosis.

[0006] To achieve the above objectives, the first aspect of this application proposes a chip defect diagnosis method based on computer vision, comprising the following steps: S1. For the same chip under test, synchronously or sequentially collect first mode data from the first sensing unit and second mode data from the second sensing unit, wherein the first sensing unit and the second sensing unit operate based on different physical principles; S2. Input the paired first modality data and second modality data into a pre-trained feature fusion neural network model; S3. The feature fusion neural network model processes data through the following sub-steps: S31. Process the first modal data through the first feature extraction branch to obtain a first feature map, and process the second modal data through the second feature extraction branch to obtain a second feature map; S32. Using an attention weight generation module, dynamically generate first weight coefficients based on the contents of the first and second feature maps. Second weighting coefficient ,in ; S33. Combine the first feature map with the first weighting coefficient. Multiply by the second feature map and the second weight coefficient. Multiply the results and sum them to obtain the fused feature map; S4. Based on the fused feature map, a defect detection result containing defect type, location information, and quantization parameters is generated through the output layer of the feature fusion neural network model.

[0007] In addition, the computer vision-based chip defect diagnosis proposed in this application may also have the following additional technical features: In one embodiment of this application, the first sensing unit and the second sensing unit are specifically a combination of an X-ray sensing unit and an optical sensing unit, or a combination of an X-ray sensing unit and an ultrasonic sensing unit; wherein, When the X-ray sensing unit and the optical sensing unit are combined, the first modal data is three-dimensional volume data characterizing the internal structure of the chip, and the second modal data is three-dimensional morphology data characterizing the surface morphology of the chip. When the X-ray sensing unit and the ultrasonic sensing unit are combined, the first modal data is three-dimensional volume data characterizing the internal structure of the chip, and the second modal data is ultrasonic scanning image data characterizing the interface state of the chip.

[0008] In one embodiment of this application, when the first modal data is X-ray three-dimensional volume data, the quantization parameters in the defect detection result include the defect volume calculated based on the defect region segmented by the fused feature map.

[0009] In one embodiment of this application, in step S32, the attention weight generation module is a lightweight convolutional neural network. The lightweight convolutional neural network performs channel concatenation, convolution, global pooling, and Softmax normalization operations on the first feature map and the second feature map to output a scalar form. and ,and , .

[0010] In one embodiment of this application, in step S1, a hardware trigger signal is sent to the first sensing unit and the second sensing unit through a synchronization trigger controller to achieve time synchronization of data acquisition; Furthermore, before data acquisition, a spatial calibration process is used to establish the transformation relationship between the coordinate systems of the first sensing unit and the second sensing unit, which is used to align the first modal data and the second modal data in space.

[0011] In one embodiment of this application, the spatial calibration process includes: Place a standard calibration object with known three-dimensional geometric features on the platform and establish a fixed platform coordinate system; The first sensing unit and the second sensing unit are controlled to scan the standard calibration object respectively to obtain their respective calibration data; The first transformation matrix from the coordinate system of the first sensing unit to the coordinate system of the stage, and the second transformation matrix from the coordinate system of the second sensing unit to the coordinate system of the stage are obtained by calculation. In subsequent testing, the first transformation matrix and the second transformation matrix are used to unify the collected data to the platform coordinate system.

[0012] In one embodiment of this application, the output layer of the feature fusion neural network model is a multi-task output head that simultaneously performs defect segmentation, defect classification, defect parameter regression, and severity assessment tasks.

[0013] A second aspect of this application provides a detection system for implementing the computer vision-based chip defect diagnosis method proposed in the first aspect, comprising: A multimodal sensing acquisition module, wherein the multimodal sensing acquisition module is provided with a first sensing unit based on a first physical principle and a second sensing unit based on different second physical principles; The synchronization control module includes a synchronization trigger controller, which is used to control the acquisition actions of the first sensing unit and the second sensing unit to achieve time synchronization. The digital twin mapping module constructs a three-dimensional digital twin model based on the design file of the chip under test, and maps and marks the defect location and type information in the defect detection results at the corresponding positions in the three-dimensional digital twin model. The computational analysis module is communicatively connected to the multimodal sensing and acquisition module, and stores and runs the pre-trained feature fusion neural network model.

[0014] In addition, the computer vision-based chip defect detection system proposed in this application may also have the following additional technical features: In one embodiment of this application, the multimodal sensing acquisition module specifically includes: A six-axis linear module, wherein the probes of the first and second sensing units are disposed at the z-axis end of the six-axis linear module; The probe of the first sensing unit is an X-ray sensing unit, and the probe of the second sensing unit is an optical sensing unit or an ultrasonic sensing unit.

[0015] In one embodiment of this application, a stage and environment control module is further included, the stage and environment control module comprising: A temperature-controlled stage, which is equipped with a semiconductor cooling chip and a temperature sensor, is used to directly heat or cool the chip under test. An environmental isolation chamber surrounds the temperature-controlled platform to maintain a local constant temperature atmosphere. A temperature controller is communicatively connected to the semiconductor cooling chip, the temperature sensor, and the synchronous trigger controller. It is used to adjust and maintain the chip temperature at a preset value according to the instructions of the synchronous trigger controller.

[0016] Beneficial effects: The chip defect diagnosis method and detection system based on computer vision in this application can comprehensively utilize complementary information from different modal data for cross-validation and comprehensive analysis, thereby significantly reducing the inherent false positive rate and false negative rate of a single detection method and improving the overall accuracy and reliability of defect diagnosis.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a computer vision-based chip defect diagnosis method according to an embodiment of this application; Figure 2 This is a system block diagram of a computer vision-based chip defect diagnosis method according to another embodiment of this application. Detailed Implementation

[0019] Embodiments of this application are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. Rather, embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0020] The following is in conjunction with the appendix Figures 1-2 The present application describes a computer vision-based chip defect diagnosis method, which includes the following steps: S1. For the same chip under test, synchronously or sequentially collect first mode data from the first sensing unit and second mode data from the second sensing unit, wherein the first sensing unit and the second sensing unit operate based on different physical principles; S2. Input the paired first modality data and second modality data into a pre-trained feature fusion neural network model; S3. The feature fusion neural network model processes data through the following sub-steps: S31. Process the first modal data through the first feature extraction branch to obtain the first feature map, and process the second modal data through the second feature extraction branch to obtain the second feature map; S32. An attention weight generation module dynamically generates first weight coefficients based on the contents of the first and second feature maps. Second weighting coefficient ,in ; S33. Combine the first feature map with the first weighting coefficients. Multiply by combining the second feature map with the second weight coefficient. Multiply the results and sum them to obtain the fused feature map; S4. Based on the fused feature map, the output layer of the feature fusion neural network model generates defect detection results containing defect type, location information and quantization parameters.

[0021] Specifically, I. Hardware system and data acquisition.

[0022] The core hardware of the system includes at least two sensing units based on different physical principles (e.g., an X-ray sensing unit and an optical sensing unit), a stage for fixing the chip and providing a temperature-controlled environment, a synchronous trigger controller, and a high-performance computing workstation.

[0023] 1. Synchronous Data Acquisition: The chip under test is placed on the stage. The synchronous trigger controller sends precise hardware trigger signals to each sensing unit, ensuring that their data acquisition time for the same chip area is strictly aligned. Simultaneously, the X-ray source emits pulses, triggering the optical camera to perform exposure.

[0024] 2. Spatial Calibration: System calibration is required before the detection begins. A standard calibration object with known three-dimensional dimensions (such as a calibration plate with a precision array of small balls) is placed on the stage. Each sensing unit scans the calibration object, and the spatial transformation matrix between the coordinate system of each sensing unit and the fixed coordinate system of the stage is calculated. In subsequent detection, these matrices are applied to uniformly transform all acquired data to the same spatial coordinate system, achieving spatial alignment of the data.

[0025] II. Data Preprocessing and Feature Extraction.

[0026] 1. The collected raw data needs to be preprocessed for model analysis.

[0027] X-ray data processing: Three-dimensional reconstruction is performed on multi-angle projection data, such as using a filtered back projection algorithm to generate grayscale data characterizing the internal structure of the chip. Subsequent processing includes normalization and noise reduction.

[0028] 2. Optical data processing: The acquired surface images are reconstructed in three dimensions to obtain data characterizing the chip surface height, texture, and other information, and then normalized.

[0029] 3. Data pairing: Using the transformation matrix obtained from spatial calibration, pre-processed 3D volume data and surface topography data from different sensors are precisely registered in space to form paired multimodal data pairs with corresponding spatial locations.

[0030] III. Processing flow of the feature fusion neural network model.

[0031] The preprocessed paired data is input into a pre-trained feature fusion neural network model. This model is the core of the method, and the processing flow is as follows: Branch Feature Extraction: The model has parallel feature extraction branch networks. The first branch (such as a 3D convolutional neural network) processes X-ray volume data and extracts internal structure feature maps. ; The second branch (such as a 2D convolutional neural network) processes optical surface data and extracts surface morphology feature maps. .

[0032] Dynamic weight generation and fusion: The attention weight generation module in the model receives... and This module analyzes the contents of two feature maps and dynamically assigns importance weights to them. and (satisfy This process allows the model to adaptively determine which modality of features is more suitable for analysis of the current region (e.g., X-ray features are more relevant when analyzing internal bubbles, while optical features are more relevant when analyzing surface scratches). The model then performs a weighted fusion calculation: This yields an enhanced feature map that incorporates dual-modal information.

[0033] Multi-task decision output: fused feature maps The data is fed into a multi-task output head and multiple analysis tasks are executed in parallel: Defect segmentation: Outputs a 3D segmentation mask that precisely marks each voxel or pixel where the defect is located.

[0034] Defect Classification: Output the type of defect (such as voids, cracks, delamination, foreign matter, etc.) and its confidence level.

[0035] Defect localization: Outputs the position coordinates and bounding box of the defect in three-dimensional space.

[0036] Quantitative analysis: Based on the segmentation results, calculate the physical parameters of the defects, such as volume, area, and depth.

[0037] IV. Training and Optimization.

[0038] This feature fusion neural network model requires training with a large number of labeled chip defect samples. The training data includes paired X-ray and optical data, as well as manually labeled real defect information (such as segmentation masks, type labels, bounding boxes, and measurement volumes). A multi-task loss function is used during training to simultaneously optimize segmentation accuracy, classification accuracy, localization error, and quantization error, enabling the model to learn the mapping relationship from multimodal data to the comprehensive detection results end-to-end.

[0039] V. Result Generation and Output.

[0040] After the model inference is completed, the system post-processes the output and generates a structured report.

[0041] Results integration: Integrate the results of segmentation, classification, location and quantification to form a complete description of each defect.

[0042] Acceptance criteria: The chip is automatically judged as qualified based on preset process standards (such as maximum allowable porosity, upper limit of single defect volume, etc.).

[0043] Reporting and Visualization: Generate a comprehensive inspection report that includes a defect list, statistical charts (such as defect size distribution and type distribution), a 3D visualization view (the location of defects in the chip digital model), and pass / fail determination.

[0044] In one embodiment of this application, the first sensing unit and the second sensing unit are specifically a combination of an X-ray sensing unit and an optical sensing unit, or a combination of an X-ray sensing unit and an ultrasonic sensing unit; wherein, When the X-ray sensing unit and the optical sensing unit are combined, the first modal data is three-dimensional volume data characterizing the internal structure of the chip, and the second modal data is three-dimensional morphology data characterizing the surface morphology of the chip. When the X-ray sensing unit and the ultrasonic sensing unit are combined, the first modal data is three-dimensional volume data characterizing the internal structure of the chip, and the second modal data is ultrasonic scanning image data characterizing the interface state of the chip.

[0045] Specifically, here are the actual usage instructions for both combinations.

[0046] Combination 1: X-ray + Optics ("Inside-out" inspection).

[0047] Application: Primarily used for comprehensive quality inspection after packaging, simultaneously checking for internal voids and foreign objects, as well as surface defects such as scratches, contamination, and warping.

[0048] Process: After the chip is fixed, the system controls the X-ray source and high-resolution optical camera to scan synchronously. The X-rays reconstruct the internal three-dimensional structure, while the optical camera captures the surface three-dimensional morphology. The fusion model is automatically analyzed; for example, if a suspected internal shadow is found (X-ray data), the corresponding surface is checked for smoothness (optical data), thereby comprehensively determining whether it is an internal bubble or surface structure interference, and accurately calculating the bubble volume or scratch depth.

[0049] Output: A comprehensive quality control report that includes both an internal volumetric defect list and a surface morphology defect report.

[0050] Combination 2: X-ray + ultrasound ("transparent interface" inspection).

[0051] Application: Specifically used to evaluate the bonding quality between material layers inside a chip, especially for interface defects such as delamination and poor adhesion. It is commonly used in the reliability verification of high-reliability chips.

[0052] Process: The chip contacts the ultrasound probe via a coupling agent (such as water). The system controls X-rays to perform a comprehensive CT scan, while the ultrasound probe emits high-frequency sound waves and receives echoes from the interfaces of each layer. The fusion model combines the three-dimensional structural image from the X-rays with the interface reflection-sensitive image from the ultrasound.

[0053] In one embodiment of this application, when the first modal data is X-ray three-dimensional volume data, the quantization parameters in the defect detection result include the defect volume calculated based on the defect region segmented from the fused feature map.

[0054] Specifically, the actual processing flow is as follows: Step 1: Intelligent 3D segmentation.

[0055] After receiving X-ray 3D volume data and optical surface data, the fusion model (trained with a large amount of data) uses its internal feature fusion mechanism to more accurately understand the true 3D morphology of defects. The "fusion feature map" output by the model is processed by the segmentation head to generate a high-precision 3D segmentation mask.

[0056] Step 2: Precise volume calculation.

[0057] The system then performs a deterministic calculation: Counting: Count the total number of voxels contained in this 3D segmentation mask. A voxel is a 3D pixel that represents a tiny cubic unit.

[0058] Conversion: Multiply the number of voxels by the actual physical volume of a single voxel.

[0059] The calculation formula is: Defect true volume = Total number of voxels × Physical volume of a single voxel.

[0060] In one embodiment of this application, in step S32, the attention weight generation module is a lightweight convolutional neural network. The lightweight convolutional neural network performs channel concatenation, convolution, global pooling, and Softmax normalization operations on the first feature map and the second feature map to output a scalar form. and ,and , .

[0061] The attention weight generation module is constructed as a lightweight convolutional neural network with a continuous and deterministic processing flow. The module receives feature maps from the first feature extraction branch. Feature map of the second feature extraction branch As inputs, both have the same spatial dimension but carry feature information of different modalities.

[0062] The processing flow is as follows: Channel stitching: combining feature maps and The features are concatenated along the channel dimension to form a combined feature map. The number of channels in the combined feature map is the sum of the two features. , This is a commonly used concatenation function in neural networks, which connects multiple feature maps along a specified dimension. This indicates that the first feature map With the second feature map The feature map is spliced ​​along the channel dimension to generate a combined feature map with the sum of the two channels.

[0063] Convolution processing: Perform one or more 1×1 convolution operations on the combined feature maps. This operation enables cross-channel feature interaction and information integration, and reduces the number of channels to a preset low dimension to meet lightweight design requirements.

[0064] Global pooling: A global average pooling operation is performed on the feature map output by the convolution, aggregating the spatial features of each channel into a single scalar value, thereby generating a one-dimensional feature vector. This step eliminates the influence of spatial dimension, making the generated weight coefficients applicable to the entire feature map.

[0065] Linear Transformation and Normalization: The one-dimensional feature vector obtained by global pooling is linearly transformed through a fully connected layer (or an equivalent 1×1 convolution) with two output nodes to obtain two initial scalar values. and .

[0066] Subsequently, the two scalars are normalized using the Softmax function, specifically calculated as follows: , Output value and The constraints are satisfied: ,and , ,in, Based on the natural constant An exponential function with base 0. Indicates computation on the input scalar of Power of 1 Indicates computation on the input scalar of The power operation is used in the Softmax normalization process to convert the original scalar obtained by the linear transformation into a positive number so that the normalization weights can be calculated later.

[0067] The final output scalar and That is, to be assigned to the first feature map Second feature map The fusion weights. These two weights will be used directly to perform the weighted fusion calculation: This generates a fused feature map. .

[0068] In one embodiment of this application, in step S1, a hardware trigger signal is sent to the first sensing unit and the second sensing unit through a synchronization trigger controller to achieve time synchronization of data acquisition. Furthermore, before data acquisition, a spatial calibration process is used to establish the transformation relationship between the coordinate systems of the first and second sensing units, which is used to align the first and second modal data in space.

[0069] In actual operation, the system first sends precise hardware trigger signals to the first and second sensing units through the synchronous trigger controller, forcing both to start data acquisition at the same time, thereby ensuring that the obtained information inside and on the surface of the chip has strict time alignment.

[0070] Before data acquisition begins, a spatial calibration process must be performed to establish a unified spatial reference. Specifically, a standard calibration object with known and precise three-dimensional dimensions is placed on the testing platform, and the first and second sensing units are controlled to scan the calibration object respectively.

[0071] By calculating the position of the feature points on the calibration object in the coordinate system of each sensor and combining it with their known position in the coordinate system of the platform, the transformation matrix from the coordinate system of each sensor to the fixed coordinate system of the platform is calculated.

[0072] In subsequent formal chip testing, the system will apply these pre-calculated transformation matrices to uniformly transform the real-time acquired first and second modal data to the same stage coordinate system, thereby achieving accurate registration and alignment of multimodal data in three-dimensional space.

[0073] In one embodiment of this application, the spatial calibration process is as follows: Place a standard calibration object with known three-dimensional geometric features on the platform and establish a fixed platform coordinate system; The first and second sensing units are controlled to scan the standard calibration object respectively to obtain their respective calibration data; The first transformation matrix from the coordinate system of the first sensing unit to the coordinate system of the stage, and the second transformation matrix from the coordinate system of the second sensing unit to the coordinate system of the stage are obtained by calculation. In subsequent testing, the first and second transformation matrices are used to unify the collected data to the platform coordinate system.

[0074] Specifically, in actual operation, the spatial calibration process is executed in the following steps: First, a standard calibration object with known and accurate three-dimensional geometric features is fixed on the platform of the system, and the physical position of the platform at this time is defined as a fixed platform coordinate system.

[0075] Next, the system controls the first sensing unit to scan the calibration object and reconstruct its three-dimensional volume data. At the same time, it controls the second sensing unit to scan the same calibration object and obtain its surface three-dimensional point cloud or depth map, thereby obtaining two sets of calibration data in their respective sensor coordinate systems.

[0076] Then, through feature point extraction and registration algorithms, the theoretical coordinates of the same set of feature points on the calibration object in the stage coordinate system, the coordinates in the first sensing unit data, and the coordinates in the second sensing unit data are automatically identified and matched.

[0077] Based on these corresponding point sets, the system calculates the first transformation matrix from the first sensing unit coordinate system to the stage coordinate system and the second transformation matrix from the second sensing unit coordinate system to the stage coordinate system using mathematical methods such as least squares.

[0078] After this calibration is completed, the two transformation matrices will be stored and locked by the system.

[0079] In the subsequent formal testing of all chips under test, the system will automatically call the stored first transformation matrix to transform the real-time acquired first modal data to the stage coordinate system; at the same time, it will call the second transformation matrix to transform the acquired second modal data to the same stage coordinate system, thereby ensuring that the data from sensors with different physical principles achieve pixel-level or voxel-level precise spatial alignment under the same three-dimensional spatial reference, providing a geometrically consistent data foundation for subsequent feature fusion and defect diagnosis.

[0080] In one embodiment of this application, the output layer of the feature fusion neural network model is a multi-task output head that simultaneously performs defect segmentation, defect classification, defect parameter regression, and severity assessment tasks.

[0081] Specifically, the defect segmentation task generates a high-resolution pixel-level or voxel-level segmentation mask based on the fused feature map. This mask accurately outlines the region contour of each suspected defect in the image or 3D volume data in the form of a binary or probabilistic map.

[0082] Simultaneously, the defect classification task performs feature analysis on each segmented independent defect region to determine its specific defect type (such as voids, cracks, delamination, or foreign matter), and outputs the corresponding category label and a confidence score. The defect parameter regression task works concurrently, calculating a series of quantitative physical parameters for the segmented and classified defects. For example, it calculates the defect volume based on the number of voxels and physical dimensions in the 3D segmentation mask, or measures geometric properties such as the length and width of cracks.

[0083] Finally, the severity assessment task will comprehensively evaluate the severity of each defect by combining the classification results, quantitative parameters (such as defect volume and location depth) and preset process quality standards, and classify them into different levels, or directly determine whether they exceed the acceptance standards.

[0084] This application proposes a detection system for implementing a computer vision-based chip defect diagnosis method, such as... Figure 2 As shown, it includes: The multimodal sensing acquisition module is equipped with a first sensing unit based on a first physical principle and a second sensing unit based on different second physical principles. The synchronization control module includes a synchronization trigger controller, which is used to control the acquisition actions of the first sensing unit and the second sensing unit to achieve time synchronization. The digital twin mapping module constructs a three-dimensional digital twin model based on the design file of the chip under test, and maps and marks the defect location and type information in the defect detection results to the corresponding positions in the three-dimensional digital twin model. The computational analysis module is connected to the multimodal sensing and acquisition module to store and run pre-trained feature fusion neural network models.

[0085] Furthermore, the multimodal sensing acquisition module specifically includes: The probes of the first and second sensing units of the six-axis linear module are set at the z-axis end of the six-axis linear module. The probe of the first sensing unit is an X-ray sensing unit, and the probe of the second sensing unit is an optical sensing unit or an ultrasonic sensing unit.

[0086] Furthermore, it also includes a stage and environment control module, which includes: The temperature control stage is equipped with a semiconductor cooling chip and a temperature sensor, which are used to directly heat or cool the chip under test. An environmental isolation chamber surrounds the temperature control platform and sensing area to maintain a local constant temperature atmosphere. The temperature controller communicates with the thermoelectric cooler, temperature sensor, and synchronous trigger controller to adjust and maintain the chip's temperature at a preset value according to the instructions of the synchronous trigger controller.

[0087] In actual operation, the testing system works according to the procedure: the operator first places the chip to be tested in the center area of ​​the temperature control platform of the stage and the environmental control module and fixes it.

[0088] Subsequently, the detection scheme is set through the software interface, including the target temperature, scanning path, and detection parameters. After the system starts, the temperature controller drives the semiconductor cooling chip to actively heat or cool the chip according to the instructions, while the temperature sensor provides real-time feedback and performs closed-loop adjustment until the chip temperature reaches and stabilizes at the preset value.

[0089] The environmental isolation chamber maintains a stable local environment during this period to reduce external interference.

[0090] When the temperature is ready, the synchronous trigger controller in the synchronous control module issues a command to coordinate the multimodal sensing and acquisition module to start working: the first sensing unit probe (such as an X-ray source and detector) and the second sensing unit probe (such as an optical camera or ultrasonic transducer) mounted on the Z-axis end of the high-precision six-axis linear module move to the starting scanning position according to the planned path.

[0091] At each sampling point, the synchronous trigger controller sends a precise hardware trigger signal to ensure that the X-ray pulse emission is strictly synchronized with the optical camera exposure (or ultrasonic pulse emission), thereby acquiring time-aligned multimodal raw data.

[0092] The collected data is transmitted to the computing and analysis module in real time. This module calls a pre-trained feature fusion neural network model to process the spatially registered data pairs and perform defect identification, segmentation, classification and quantitative analysis.

[0093] The generated defect detection results (including information such as 3D location, type, and size) are sent to the digital twin mapping module. This module automatically imports the design files (such as CAD models) of the chip model, builds or calls the corresponding 3D digital twin model, and after converting the spatial coordinates of each defect, accurately marks it on the corresponding structural position of the twin model with visual identifiers (such as highlighting, color coding, and annotation labels), thereby generating a digital inspection report that corresponds one-to-one with the physical chip and whose defect information is intuitively verifiable, completing the entire process from physical inspection to digital mirroring.

[0094] This application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements a chip defect diagnosis method based on computer vision.

[0095] In summary, the computer vision-based chip defect diagnosis method and detection system of this application can comprehensively utilize complementary information from different modal data for cross-validation and comprehensive analysis, thereby significantly reducing the inherent false positive and false negative rates of a single detection method and improving the overall accuracy and reliability of defect diagnosis.

[0096] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0097] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0098] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A chip defect diagnosis method based on computer vision, characterized in that, Includes the following steps: S1. For the same chip under test, synchronously or sequentially collect first mode data from the first sensing unit and second mode data from the second sensing unit, wherein the first sensing unit and the second sensing unit operate based on different physical principles; S2. Input the paired first modality data and second modality data into a pre-trained feature fusion neural network model; S3. The feature fusion neural network model processes data through the following sub-steps: S31. Process the first modal data through the first feature extraction branch to obtain a first feature map, and process the second modal data through the second feature extraction branch to obtain a second feature map; S32. Using an attention weight generation module, dynamically generate first weight coefficients based on the contents of the first and second feature maps. Second weighting coefficient ,in ; S33. Combine the first feature map with the first weighting coefficient. Multiply by the second feature map and the second weight coefficient. Multiply the results and sum them to obtain the fused feature map; S4. Based on the fused feature map, a defect detection result containing defect type, location information, and quantization parameters is generated through the output layer of the feature fusion neural network model.

2. The method according to claim 1, characterized in that, The first sensing unit and the second sensing unit are specifically a combination of an X-ray sensing unit and an optical sensing unit, or a combination of an X-ray sensing unit and an ultrasonic sensing unit; wherein, When the X-ray sensing unit and the optical sensing unit are combined, the first modal data is three-dimensional volume data characterizing the internal structure of the chip, and the second modal data is three-dimensional morphology data characterizing the surface morphology of the chip. When the X-ray sensing unit and the ultrasonic sensing unit are combined, the first modal data is three-dimensional volume data characterizing the internal structure of the chip, and the second modal data is ultrasonic scanning image data characterizing the interface state of the chip.

3. The method according to claim 2, characterized in that, When the first modal data is X-ray three-dimensional volume data, the quantization parameters in the defect detection result include the defect volume calculated based on the defect region segmented by the fused feature map.

4. The method according to claim 1, characterized in that, In step S32, the attention weight generation module is a lightweight convolutional neural network. This lightweight convolutional neural network performs channel concatenation, convolution, global pooling, and Softmax normalization operations on the first and second feature maps to output a scalar value. and ,and , .

5. The method according to claim 1, characterized in that, In step S1, a hardware trigger signal is sent to the first sensing unit and the second sensing unit through a synchronous trigger controller to achieve time synchronization of data acquisition; Furthermore, before data acquisition, a spatial calibration process is used to establish the transformation relationship between the coordinate systems of the first sensing unit and the second sensing unit, which is used to align the first modal data and the second modal data in space.

6. The method according to claim 5, characterized in that, The spatial calibration process includes: Place a standard calibration object with known three-dimensional geometric features on the platform and establish a fixed platform coordinate system; The first sensing unit and the second sensing unit are controlled to scan the standard calibration object respectively to obtain their respective calibration data; The first transformation matrix from the coordinate system of the first sensing unit to the coordinate system of the stage, and the second transformation matrix from the coordinate system of the second sensing unit to the coordinate system of the stage are obtained by calculation. In subsequent testing, the first transformation matrix and the second transformation matrix are used to unify the collected data to the platform coordinate system.

7. The method according to claim 1, characterized in that, The output layer of the feature fusion neural network model is a multi-task output head, which simultaneously performs defect segmentation, defect classification, defect parameter regression, and severity assessment tasks.

8. A detection system for implementing the method according to any one of claims 1-7, characterized in that, include: A multimodal sensing acquisition module, wherein the multimodal sensing acquisition module is provided with a first sensing unit based on a first physical principle and a second sensing unit based on different second physical principles; The synchronization control module includes a synchronization trigger controller, which is used to control the acquisition actions of the first sensing unit and the second sensing unit to achieve time synchronization. The digital twin mapping module constructs a three-dimensional digital twin model based on the design file of the chip under test, and maps and marks the defect location and type information in the defect detection results at the corresponding positions in the three-dimensional digital twin model. The computational analysis module is communicatively connected to the multimodal sensing and acquisition module, and stores and runs the pre-trained feature fusion neural network model.

9. The system according to claim 8, characterized in that, The multimodal sensing acquisition module specifically includes: A six-axis linear module, wherein the probes of the first and second sensing units are disposed at the z-axis end of the six-axis linear module; The probe of the first sensing unit is an X-ray sensing unit, and the probe of the second sensing unit is an optical sensing unit or an ultrasonic sensing unit.

10. The system according to claim 8, characterized in that, It also includes a stage and environment control module, which includes: A temperature-controlled stage, which is equipped with a semiconductor cooling chip and a temperature sensor, is used to directly heat or cool the chip under test. An environmental isolation chamber surrounds the temperature-controlled platform to maintain a local constant temperature atmosphere. A temperature controller is communicatively connected to the semiconductor cooling chip, the temperature sensor, and the synchronous trigger controller. It is used to adjust and maintain the chip temperature at a preset value according to the instructions of the synchronous trigger controller.

Images