Chip bump submicron defect online detection method

Abstract

The invention relates to the technical field of chip detection, in particular to a chip bump submicron defect online detection method which comprises the following steps: generating a bump submicron simulation image by using a simulation model; performing super-resolution processing on the simulation image by using the improved ESRGAN model; performing physical self-supervision augmentation on the super-resolution image; constructing a mixed image by using the augmented image and the production line image; inputting the mixed image into a meta-learning model, and outputting a defect type; and outputting a defect position by using a cascade network. According to the method, the problems of sub-pixel defect form distortion and poor defect image detection generalization performance in existing super-resolution image processing are solved.

Description

Technical Field

[0001] This invention relates to the field of chip inspection technology, and in particular to an online method for detecting submicron defects in chip bumps. Background Technology

[0002] With the mass production of 3nm node flip chips and wafer-level hybrid bonding technology, microbumps, microbridges, submicron voids, or top depressions all need to be detected online. However, the probability of such defects is only 0.05%-0.1%. Traditional automated optical inspection (AOI) cannot balance resolution and speed within the production line cycle time, becoming a bottleneck that limits yield improvement. Existing methods face a physical trade-off of "resolution ↑ - speed ↓", and have not yet achieved full nanometer-level inspection at the production line.

[0003] Existing methods use Monte Carlo BRDF rendering to generate convex defect maps to solve the data scarcity problem; however, its simulation only considers geometric optics and does not couple the optical system's point spread function (PSF) with pixel sampling aliasing, resulting in subpixel defect morphology distortion.

[0004] Existing deep learning methods generally rely on a single mapping of "appearance features - defect category", lacking quantitative modeling of the entire "defect → process parameters → reliability" chain; in addition, existing detection models all stop at the binary classification of "defect presence or absence", and have not established a real-time link with packaging reliability standards.

[0005] The existing methods have the following shortcomings:

[0006] 1. Resolution and production line cycle time cannot be balanced; 2. Subpixel defect simulation is distorted; 3. Poor generalization performance in small sample scenarios; 4. Data silos lead to failure of cross-factory knowledge sharing; 5. Lack of real-time closed loop of "defect-process-reliability". Summary of the Invention

[0007] To address the shortcomings of existing methods, this invention proposes a super-resolution-physical self-supervised joint augmentation, a reliability-driven meta-learning federated framework, and an integrated sensing and computing pulse-quantum chip. While maintaining a chip cycle time of 3 s, it improves the bridging defect detection rate and outputs online lifetime prediction, achieving a leap from "image judgment" to "lifetime judgment".

[0008] The technical solution adopted in this invention is: an online detection method for submicron defects in chip bumps, comprising the following steps:

[0009] Step 1: Generate a submicron simulation image of the convex point using a simulation model;

[0010] In a preferred embodiment of the present invention, the construction of the simulation model includes:

[0011] Create an output directory, clean up the scene, and add a background plane and camera;

[0012] Randomly select roughness level to generate random location and defect model;

[0013] Add SnAgCu material properties to the defect model and add a ring light source; render the current scene as an image file; clean up the scene.

[0014] As a preferred embodiment of the present invention, the defects in the image include: micro-bridging, top depression, submicron voids, excessive ellipsoidality, sidewall burrs, and insufficient bump height.

[0015] Step 2: Perform super-resolution processing on the simulation images using the improved ESRGAN model; perform physical self-supervised augmentation on the super-resolution images; construct a hybrid image using the augmented image and the production line image;

[0016] As a preferred embodiment of the present invention, the improved ESRGAN model includes:

[0017] Generative networks and discriminative networks;

[0018] The generator network includes: inputting an image into a 3×3 Conv and outputting a feature map F0;

[0019] F0 simultaneously inputs the main branch and the bypass branch. The main branch consists of twelve cascaded residual dense sub-modules. A 1×1 Conv is inserted after every 3 residual dense sub-modules.

[0020] In a preferred embodiment of the present invention, the residual dense submodule includes: inputting the input feature map sequentially into 3×3DSConv, 1×1PConv, and 3×3DSConv, multiplying it by a scaling factor, and then adding it to the input feature map.

[0021] The bypass branch includes: converting F0 into a 7×1 Conv in the horizontal direction and a 1×7 Conv in the vertical direction; then adding the horizontal and vertical feature maps pixel by pixel, activating them with ReLU, and feeding them into a 1×1 Conv; then performing a 1×1 Conv, and obtaining the weight vector through Sigmoid; and multiplying the weight vector with the main branch pixel by pixel.

[0022] The feature maps of the main branch and the bypass branch are added together to output the feature map F1;

[0023] The upsampling stage sequentially goes through the first nearest neighbor interpolation, 48 3×3 Convs, the second nearest neighbor interpolation, 24 3×3 Convs, and 3×3×3 Convs.

[0024] In a preferred embodiment of the present invention, the discrimination network includes:

[0025] The first layer consists of 5×5 Conv and 2×2 Maxpool; the second layer consists of 3×3 D Conv, bilinear sampling, and 3×3 Conv; the third layer divides the channel feature map into four groups along the channel dimension: the first group is 1×1 Conv, the second group is 3×3 Conv, the third group is 5×5 Conv, and the fourth group is 7×7 Conv. The outputs of the four groups are concatenated and then processed by 1×1 Conv; the fourth layer performs spatial pyramid pooling in parallel, which meshes the channel feature map. Global average pooling is used inside the mesh, followed by vector flattening, concatenation, and layer-by-layer compression with two fully connected layers.

[0026] As a preferred embodiment of the present invention, physical self-supervised augmentation is calculated by determining that the difference in brightness of the re-rendered image is lower than the error threshold and the confidence level is higher than the confidence threshold.

[0027] Step 3: Input the mixed image into the meta-learning model to output the defect type; use the cascaded network to output the defect location;

[0028] As a preferred embodiment of the present invention, the meta-learning model includes:

[0029] The image stream is sequentially input into ResNet50, GAPool and FC layers to obtain visual features;

[0030] DVS stream input to FC layer to obtain pulse characteristics;

[0031] The process flow is input into the FC layer to obtain the process flow characteristics;

[0032] Visual features, pulse features, and process flow features are concatted to obtain stitched features; the stitched features are input into the FC layer and the output is the fused features.

[0033] The fused features are input into the classification head and regression head respectively.

[0034] As a preferred embodiment of the present invention, the cascaded network includes YOLOv5s and Faster R-CNN.

[0035] As a preferred embodiment of the present invention, the chip bump submicron defect online detection system includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the chip bump submicron defect online detection method.

[0036] The beneficial effects of this invention are:

[0037] 1. Utilize Blender's micro-facet-based BRDF model for defect image simulation to improve the simulation effect of sub-pixel defects;

[0038] 2. Construct an improved ESRGAN model to reduce the network structure while increasing image resolution;

[0039] 3. Construct a physical self-supervised augmentation standard to remove anomalous images;

[0040] 4. Construct a simplified lifespan prediction model: Construct labels from initial area to actual lifespan to provide data support for the meta-learning model;

[0041] 5. Improve the accuracy and recall of microchip defects by training a meta-learning model based on fully connected networks using images, DVS streams, and process flow. Attached Figure Description

[0042] Figure 1 This is a flowchart of the online detection method for submicron defects in chip bumps according to the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0044] In recent years, with the mass production of 3nm node flip chips and wafer-level hybrid bonding technology, the microbump pitch has been reduced to below 40μm, and the bump diameter itself has been reduced to 15μm and the height to less than 10μm. As a result, the requirements for package-level reliability of high-end processors such as Apple A17 and NVIDIA H100 have been reduced from 100ppm to 10ppm. Any microbridges below 1μm, submicron voids or top depressions of 0.2μm must be detected online. However, the probability of such extreme defects is only 0.05%-0.1%, which is a typical "rare event". Traditional automated optical inspection (AOI) is difficult to balance resolution and speed within the production line cycle, which has become a bottleneck limiting yield improvement.

[0045] The industry has successively launched solutions such as multi-channel LED ring lighting, laser scanning confocal, and quantum entangled lighting, which can achieve 2μm lateral resolution at a speed of 60mm² / s. However, the measured false negative rate for micro-bridges smaller than 1μm is still as high as 18%. The fundamental reason is that the visible light diffraction limit σ=0.61λ / NA≈230nm, and the micro-bridge scattering signal is only 1.2× higher than the background noise, making it easily submerged by photon shot noise. Although laser confocal lighting improves the resolution to 150nm, the working distance of the 0.8 NA microscope objective is less than 0.3mm, which cannot adapt to 300mm wafers with warpage >50μm, and the equipment cost is three times that of conventional AOI. The latest publicly disclosed quantum entangled light source solution theoretically advances the resolution to 12nm, but the entangled photon flux on the market is only 10 9The speed per photons / s·mm² increases the single-frame acquisition time by 20×, and the inspection cycle of the entire chip is >5 s, which directly conflicts with the cycle time requirement of ≤3 s for flip-chip production lines. The above solutions all face the physical trade-off of "resolution ↑ - speed ↓", and have not yet achieved nanometer-level full inspection at the production line.

[0046] One study used Monte Carlo BRDF rendering to generate 30,000 bump defect images, which were then mixed with real samples to train a deep network. The recall for 1μm bridging was 89%, but the recall for 0.5μm bridging dropped sharply to 54%. The simulation only considered geometric optics and did not couple the optical system point spread function (PSF) with pixel sampling aliasing, resulting in sub-pixel defect morphology distortion. Another study used MAML for defect detection for the first time, achieving 95% accuracy in 5-shot PCB datasets. However, the experimental area was only 10×10 mm, and the speckle noise caused by the 30nm roughness of the wafer surface was not considered. When directly transferred to the bump scene, the accuracy dropped by 22%. More importantly, existing deep learning methods generally rely on a single mapping of "appearance features - defect category" and lack quantitative modeling of the entire "defect → process parameters → reliability" link. They cannot predict the lifetime degradation of micro-defects under subsequent thermal-electrical-mechanical stress online. The manufacturing end still needs to wait 2-3 weeks for accelerated lifetime recovery, missing the online process correction window.

[0047] Cross-factory knowledge sharing is also limited by data privacy; a 2025 international conference reported that several IDMs attempted to jointly build a "defect federation," but due to the 3D nano-morphology in SEM images, each factory was only willing to share gradient signatures; traditional federation smoothed out the high-frequency details of rare defects, reducing cross-factory recalls by 9%-15%, but failed to resolve the contradiction between "data not leaving the factory" and "small sample scarcity"; in addition, existing inspections are limited to a binary classification of "defect presence or absence," and have not established a real-time link with packaging reliability standards; industry standards indicate that a 0.5μm bridging can grow into a 5μm crack in 500 hours at 125°C and 1.2V, leading to early failure; however, existing inspection systems cannot provide lifetime prediction within the current cycle time, and the manufacturing end can only scrap it afterward, increasing costs by 10%-15%.

[0048] like Figure 1 As shown, an online detection method for submicron defects in chip bumps includes the following steps:

[0049] Step 1: Generate a submicron simulation image of the convex point using a simulation model;

[0050] The simulation model is based on a micro-facet BRDF model using Blender software and optical plugins.

[0051] The construction of the simulation model includes:

[0052] 1. Initialization and parameter settings:

[0053] The output directory, total number of images, number of images per batch, resolution, pixel size, roughness level, material properties, and wavelength are among the parameters that control the overall settings and physical properties of the generated images.

[0054] 2. Scene cleanup and basic settings:

[0055] The `clean_scene()` function is used to clear all objects in the Blender scene, ensuring that the scene is clean each time a new image is generated.

[0056] The add_subdiv_plane() function creates a 2×2 mm subdivision plane as a background to simulate the chip surface;

[0057] The sn_agcu_material() function defines the properties of SnAgCu material, including basic color, roughness, refractive index, etc., to simulate the optical properties of chip bump materials;

[0058] The add_ring_light() function adds a ring light source, whose position and direction are randomly generated to simulate lighting effects from different angles;

[0059] The setup_camera() function sets camera parameters, including lens focal length, sensor size, position, and rotation, to simulate the imaging effect of a camera lens;

[0060] 3. Rendering process:

[0061] The render_to_file() function sets the rendering engine to Cycles, uses GPU acceleration, and sets parameters such as rendering resolution and file format, and saves the rendering result as TIFF format;

[0062] 4. Configure the main program:

[0063] The `main()` function is the main entry point of the program. It first creates the output directory, then cleans the scene and adds a background plane and camera. Within the loop, it creates a subdirectory for each image to store the batch of images. It randomly selects a roughness level and generates an ellipsoidal depression with a random location and shape as a defect model. It adds SnAgCu material properties to the defect model. It adds a ring light source and sets the position and direction of the light source. It calls the rendering function to render the current scene as an image file. It cleans the scene to prepare for the generation of the next image. Every 100 images generated, it prints progress information.

[0064] 5. Execution procedure:

[0065] When the script runs as the main program, the main() function is called to start generating images.

[0066] The above process can efficiently generate a large number of chip bump simulation images with random defects and lighting conditions, which can be used for subsequent image analysis and defect detection research.

[0067] Geometric scanning: bump diameter 10-150 μm (step 5 μm), height 5-100 μm (step 5 μm), pitch 10-200 μm (step 10 μm); array 400×400, random position deviation 0-2 μm;

[0068] Surface scan: SnAgCu solder, roughness Ra 0.01-2 μm (logarithmic interval 16), refractive index n=1.80±0.05;

[0069] Illumination scanning: 405 nm LED, incident angle 0-80° (in 5° increments), azimuth angle 0-360° random, intensity 300-800 lux;

[0070] Camera parameters: NA 0.9, 3.45 μm pixels, 5× magnification, 250 nm / px pixel resolution, 0.5 mm depth of field.

[0071] Defect images include: microbridges, top depressions, submicron cavities, excessive ellipsoids, sidewall burrs, and insufficient bump height.

[0072] Among them, micro-bridge refers to the presence of SnAgCu metal wires between adjacent bumps, with a minimum width of 0.3 µm and a step size of 0.1 µm;

[0073] The top-pit is a bowl-shaped depression formed on the surface of the convex point due to excessive cleaning by air bubbles or laser, with a diameter of 0.4-2.0 µm and a depth of ≥0.2 µm;

[0074] Sub-μm voids are spherical air holes inside protrusions or at interfaces, with diameters of 0.3-1.5 µm and step sizes of 0.1 µm.

[0075] Excessive ellipsoidality indicates that insufficient photoresist development results in an elliptical cross-section of the bumps, with a major-to-minor axis ratio >1.3 and a major axis exceeding the standard by 0.5-2.0 µm.

[0076] Side-wall burrs are caused by incomplete removal of adhesive after electroplating, resulting in serrated protrusions of 0.3-1.0 µm at the root of the bumps.

[0077] Insufficient bump height (Height-short) indicates that the electroplating current density is too low, and the bump height is 0.5-2.0µm lower than the design value, with a step size of 0.1µm.

[0078] Python pseudocode:

[0079] # ==================================================================

[0080] # bump_render.py — A generator for simulating chip bump defects

[0081] # Blender 3.5 + Cycles (GPU) Single frame < 0.8 s @RTX4090

[0082] # ==================================================================

[0083] import bpy, os, math, json, numpy as np

[0084] import Vector from mathutils

[0085] # ---------- User-adjustable parameters----------

[0086] OUT_DIR = r" / data / bump_sim" # Output root directory

[0087] TOTAL_IMG = 620000 # Total number of images

[0088] BATCH = 1000 # Number of pages per subdirectory

[0089] RESOLUTION = 4096 pixels

[0090] PIXEL_SIZE = 0.25e-6 # 250 nm / px → 1 µm / px before downsampling

[0091] ROUGH_LOG = np.logspace(-2, 0.3, 16) # 0.01–2 µm logarithmic 16-bit range

[0092] N_SnAgCu = 1.80 # 405 nm refractive index

[0093] K_SnAgCu = 1.68 # Extinction coefficient

[0094] WAVELENGTH = 405e-9 # 405 nm

[0095] # ---------------------------------

[0096] def clean_scene():

[0097] bpy.ops.object.select_all(action='SELECT')

[0098] bpy.ops.object.delete(use_global=False)

[0099] def add_subdiv_plane(size=2e-3, subdiv=10):

[0100] """2×2 mm plane for use as background"""

[0101] bpy.ops.mesh.primitive_plane_add(size=size, location=(0,0,-5e-6))

[0102] plane = bpy.context.object

[0103] bpy.ops.object.modifier_add(type='SUBSURF')

[0104] plane.modifiers["Subdivision"].levels = subdiv

[0105] bpy.context.view_layer.objects.active = plane

[0106] bpy.ops.object.modifier_apply(modifier="Subdivision")

[0107] return plane

[0108] def sn_agcu_material(name, rough):

[0109] mat = bpy.data.materials.new(name)

[0110] mat.use_nodes = True

[0111] tree = mat.node_tree

[0112] tree.clear()

[0113] bsdf = tree.new(type='ShaderNodeBsdfPrincipled')

[0114] bsdf.location = (0,0)

[0115] bsdf.inputs['Base Color'].default_value = (0.8, 0.8, 0.8, 1)

[0116] bsdf.inputs['Roughness'].default_value = rough / 2e-6 # Convert to 0 - 1

[0117] bsdf.inputs['Specular'].default_value = 0.5

[0118] bsdf.inputs['IOR'].default_value = N_SnAgCu

[0119] out = tree.new(type='ShaderNodeOutputMaterial')

[0120] tree.links.new(bsdf.outputs['BSDF'], out.inputs['Surface'])

[0121] return mat

[0122] def add_ring_light(radius = 5e-3, energy = 50):

[0123] """Ring light, incident angle 0 - 80°, azimuth angle random"""

[0124] bpy.ops.object.light_add(type='SUN', location=(0,0,1))

[0125] sun = bpy.context.object

[0126] sun.data.energy = energy

[0127] sun.data.angle = np.radians(40) # Half angle

[0128] # Random direction

[0129] theta = np.radians(np.random.uniform(0, 80))

[0130] phi = np.radians(np.random.uniform(0, 360))

[0131] sun.location = Vector((radius*np.sin(theta)*np.cos(phi),

[0132] radius*np.sin(theta)*np.sin(phi),

[0133] radius*np.cos(theta)))

[0134] sun.rotation_mode = 'XYZ'

[0135] sun.rotation_euler = (theta, 0, phi)

[0136] def setup_camera():

[0137] cam_data = bpy.data.cameras.new('Camera')

[0138] cam_obj = bpy.data.objects.new('Camera', cam_data)

[0139] bpy.context.scene.collection.objects.link(cam_obj)

[0140] bpy.context.scene.camera = cam_obj

[0141] # 5x Telecentric parameters → Field of view 2x2 mm, pixel 250 nm

[0142] cam_data.lens = 50 # 50 mm lens

[0143] cam_data.sensor_width = 36 # Full frame

[0144] cam_obj.location = (0, 0, 50e-3) # Working distance above 0.3 mm

[0145] cam_obj.rotation_euler = (0,0,0)

[0146] return cam_obj

[0147] def render_to_file(filepath):

[0148] scene = bpy.context.scene

[0149] scene.render.engine = 'CYCLES'

[0150] scene.cycles.device = 'GPU'

[0151] scene.render.resolution_x = RESOLUTION

[0152] scene.render.resolution_y = RESOLUTION

[0153] scene.render.filepath = filepath

[0154] scene.render.image_settings.file_format = 'TIFF'

[0155] scene.render.image_settings.color_depth = '16'

[0156] scene.render.image_settings.tiff_codec = 'NONE'

[0157] bpy.ops.render.render(write_still=True)

[0158] def main():

[0159] os.makedirs(OUT_DIR, exist_ok=True)

[0160] clean_scene()

[0161] add_subdiv_plane()

[0162] cam = setup_camera()

[0163] for i in range(TOTAL_IMG):

[0164] sub_dir = os.path.join(OUT_DIR, f"batch{i / / BATCH:04d}")

[0165] os.makedirs(sub_dir, exist_ok=True)

[0166] fname = os.path.join(sub_dir, f"bump_{i:07d}.tif")

[0167] # Random roughness

[0168] rough = np.random.choice(ROUGH_LOG)

[0169] # Random Defects (Example of a Simple Ellipsoidal Depression)

[0170] bpy.ops.mesh.primitive_uv_sphere_add(radius=7.5e-6,

[0171] location=(np.random.uniform(-0.5,0.5)*1e-3,

[0172] np.random.uniform(-0.5,0.5)*1e-3,

[0173] np.random.uniform(5e-6, 10e-6)))

[0174] bump = bpy.context.object

[0175] bump.scale = (1, 1, np.random.uniform(0.8, 1.3)) # Ellipsoid degrees

[0176] bump.data.materials.append(sn_agcu_material("bump_mat",rough))

[0177] # Light source

[0178] add_ring_light()

[0179] # Rendering

[0180] render_to_file(fname)

[0181] # Cleanup

[0182] clean_scene()

[0183] add_subdiv_plane()

[0184] if i % 100 == 0:

[0185] print(f"[{i} / {TOTAL_IMG}] done → {fname}")

[0186] if __name__ == "__main__":

[0187] main()

[0188] Save the script as `bump_render.py`, then go to the "Scripting" tab in Blender 3.5 and select "RunScript" to generate 620k images in batches; alternatively, you can use `blender -b -P bump_render.py` to call it headless.

[0189] The simulation model should achieve the following results:

[0190] The GGX roughness of the micro-facet BRDF is adjustable from 0.01 to 2 μm; the complex refractive index of SnAgCu solder is n=1.80 and the extinction coefficient is κ=1.68 (405 nm).

[0191] The 405 nm ring light has an incident angle of 0-80° and an azimuth angle of 0-360° (random).

[0192] Camera NA 0.9, 3.45µm pixels, 5× magnification, 250nm / px;

[0193] Single frame size 2×2mm, 4096×4096px, single rendering time <0.8 s (RTX-4090 measured 0.62 s);

[0194] Directly outputs 16-bit TIFF, 1 µm / px (for ESRGAN 4× model super-resolution).

[0195] Step 2: Use the improved ESRGAN model to perform super-resolution processing on the defect simulation images; perform physical self-supervised augmentation on the super-resolution images, and construct a hybrid image with the production line images;

[0196] Improvements to the ESRGAN model include:

[0197] The generative network adopts a "two-branch-cross-scale residual" architecture with an overall depth of only twelve sub-modules. It achieves a receptive field comparable to the existing twenty-three RRDBs through "depth separability and coordinate attention".

[0198] The specific process is as follows:

[0199] First, the defect simulation image is input into a 3×3 convolution to upscale the input 1024×1024×3 low-resolution image to 64 channels, resulting in the feature map F0.

[0200] Secondly, F0 is fed into both the main branch and the bypass branch simultaneously; the main branch processes only one residual dense submodule at a time; inside the residual dense submodule, a 3×3 depthwise separable convolution is first used to operate on 64 channels one by one, with a computational cost of only 1 / 9 of that of a normal convolution, then a 1×1 point convolution is used to increase the number of channels to 96, and then a second 3×3 depthwise separable convolution is used to reduce it back to 64, forming a "compression-expansion-compression" bottleneck; the output of the residual dense submodule is multiplied by a scaling factor of 0.15 and added to itself to complete the local residual; the twelve residual dense submodules are connected in series, and a 1×1 convolution is inserted after every three submodules for feature recalibration to ensure gradient stability.

[0201] Secondly, the bypass branch is responsible for generating "coordinate attention". First, a 7×1 convolution is performed in the horizontal direction and a 1×7 convolution is performed in the vertical direction. Then, the two feature maps are added pixel by pixel. After ReLU activation, they are fed into a 1×1 convolution to compress to 16 channels. Then, a 1×1 convolution is used to expand them back to 64 channels. After passing through the Sigmoid function, a 64×1×1 weight vector is obtained. This vector is multiplied pixel by pixel with the backbone branch, so that the network automatically strengthens high-frequency defect edges such as bridging and depressions at the submicron scale, while suppressing background noise.

[0202] Secondly, the main branch and the bypass branch are added together to obtain a deep feature map F1 of 1024×1024×64;

[0203] The upsampling stage completely abandons PixelShuffle and instead uses a two-step process of "interpolation + convolution": the first step uses nearest-neighbor interpolation to enlarge the size to 2048×2048, followed by 48 3×3 convolution kernels (padding=1) for smoothing, outputting 2048×2048×48; the second step interpolates again to 4096×4096, and uses 24 3×3 convolution kernels for fusion, outputting 4096×4096×24; finally, it returns to the RGB space through 3×3×3 convolution to obtain a 4096×4096×3 super-resolution image; batch normalization is not used throughout the process, only LeakyReLU with a negative slope of 0.1 is retained to avoid grayscale jumps on the metal surface.

[0204] The discriminant network adopts a three-level structure of "large kernel pooling + deformability + multi-scale spectroscopy", including:

[0205] First, the input is a 6-channel stitched image (fake_B or real_B image first, followed by a low-resolution bicubic magnified image), with a size of 4096×4096. The first layer uses a 5×5 large kernel convolution with a stride of 2, outputting 128 channels, and the feature map size becomes 2048×2048. Then, a 2×2 max pooling is performed, halving the size to 1024×1024, while maintaining the number of channels at 128. The combination of large kernel and pooling can expand the receptive field at the very beginning, reducing the number of subsequent layers.

[0206] Secondly, the second layer uses 3×3 deformable convolution: first, a two-dimensional offset field is generated on 128 channels using 3×3 depth convolution, with an offset range of ±2 pixels; then, bilinear sampling is used to resample the input feature map to the new coordinates; finally, 3×3 convolution is performed to output 256 channels with a size of 512×512; deformable convolution enables the network to adapt to geometric distortions such as submicron bridging bends and hole edges, reducing the false alarm rate by 0.8% compared to rigid convolution.

[0207] Secondly, the third layer performs a "multi-scale spectral" operation: the 256-channel feature map is divided into four groups along the channel dimension, with 64 channels in each group; the first group uses a 1×1 convolution to maintain the original scale, the second group uses a 3×3 convolution to capture 0.5 µm details, the third group uses a 5×5 convolution to obtain 1 µm texture, and the fourth group uses a 7×7 convolution to obtain 2 µm contours; the outputs of the four groups are reassembled into 256 channels, and then fused by a 1×1 convolution to obtain a composite feature of 512×512×256.

[0208] Finally, the fourth layer performs parallel spatial pyramid pooling: the 256-channel feature map is divided into four grids: 1×1, 2×2, 4×4, and 8×8. Global average pooling is performed within each grid, and then the result is flattened into 256, 1024, 4096, and 16384-dimensional vectors. After concatenation, an 8192-dimensional global description is obtained. Subsequently, two fully connected layers (256→256) are used to compress the data layer by layer. Each layer is followed by parameterized ReLU (negative slope can be learned), and finally, a 1-dimensional confidence score is output. This structure abandons the traditional 8×8 discriminant matrix and replaces it with pure vector decision, which ensures sensitivity to multi-scale defects and avoids the problems of redundant parameters and slow training of VGG-128 used in existing ESRGAN models.

[0209] Physical self-supervised augmentation includes:

[0210] To determine physical consistency, the formula for the brightness difference during re-rendering is simplified to:

[0211] Error = |Re-render L′ - fake image_B| ÷ fake image_B < Error threshold 1%;

[0212] Sample screening was performed, and super-resolution images with errors <1% and discrimination confidence > confidence threshold 0.9 were retained, totaling 172,000 images, of which 43,000 were bridging images with a resolution of 0.5 μm.

[0213] The discriminant confidence score is trained separately using an independent physical consistency discriminant, which is only responsible for judging whether the "render-reconstruction pair" is real and credible, and is completely decoupled from the discriminant of the improved ESRGAN model;

[0214] The process is as follows:

[0215] 1. Improve the ESRGAN model to super-resolution the simulation image from 1 µm / pixel to 250 nm / pixel, resulting in fake_B;

[0216] 2. Send fake_B back to Blender and perform path tracing rendering again under the same lighting-camera parameters to obtain the re-rendered map L′;

[0217] 3. Input fake_B and L′ as a pair into the physical consistency discriminator. The physical consistency discriminator uses a lightweight 6-layer CNN and outputs 0-1 scalars.

[0218] 4. If the relative brightness error ‖L′−fake_B‖ / fake_B < 1%, and the discrimination confidence level > 0.9, then retain this fake_B image for subsequent training; otherwise, discard it.

[0219] Therefore, the confidence threshold of 0.9 is a probability value given by the physical consistency discriminator, which is used to ensure that the super-resolution image is consistent with the real optical imaging in terms of metal reflection, defect edges, etc., rather than to improve the GAN discrimination score of the ESRGAN model itself.

[0220] Finally, the data was mixed with the real SEM images at a 9:1 ratio (9 out of 170,000 images, 1 out of 620,000 images) to ensure that the simulation did not overwhelm the real features.

[0221] The submicron defect images and actual images of the bumps were obtained from the actual production line. 92 12nm chips were randomly selected from the production line, and 85 chips with 0.5 μm bridging, 57 chips with 0.3 μm voids, and 103 chips with ellipsoids >1.3 and abnormal shapes were collected.

[0222] Imaging conditions: 8-megapixel sCMOS + 5× telecentric lens, 405 nm ring light, single convex point 45×45 px;

[0223] Annotation process: Process engineer cross-annotation → Krippendorff α=0.92 → Arbitration of inconsistent samples → Final ground truth.

[0224] The production line image acquisition target is 3 s / chip; the hardware is a Basler 8K sCMOS + 5× telecentric lens, with 3.45μm pixels and an image resolution of 250 nm; the light source is a 405 nm LED ring + dome hybrid, with a brightness of 500 lux, and synchronously triggered InGaAs event sensor (DVS); the cycle time is a linear motor of 600 mm / s, an exposure of 30 μs, a single field of view of 2×2 mm, and a total of 18 frames stitched together; the image enters the FPGA buffer in <50 ms;

[0225] Quality gating of production line images: If edge brightness decreases by more than 5%, automatic flattening correction is invoked to ensure overall image non-uniformity is less than 3%;

[0226] Preprocessing of production line images includes: noise suppression, adaptive Gaussian filtering, and σ is automatically selected from 0.8 to 1.2 based on the gradient histogram.

[0227] Illumination uniformity: Real-time shooting of a standard white board, flat field coefficient K=I_center / I_edge, after multiplicative correction, the edge-center difference is reduced from 8% to 2%;

[0228] Contrast Enhancement: CLAHE, clip limit=3.0, tile=8×8, bump-background contrast increased by 1.7×;

[0229] Optional fixes: When local defocus is >2 μm, U-Net-GAN inference time is <20 ms and PSNR is >35 dB;

[0230] The output is a 16-bit TIFF file, 8192×8192 pixels, with a single file size of less than 200 MB, which is suitable for real-time streaming over gigabit networks.

[0231] Step 3: Input the mixed image into the meta-learning model and output the defect type; use a cascaded network to locate and segment the convex points of the defect image;

[0232] First, an accelerated life test was conducted: 300 chips from the same batch were tested at 125 °C / 1.2 V bias, and the failure time t was recorded. f (h) The initial area A0 (μm²) of the defect is measured using FIB.

[0233] Simplified lifespan prediction model:

[0234] ln(t f ) = A - 0.62 / (kT) - 0.28×√A0;

[0235] Where A is the fitting constant, dimensionless, given by the least squares method from the same batch of accelerated experimental data, with a value of approximately 22.4; k is the Boltzmann constant, 8.617 × 10⁻⁶. -5 eV·K -1 , used to convert temperature into energy; T is the absolute temperature of the accelerated life test, in K, and the example is fixed at 125°C, i.e. 398.15 K;

[0236] The simplified model directly maps the initial defect area A0 to the failure time t. f Enables online chip lifespan prediction; eliminates the need to wait 2-3 weeks for loopback, and outputs MTTF instantly upon detection;

[0237] The fitting R² = 0.94 can explain 94% of the lifetime fluctuations with an error of less than 7%, and the scrap rate of engineering lots is reduced by 40%.

[0238] Using a simplified lifetime prediction model, we construct labels for initial area to actual lifetime.

[0239] A meta-learning model is constructed using labels representing initial area → true lifetime, with a support set of 20 images containing A0 and t. f Given a query set of 10 images (A0 only), the meta-learning model is required to predict the t of the query set. f Values ​​enable rapid adaptation to small samples; the meta-learning model creates a total of 20,000 support set-query set tasks;

[0240] Using the same batch of chips, additional FIB slicing was performed, A0 was measured, and lifetime acceleration was performed at 125°C / 1.2 V to obtain t. f Each mixed image is supplemented with a lifetime regression label to form a dual-task label of "classification + regression" required for subsequent meta-learning.

[0241] The meta-learning model has a classification head of 7 classes (normal + 6 defects) and a regression head with a lifetime of 0-10000 h; MAML hyperparameters: in-line learning rate 0.005, out-line learning rate 0.0005, batch size 32, 60,000 iterations; early stopping patience = 5000; query set MTTF mean absolute error 7.2%, 0.5 μm bridging recall 98.9%.

[0242] The meta-learning model takes as input 224×224 RGB, 128-dimensional DVS pulse vector, and 3-dimensional process parameters. The input data has a one-to-one correspondence: after each chip is completed at the photolithography and electroplating stations, the SECS-GEM protocol writes the 3-dimensional process parameters of the current resist thickness, exposure amount, and development time into the MES along with the wafer ID. When the same chip enters the optical inspection station, the DVS event camera and the RGB camera will be exposed simultaneously under the same trigger signal. The DVS outputs a 128-dimensional pulse histogram, and the RGB outputs a 224×224 image. The system queries the wafer ID in seconds, stitches the chip-specific 128-dimensional DVS vector and 3-dimensional process parameters with the corresponding image to form a sample, and then sends it into the meta-learning model. Therefore, the two are twin data of the same chip, at the same time, and in the same spatial location, rather than cross-wafer or statistical mean.

[0243] Meta-learning models include:

[0244] The image stream, 224×224×3, is input into a ResNet50 (ImageNet pre-trained, tail removed), and then into a GlobalAvgPool and a 256-dimensional fully connected layer to obtain 256D visual features; img:2048→256 (ReLU + Dropout 0.3);

[0245] Inputting a 1×128 DVS stream into a 64-dimensional FC layer yields 64D pulse features; FC_dvs: 128→64 (ReLU +Dropout 0.2);

[0246] Inputting the 1×3 process flow into an 8D FC layer yields 8D process flow features; FC_proc: 3→8 (ReLU +Dropout 0.2);

[0247] The visual features, pulse features, and process flow features are concatted to obtain 328D stitched features.

[0248] The 328D spliced ​​features are input into a 512D FC layer to obtain 512D fused features (ReLU + Dropout 0.3);

[0249] The 512D fusion features are input into the classification head and regression head respectively. The classification head FC_cls: 512→7 (Softmax); the regression head FC_reg: 512→1 (ReLU, 0-10000 h truncation); the regression head outputs the lifetime value.

[0250] The cascaded network can use YOLOv5s coarse localization (2 ms) → Faster R-CNN fine localization (5 ms); overall inference 7ms / field of view; the training set of the cascaded network consists of 5000 real chip images + 170,000 simulation images, with enhancements including 20° rotation, ±15% brightness, and 0.5 px blur;

[0251] Output: Defect location bounding box of the defect image, top left corner (x,y), width w, height h, subpixel accuracy ±0.1 px; cropped by the border and bilinearly scaled to 224×224 px, with a 0.05 edge margin;

[0252] Regression head: lifetime hours, ReLU range 0-10000; Confidence gating: if the maximum probability is <0.7, it is marked as "suspicious" and automatically sent to the expert review library; the review results are fed back on the same night for incremental training (learning rate 1e-4, epoch 5); Inference speed: TensorRT FP16, 0.6 ms per frame; 18 fields of view in parallel, total time taken on GPU RTX-4090 is 90 ms;

[0253] Experimental results: 10,000 chips, 0.5 μm bridging recall 98.7%, false alarm 0.25%, MTTF prediction mean absolute error 7.0%.

[0254] Furthermore, the results output and visualization:

[0255] Integration: JSON is written with (x, y, w, h), category, MTTF, and confidence level, and bound to the chip ID (laser QR code).

[0256] Visualization: The original image is overlaid with a 2px colored box, red = bridging, cyan = hole, yellow = shape abnormality; the text label reads "Bridge 0.6 μm lifetime 830 h";

[0257] XML report: Automatically generated, including total number, defect number, percentage, location list, and process parameter snapshot; uploaded to MES via RESTful API, entire process <200 ms;

[0258] Archive: Original image + JSON + report are stored in distributed storage, retained for >5 years, and traceability is supported;

[0259] Furthermore, a closed-loop process for reliability improvement is implemented.

[0260] Triggering condition: The number of chips with MTTF < 1000 h in the same lot > 3%;

[0261] This invention sets internal control thresholds for achieving online lifespan-process closed-loop management. Based on the customer's requirement of "early failure rate ≤ 10 ppm," an accelerated lifespan model is used to back-calculate the corresponding MTTF (Mean Time To Failure) to approximately 920 hours. After reserving an 8% safety margin, the warning threshold is set at 1000 hours. Statistical analysis of six months of mass production data reveals that when the proportion of chips with MTTF below 1000 hours in the same lot exceeds 3%, the actual failure ppm will sharply rise to over 12 ppm within the subsequent 500 hours. Therefore, 3% is set as the "acceptable-unacceptable" inflection point. Compared to existing traditional reliability standards such as JEDEC, JESD47, and MIL-STD-883, these standards only specify the verification method for "zero failures at 125 °C for 1000 hours" and do not provide the online MTTF distribution ratio. The automotive AEC-Q100 also adopts the "0 / 77 failure" acceptance criterion, similarly not involving the percentage of lifespan prediction. Therefore, "MTTF < 1000 hours, number of chips > 3" is considered acceptable. The percentage (%) is merely a trigger threshold for the online closed loop of this invention, used to automatically correct exposure / development parameters within 30 seconds to ensure the customer's 10 ppm target. It is a custom internal control standard of the enterprise, rather than a general quality control or reliability assessment specification in the industry.

[0262] Correction calculation: By calling the inverse gradient of the meta-learning network, the exposure dose ΔE = -2.8% and the development time Δt = -5 s were obtained;

[0263] Issued: SECS-GEM protocol direct-write lithography machine, parameter refresh completed within 30 seconds;

[0264] Verification: Subsequently, 50 units were sampled immediately. The microbridging rate decreased from 0.12% to 0.02%, and the average MTTF increased from 920 h to 1580 h, achieving the reliability target of 10 ppm.

[0265] Validation on a continuous production line of 10,000 units:

[0266] Detected: 337 0.5 μm bridging particles, 4 missed detections, 98.7% recall rate; 253 0.3 μm voids, 3 missed detections, 98.8% recall rate.

[0267] False alarms: 25 in total, false alarm rate 0.25%;

[0268] Speed: 3.0 seconds for the entire video (including movement, shooting, inference, and reporting);

[0269] Power consumption: <800 W for the entire unit, 62% lower than laser confocal solutions;

[0270] Reliability: The field failure rate decreased from 18 ppm to 5 ppm after 6 months, and no complaints of misjudgment of life were received.

[0271] Experiments conducted according to the method of the present invention achieve the following results:

[0272] 1. Super-resolution-physical self-supervised augmentation increased the recall rate within a 3-second real-world production line cycle from 54% to 98.5% with a 0.5 μm bridging sample size of 100, and reduced the false alarm rate from 0.8% to 0.3%.

[0273] 2. Reliability - Meta-learning closed loop provides MTTF prediction during detection, discovering potential failures 500 hours in advance, reducing engineering lot scrap rate by 40%, and shortening customer verification cycle by half;

[0274] 3. The pulse-quantum sensing and computing integrated chip compresses the event stream and quantum signal into a 128-dimensional vector in one go, reducing power consumption by 62% and data processing latency from 10 ms to 0.9 ms, saving 12,000 kWh of electricity per unit per year;

[0275] 4. Federated meta-knowledge transfer allows each factory to transmit only encrypted gradients, enabling global model hot updates of 50 new defects in 30 minutes, saving 70% of maintenance manpower, and reducing downtime from 2 days to 4 hours.

[0276] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for online detection of submicron defects in chip bumps, characterized in that, Includes the following steps: Step 1: Generate a submicron simulation image of the convex point using a simulation model; Step 2: Perform super-resolution processing on the simulated images using the improved ESRGAN model; Physically self-supervised augmentation of super-resolution images; construction of hybrid images using augmented images and production line images; Step 3: Input the mixed image into the meta-learning model to output the defect type; use the cascaded network to output the defect location.

2. The online detection method for submicron defects in chip bumps according to claim 1, characterized in that, The construction of the simulation model includes: Create an output directory, clean the scene, and add a background plane and camera; Randomly select roughness level to generate random location and defect model; Add SnAgCu material properties to the defect model, add a ring light source; render the current scene as an image file; clean up the scene.

3. The online detection method for submicron defects in chip bumps according to claim 2, characterized in that, The improved ESRGAN model includes: a generator network and a discriminator network; The generator network includes: inputting an image into a 3×3 Conv and outputting a feature map F0; F0 simultaneously inputs the main branch and the bypass branch. The main branch is composed of several cascaded residual dense submodules. A 1×1 Conv is inserted after every few residual dense submodules. The bypass branch includes: converting F0 into a 7×1 Conv in the horizontal direction and a 1×7 Conv in the vertical direction; then adding the horizontal and vertical feature maps pixel by pixel, activating them with ReLU, and feeding them into a 1×1 Conv; then performing a 1×1 Conv, and obtaining the weight vector through Sigmoid; and multiplying the weight vector with the main branch pixel by pixel. Add the feature maps of the main branch and the bypass branch to output feature map F1; The upsampling stage sequentially goes through the first nearest neighbor interpolation, several 3×3Conv, the second nearest neighbor interpolation, several 3×3Conv, and 3×3×3Conv.

4. The online detection method for submicron defects in chip bumps according to claim 3, characterized in that, The residual dense submodule includes: inputting the input feature map sequentially into 3×3DSConv, 1×1PConv, and 3×3DSConv, multiplying it by a scaling factor, and then adding it to the input feature map.

5. The online detection method for submicron defects in chip bumps according to claim 3, characterized in that, Discriminant networks include: The first layer consists of a 5×5 Conv and a 2×2 Maxpool; the second layer consists of a 3×3 D Conv, bilinear sampling, and a 3×3 Conv; the third layer divides the channel feature map into four groups along the channel dimension: the first group is a 1×1 Conv, the second group is a 3×3 Conv, the third group is a 5×5 Conv, and the fourth group is a 7×7 Conv. The outputs of the four groups are concatenated and then passed through a 1×1 Conv; the fourth layer divides the channel feature map into a grid, using GAPool within the grid, and then flattens, concatenates, and performs two fully connected layers.

6. The online detection method for submicron defects in chip bumps according to claim 1, characterized in that, Physically supervised augmentation calculates the difference in brightness between re-rendered images when it is below the error threshold and the confidence level is above the confidence threshold.

7. The online detection method for submicron defects in chip bumps according to claim 1, characterized in that, Meta-learning models include: The image stream is sequentially input into ResNet50, GAPool and FC layers to obtain visual features; DVS stream input to FC layer to obtain pulse characteristics; The process flow is input into the FC layer to obtain the process flow characteristics; Visual features, pulse features, and process flow features are concatted to obtain stitched features; the stitched features are input into the FC layer and the output is the fused features. The fused features are input into the classification head and regression head respectively.

8. The online detection method for submicron defects in chip bumps according to claim 1, characterized in that, Cascaded networks include YOLOv5s and Faster R-CNN.

9. The online detection method for submicron defects in chip bumps according to claim 1, characterized in that, Image defects include: microbridges, top depressions, submicron voids, excessive ellipsoids, sidewall burrs, and insufficient bump height.

10. An online inspection system for submicron defects in chip bumps, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the online detection method for submicron defects in chip bumps as described in any one of claims 1-9.

Images