Semiconductor ultra-depth image fusion and defect detection method, system and medium

Abstract

The present invention discloses a semiconductor super-depth of field image fusion and defect detection method, system and medium. The image fusion method includes obtaining a multi-channel image set; the multi-channel image set includes multiple fly-by-flight image sets, each of which includes fly-by-flight images of a target to be detected at multiple height layers collected under the illumination of a corresponding light source, and each fly-by-flight image is accompanied by the height data of each pixel at the height layer; based on the Laplace pyramid fusion method, all fly-by-flight images in each fly-by-flight image set are fused to obtain a fused image corresponding to each fly-by-flight image set; any fly-by-flight image set is selected, all height data in the fly-by-flight image set is optimized to obtain a corresponding target height map; and a target fusion result is obtained based on the target height map and all fused images. The present invention can fuse images of different height layers into one image, and also display the height data in the form of a map, thereby achieving 3D restoration of multi-channel images.

Description

Technical Field

[0001] The present invention relates to the field of semiconductor processing technology, and in particular to a semiconductor ultra-depth-of-field image fusion and defect detection method, system and medium. Background Art

[0002] Surface defect detection technology is often used to measure semiconductor processing quality during the semiconductor manufacturing process, such as during wire bonding and die bonding. Currently, automated optical (visual) inspection (AOI) of surface defects based on optical image sensing (hereinafter referred to as automated optical inspection or AOI) has gradually become the primary method for semiconductor surface defect detection due to its advantages of automation, non-contact, high speed, high accuracy, and high stability.

[0003] To adapt to the varying conditions and defect types on the surfaces of semiconductor products, multi-channel lighting technologies are currently commonly used in AOI inspections, including RGBW ring light sources (a light source composed of red, green, blue, and white point light sources), darkfield lighting (referring to lighting in which diffuse light is reflected into the camera but mirror-reflected light is reflected away), and coaxial point light sources. RGBW ring light sources are used to synthesize high-resolution true-color images and highlight certain specific reflection angle features, introducing color data to the AOI, while darkfield lighting and coaxial point light sources are used to capture specific defect features. In summary, AOI inspections typically require images from multiple light sources for calculation.

[0004] With the increasing application of AOI inspection technology, semiconductor surface defect inspection requires increasingly higher visual resolution, particularly for semiconductors such as Miniled (LED chips with a size between 100 and 200μm, a size between fine-pitch LEDs and MicroLEDs) and MicroLEDs (LED chips with a size below 100μm). However, as camera and lens resolutions increase, the effective depth of field decreases. When the device under inspection has high and low layers or the device's base height is unstable, such as when inspecting wirebonding materials with numerous height levels of gold, aluminum, or dies, a zoom system is essential for obtaining a clear image. Currently, the most widely used zoom technologies include Z-axis zoom and liquid lens zoom. However, the presence of multiple height levels complicates visual inspection equipment programming and focus control, and consumes a large amount of memory. Furthermore, some areas may suffer from height data loss due to total internal reflection (e.g., moiré structured light method), or AOI inspection accuracy may be insufficient due to factors such as the laser beam's accuracy and specular reflection (e.g., triangulation laser measurement).

[0005] Therefore, it is very necessary to directly fuse images of different height layers into one image so that the image is clear in every area and attach the image height data, which is very necessary in the subsequent high-precision semiconductor surface defect detection and analysis. Summary of the Invention

[0006] In view of this, the present invention provides a semiconductor super-depth of field image fusion and defect detection method, system and medium to solve the problems in the existing technology of applying AOI to semiconductor defect detection (specifically defect detection in wire bonding process and chip bonding process), which is unable to fuse multi-channel super-depth of field images to obtain a clear image and does not include pixel height data in the image, resulting in complex operation, large memory usage and low detection accuracy.

[0007] The present invention provides a semiconductor super-depth-of-field image fusion method, comprising:

[0008] Acquire a total set of multi-channel images; the total set of multi-channel images includes multiple flight image sets, each flight image set includes flight images of the target to be measured at multiple altitude layers collected under the illumination of a corresponding light source, wherein each flight image is accompanied by altitude data of each pixel at the altitude layer;

[0009] Based on the Laplacian pyramid fusion method, all the flight-captured images in each flight-captured image set are fused to obtain a fused image corresponding to each flight-captured image set;

[0010] Selecting any one of the flight image sets, optimizing all height data in the selected flight image set, and obtaining a target height map corresponding to the selected flight image set;

[0011] A target fusion result is obtained according to the target height map and all the fused images.

[0012] Optionally, acquiring the total set of multi-channel images includes:

[0013] Obtaining an action sequence and an action trajectory corresponding to the light source camera used in a Z-axis flying photography scenario of the target to be measured; the action sequence includes a plurality of light source combinations that the light source camera needs to light up during the Z-axis flying photography and the lighting time of each light source in each light source combination; the action trajectory includes altitude data of the light source camera at multiple altitude layers during the Z-axis flying photography;

[0014] According to the action sequence and the action trajectory, the light source camera is controlled to perform Z-axis flying photography of the target to be measured, so as to obtain a plurality of flying photography images of the target to be measured at each height data and arranged according to the lighting time of each light source combination;

[0015] Clustering all the flight-by-flight images according to the light source types in the light source combination to obtain a plurality of flight-by-flight image sets;

[0016] The total set of multi-channel images is obtained based on all the flight image sets.

[0017] Optionally, the Laplacian pyramid fusion method is used to fuse all the fly-by-fly images in each of the fly-by-fly image sets to obtain a fused image corresponding to each of the fly-by-fly image sets, including:

[0018] Selecting the kth flying image set from the total multi-channel image set;

[0019] According to the preset pyramid layer N, constructing a Gaussian pyramid from the 1st layer to the N+1th layer for each of the kth flying images in the flying image set;

[0020] Selecting the jth flying image in the kth flying image set, and constructing a Laplacian pyramid of the jth flying image in the first to Nth layers according to each Gaussian pyramid of the jth flying image in the first to Nth layers;

[0021] The formula for constructing the Laplacian pyramid of the jth flying image in the kth flying image set at the i-th layer is:

[0022]

[0023] in, is the Laplacian pyramid of the jth flying image in the kth flying image set at the i-th layer, is the Gaussian pyramid of the jth flying image in the kth flying image set at the i-th layer, P yr Up(·) is the upsampling function, P yr Down(·) is a downsampling function; i is a positive integer and satisfies 1≤i≤N; j is a positive integer and satisfies 1≤j≤M, where M refers to the number of images in each of the flight-by-flight image sets; j is a positive integer and satisfies 1≤j≤M, where M refers to the number of images in each of the flight-by-flight image sets; k is a positive integer and satisfies 1≤k≤K, where K is the total number of light source types, and is also the total number of the flight-by-flight image sets in the total multi-channel image set;

[0024] Traversing each of the k-th flying images in the flying image set, and obtaining a Laplacian pyramid of each flying image from the 1st layer to the Nth layer according to the same method;

[0025] Select the Laplacian pyramid of all the flying images in the k-th flying image set at the i-th layer, obtain the maximum absolute value of all the selected Laplacian pyramids at each pixel point, and obtain the target pixel value of each pixel point of the k-th flying image set according to the maximum absolute value at each pixel point; and obtain the target pyramid of the k-th flying image set at the i-th layer according to the target pixel values ​​of all pixels of the k-th flying image set, which is recorded as

[0026] Traverse the Laplacian pyramid of all the flying images in the k-th flying image set at each layer, and follow the same method to obtain the target pyramid of the k-th flying image set at each layer, which is recorded as Wherein, respectively refer to the target pyramid of the kth flight image set at the 1st layer, the 2nd layer, ..., the Nth layer;

[0027] Performing mean calculation on the Gaussian pyramid of the kth flying image set at the N+1 layer to obtain a Gaussian pixel map of the kth flying image set at the N+1 layer;

[0028] According to the Gaussian pixel map of the kth in-flight image set at the N+1 layer and all target pyramids, a fused image corresponding to the kth in-flight image set is obtained and output;

[0029] Each of the flying image sets in the multi-channel image set is traversed, and the fused image corresponding to each of the flying image sets is obtained and output according to the same method.

[0030] Optionally, for the kth in-flight image set, obtaining and outputting a fused image corresponding to the kth in-flight image set based on the Gaussian pixel map of the kth in-flight image set at the N+1 layer and all target pyramids includes:

[0031] Perform Gaussian upsampling on the Gaussian pixel map of the kth in-flight image set at the N+1 layer to obtain a Gaussian sampling map at the N+1 layer; and sum the Gaussian sampling map of the kth in-flight image set at the N+1 layer with the target pyramid at the N layer to obtain the target pixel map of the kth in-flight image set at the N layer; according to the same method, perform Gaussian upsampling on the target pixel map of the kth in-flight image set at the i+1 layer layer by layer, and sum it with the target pyramid at the i layer of the kth in-flight image set until i reaches 1, and obtain the target pixel map of the kth in-flight image set at the 1st layer, which is used as the fused image corresponding to the kth in-flight image set and outputted;

[0032] The specific formula for performing Gaussian upsampling on the target pixel map of the kth in-flight image set at the i+1 layer and summing it with the target pyramid of the kth in-flight image set at the i layer is:

[0033]

[0034] Among them, TOP i k is the target pixel map of the kth flying image set at layer i, is the target pixel map of the kth flying image set at the i+1th layer.

[0035] Optionally, when selecting the kth in-flight image set from the total multi-channel image set, optimizing all height data in the selected in-flight image set to obtain a target height map corresponding to the selected in-flight image set includes:

[0036] Performing filtering and absolute value processing on each of the k-th flying images in the flying image set in sequence to obtain a source image corresponding to each flying image;

[0037] Select the source image corresponding to the j-th flying image, perform block sampling on the j-th source image according to a preset block sampling size, and obtain a sampled image corresponding to the j-th flying image;

[0038] According to the preset neighborhood size, a neighborhood summation operation is performed on each pixel of the j-th sampling image, and the result of the neighborhood summation of each pixel is filled into each corresponding pixel point to obtain the evaluation image corresponding to the j-th sampling image;

[0039] Assuming that the preset neighborhood size is 2f+1 pixels, and f is a positive integer, the pixel value corresponding to the pixel point (x, y) in the jth evaluation image is:

[0040]

[0041] in, is the pixel value corresponding to the pixel point (x, y) of the j-th evaluation image in the k-th flight image set, is the pixel value corresponding to the pixel point (p, q) in the neighborhood of the pixel point (x, y) of the j-th sampled image;

[0042] Traversing each of the k-th flying images in the flying image set, and obtaining an evaluation image corresponding to each of the k-th flying images in the flying image set according to the same method;

[0043] Extracting pixel values ​​of each pixel in all evaluation images of the kth in-flight image set to form an evaluation array corresponding to each pixel; extracting height data of each pixel in all in-flight images of the kth in-flight image set to form a height array corresponding to each pixel; the data in the evaluation array corresponds to the data in the height array in a one-to-one manner;

[0044] Extract the maximum pixel value in the evaluation array and the maximum height data in the corresponding height array for each pixel point, and determine whether the corresponding pixel point is an abnormal point based on each maximum pixel value;

[0045] When the pixel point (x, y) is a normal point, extract the two adjacent pixel values ​​adjacent to the maximum pixel value in the evaluation array corresponding to the pixel point (x, y), and extract the height data corresponding to the two adjacent pixel values ​​in the height array corresponding to the pixel point (x, y); calculate the actual height of the kth flying image set at the pixel point (x, y) based on the extracted two adjacent pixel values, the height data corresponding to the two adjacent pixel values, the maximum pixel value, and the maximum height data;

[0046] According to the same method, the actual height of each normal point of the k-th flight image set is obtained; and based on all normal points and the actual heights at all normal points, the target height map corresponding to the k-th flight image set is obtained.

[0047] Optionally, when the pixel point (x, y) is a normal point, the calculation formula for the actual height of the kth flying image set at the pixel point (x, y) is:

[0048]

[0049] in, is the actual height of the kth flying image set at the pixel point (x, y), T1 k,(x,y) and are the first intermediate parameter and the second intermediate parameter respectively, and They are the maximum pixel value of the pixel point (x, y) in the corresponding evaluation array and the maximum height data in the corresponding height array, and They are the two adjacent pixel values ​​adjacent to the maximum pixel value in the evaluation array corresponding to the pixel point (x, y), and They are the height data corresponding to the two adjacent pixel values ​​in the height array corresponding to the pixel point (x, y).

[0050] Optionally, judging whether a corresponding pixel point is an abnormal point according to each maximum pixel value includes:

[0051] For the pixel point (x, y), calculate the pixel error between each pixel value in the evaluation array corresponding to the pixel point (x, y) and the maximum pixel value;

[0052] When all pixel errors meet the first outlier criterion, the pixel (x, y) is determined to be an outlier; otherwise, the pixel (x, y) is determined to be a normal point.

[0053] The first abnormal point criterion is:

[0054]

[0055] Among them, V l k,(x,y) is the lth pixel value in the evaluation array of the kth flight image set at the pixel point (x, y), is the error between the lth pixel value and the maximum pixel value in the evaluation array at the pixel point (x, y), u is the preset pixel error threshold, and abs(·) is the absolute value function;

[0056] Traverse each pixel and judge each pixel in the same way.

[0057] Optionally, after obtaining the target height map corresponding to the kth flight image set, the method further includes:

[0058] For a pixel point (x, y) in the target height map corresponding to the kth in-flight image set, a preset search size and a preset height error threshold are obtained, and outlier points in the target height map are again screened out according to the preset search size and the preset height error threshold to obtain an outlier point set in the target height map of the kth in-flight image set;

[0059] Filling all outliers in the outlier point set of the target height map of the kth in-flight image set with height data, overwriting the target height map corresponding to the kth in-flight image set based on the filled outlier point set and the corresponding height data, and performing Gaussian filtering on the overwritten target height map to obtain an optimized height map corresponding to the kth in-flight image set;

[0060] Obtaining a target fusion result according to the target height map and all the fused images includes:

[0061] The optimized height map and all the fused images are output to obtain the target fusion result.

[0062] Optionally, the step of further performing outlier screening on the target height map according to the preset search size and the preset height error threshold to obtain an outlier point set of the target height map of the kth flight image set includes:

[0063] Select the height data of any pixel point (x, y) in the target height map, set the preset search size to v, and search the height data of four pixel points (xv, y), (x+v, y), (x, yv), and (x, y+v) in the target height map respectively;

[0064] Determine whether the height errors between the height data of the four pixel points (xv, y), (x+v, y), (x, yv) and (x, y+v) and the height data of the pixel point (x, y) are all greater than the preset height error threshold. If so, determine that the pixel point (x, y) is an abnormal point; otherwise, determine that the pixel point (x, y) is a normal point;

[0065] Traversing the height data of each pixel point (x, y) in the target height map, judging each pixel point (x, y) according to the same method, and obtaining all abnormal points in the target height map after the abnormal points are screened out;

[0066] According to all the obtained abnormal points, a set of abnormal points corresponding to the target height map of the k-th flight image set is obtained.

[0067] Optionally, filling height data for all outliers in an outlier point set of the target height map of the kth flight image set includes:

[0068] Select any non-edge outlier point (a1, b1) in the outlier point set of the k-th flight image set, and search for height data corresponding to four normal points adjacent to the non-edge outlier point (a1, b1) in the target height map in four search directions of up, down, left, and right, with the selected non-edge outlier point (a1, b1) as the center; calculate the mean of the height data of the four normal points found, and fill the height data obtained by the mean calculation into the height data of the non-edge outlier point (a1, b1) in the target height map to complete the filling of the height data of the non-edge outlier point (a1, b1);

[0069] Traversing each non-edge outlier point in the outlier point set of the k-th flight image set, and completing the height data filling of each non-edge outlier point in the same way;

[0070] Select any edge outlier point (a2, b2) in the outlier point set of the k-th flight image set, and use the selected edge outlier point (a2, b2) as a reference to search for height data corresponding to a normal point that is closest to the edge outlier point (a2, b2) in the target height map; fill the height data corresponding to the searched normal point into the height data of the edge outlier point (a2, b2) in the target height map to complete the height data filling of the edge outlier point (a2, b2);

[0071] Traverse each edge outlier point in the outlier point set of the k-th flight image set, and complete the height data filling of each edge outlier point in the same way.

[0072] In addition, the present invention also provides a semiconductor defect detection method, comprising:

[0073] According to the aforementioned semiconductor super-depth of field image fusion method, a total set of multi-channel images in the semiconductor processing process is obtained, and image fusion is performed on the total set of multi-channel images to obtain a target fusion result;

[0074] Image analysis is performed based on the target fusion results to obtain semiconductor defect detection results.

[0075] In addition, the present invention also provides a semiconductor super-depth-of-field image fusion system, which is applied to the aforementioned semiconductor super-depth-of-field image fusion method, comprising:

[0076] An image acquisition module is configured to acquire a multi-channel image collection; the multi-channel image collection includes a plurality of flight image collections, each of which includes flight images of the target under test at multiple altitude levels captured under illumination from a corresponding light source, wherein each flight image is accompanied by altitude data for each pixel at the altitude level at which it is located;

[0077] An image fusion module is configured to fuse all the fly-by-flight images in each of the fly-by-flight image sets based on a Laplace pyramid fusion method to obtain a fused image corresponding to each of the fly-by-flight image sets;

[0078] a height optimization module, which selects one of the flight image sets and optimizes all height data in the selected flight image set to obtain a target height map corresponding to the selected flight image set;

[0079] An output module is used to obtain a target fusion result based on the target height map and all the fused images.

[0080] In addition, the present invention also provides a semiconductor defect detection system, comprising:

[0081] The aforementioned semiconductor super-depth-of-field image fusion system is used to obtain a total set of multi-channel images during semiconductor processing, and perform image fusion on the total set of multi-channel images to obtain a target fusion result; and

[0082] The image analysis module is used to perform image analysis based on the target fusion results to obtain semiconductor defect detection results.

[0083] In addition, the present invention also provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the aforementioned semiconductor super-depth of field image fusion method when the instruction is executed.

[0084] The beneficial effects of the present invention are as follows: by obtaining a total set of multi-channel images that include both fly-by images under different light sources and height data at different height layers, it is possible to adapt to different conditions and different defect types on the surface of semiconductor products based on multi-channel lighting technology, better reflect the characteristics of the surface of semiconductor products, and improve the accuracy of subsequent defect detection; based on the Laplace pyramid fusion method, it is possible to fuse pixel information of different scales to be compatible with the current multi-channel light source detection mode of AOI detection; then, by optimizing the height data in any selected fly-by image set, the height map formed can directly fuse images of different height layers into one image, so that the image is clear in each area, and at the same time, the original accompanying height data is displayed in the form of an image, thereby realizing 3D restoration of the multi-channel image, which can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis;

[0085] The semiconductor super-depth of field image fusion and defect detection method, system and medium in the present invention realize the fusion of multi-channel super-depth of field images, and can directly fuse images of different height layers into one image, so that the image is clear in each area, and at the same time, the original accompanying height data is displayed in the form of an image, realizing 3D restoration of multi-channel images, which can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis. The entire method is relatively simple to operate and is compatible with the current AOI detection of wire bonding and chip bonding multi-channel light source detection mode without the need for equipment programming and will not take up too much memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0087] Figure 1 A flow chart of a semiconductor super-depth of field image fusion method in embodiment 1 of the present invention is shown;

[0088] Figure 2 A flowchart of obtaining a multi-channel image set in the first embodiment of the present invention is shown;

[0089] Figure 3 shows a complete structural diagram of the image acquisition device used in the first embodiment of the present invention;

[0090] Figure 4 A flow chart of a semiconductor defect detection method in a second embodiment of the present invention is shown;

[0091] Figure 5 The structure diagram of a semiconductor super-depth-of-field image fusion system in the third embodiment of the present invention is shown;

[0092] Figure 6 The figure shows a structural diagram of a semiconductor defect detection system in a fourth embodiment of the present invention.

[0093] Description of reference numerals:

[0094] 1. MCB controller, 2. Servo drive, 3. Industrial camera, 4. Telecentric lens, 5. Hybrid lighting source, 6. Screw, 7. Servo motor, 8. Camera backplane. DETAILED DESCRIPTION

[0095] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0096] Example 1

[0097] like Figure 1 As shown, a semiconductor super-depth of field image fusion method includes:

[0098] SA1: Acquire a total set of multi-channel images; the total set of multi-channel images includes multiple flight image sets, each of which includes flight images of the target under test at multiple altitude levels captured under the illumination of a corresponding light source, wherein each flight image is accompanied by altitude data of each pixel at the altitude level;

[0099] SA2: Based on the Laplacian pyramid fusion method, all the flight-captured images in each flight-captured image set are fused to obtain a fused image corresponding to each flight-captured image set;

[0100] SA3: Select any one of the flight image sets, optimize all height data in the selected flight image set, and obtain a target height map corresponding to the selected flight image set;

[0101] SA4: Obtain a target fusion result based on the target height map and all the fused images.

[0102] By acquiring a multi-channel image collection that includes both fly-by images illuminated by different light sources and height data at different height layers, it is possible, based on multi-channel lighting technology, to adapt to different surface conditions and defect types of semiconductor products, better reflect the characteristics of the semiconductor product surface, and improve the accuracy of subsequent defect detection. Based on the Laplace pyramid fusion method, pixel information at different scales can be fused to be compatible with the current multi-channel light source detection mode of AOI inspection. Then, by optimizing the height data in any selected fly-by image set, the height map formed can directly fuse images at different height layers into a single image, making the image clear in every area. At the same time, the original accompanying height data can be displayed in the form of an image, realizing 3D restoration of the multi-channel image, which can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis.

[0103] The semiconductor super-depth-of-field image fusion method in this embodiment realizes the fusion of multi-channel super-depth-of-field images, and can directly fuse images of different height layers into one image, so that the image is clear in each area, and at the same time, the original accompanying height data is displayed in the form of an image, thereby realizing 3D restoration of multi-channel images, and can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis. The entire method is relatively simple to operate and is compatible with the current multi-channel light source detection mode of AOI detection of wire bonding and chip bonding, without the need for equipment programming and will not occupy too much memory.

[0104] Preferably, if Figure 2 As shown, SA1 includes:

[0105] SA11: Obtaining an action sequence and action trajectory corresponding to the light source camera used in a Z-axis flying photography scenario for the target to be measured; the action sequence includes multiple light source combinations that need to be illuminated by the light source camera during the Z-axis flying photography, and the lighting time of each light source in each light source combination; the action trajectory includes altitude data of the light source camera at multiple altitude levels during the Z-axis flying photography;

[0106] SA12: Controlling the light source camera to perform Z-axis flying photography of the target to be measured according to the action sequence and the action trajectory, to obtain a plurality of flying images of the target to be measured at each height data and arranged according to the lighting time of each light source combination;

[0107] SA13: Clustering all the flight-by-flight images according to the light source types in the light source combination to obtain a plurality of flight-by-flight image sets;

[0108] SA14: Obtain the total set of multi-channel images based on all the flight image sets.

[0109] The above method is used to obtain multi-channel images at different height levels, so that based on multi-channel lighting technology, it can adapt to different conditions and different defect types on the surface of semiconductor products, better reflect the characteristics of the surface of semiconductor products, and improve the accuracy of subsequent defect detection.

[0110] Specifically, if Figure 3 As shown, the image acquisition device used in this embodiment SA1 includes JUTZE's self-developed MCB (Motion Controll Board) controller 1, a servo driver 2, an imported large-target-area, high-frame industrial camera 3, a telecentric lens 4, JUTZE's self-developed RGBW+dark-field+point-light mixed lighting source 5, a high-precision ball screw 6, and a servo motor 7. Among them, the industrial camera 3, the telecentric lens 4 (located on the side of the industrial camera 3 facing the target to be measured), and the mixed lighting source 5 are arranged as a whole on one side of the camera back plate 8, and the screw 6 and the servo motor 7 are arranged as a whole on the other side of the camera back plate 8. The MCB controller 1 is communicatively connected to the industrial camera 3 and the servo driver 2, and the servo driver 2 is communicatively connected to the servo motor 7. The MCB controller 1 sends control instructions to the servo driver 2 to drive the servo motor 7, which in turn drives the lead screw 6, the industrial camera 3, the telecentric lens 4, and the hybrid lighting source 5 to move up and down along the Z axis. The MCB controller 1 also sends control instructions to control operations such as the camera triggering and multi-channel light source triggering of the industrial camera 3, thereby achieving Z-axis flying photography of the target to be measured on the surface.

[0111] Specifically, a light source combination refers to the combination of light sources within the mixed lighting source that is triggered during Z-axis flight shooting. This includes a free combination of RGBW light sources, dark field light sources, and point light sources. The lighting time within a light source combination refers to the time at which each light source in any light source combination is activated, as well as the duration of its lighting. These light source combinations and lighting times, in chronological order, form an action sequence; this action sequence enables the capture of more comprehensive images containing multi-channel light source information.

[0112] Specifically, since this embodiment not only captures images containing multi-channel light source information but also captures images at different height levels, it is also necessary to obtain information about each height level during Z-axis flight. The motion trajectory includes the altitude data for each altitude level during Z-axis flight, for each action sequence. Furthermore, the motion trajectory in this embodiment also includes the starting and ending positions of the Z-axis flight, as well as the interval between the actions. This information allows for better control of Z-axis flight at each altitude level.

[0113] Specifically, in SA12, through the action sequence and action trajectory in SA11, multiple images arranged according to the action sequence can be obtained at each altitude layer; in SA13, according to the type of light source, images of the same light source are put into a set, and the height data of each image is attached, so as to realize image clustering according to the type of light source, that is, multiple flight image sets; the number of images in each flight image set is the same (the same as the number of altitude layers), and the total set of these flight image sets is the multi-channel image set.

[0114] Preferably, SA2 includes:

[0115] SA21: Select the kth flying image set from the total multi-channel image set;

[0116] SA22: Constructing a Gaussian pyramid from the 1st layer to the N+1th layer for each of the kth flying images in the flying image set according to the preset pyramid layer N;

[0117] SA23: Select the jth flying image in the kth flying image set, and construct a Laplacian pyramid of the jth flying image in the 1st to Nth layers according to each Gaussian pyramid of the jth flying image in the 1st to Nth layers;

[0118] The formula for constructing the Laplacian pyramid of the jth flying image in the kth flying image set at the i-th layer is:

[0119]

[0120] in, is the Laplacian pyramid of the jth flying image in the kth flying image set at the i-th layer, is the Gaussian pyramid of the jth flying image in the kth flying image set at the i-th layer, P yr Up(·) is the upsampling function, P yr Down(·) is a downsampling function; i is a positive integer and satisfies 1≤i≤N; j is a positive integer and satisfies 1≤j≤M, where M refers to the number of images in each of the flight-by-flight image sets; j is a positive integer and satisfies 1≤j≤M, where M refers to the number of images in each of the flight-by-flight image sets; k is a positive integer and satisfies 1≤k≤K, where K is the total number of light source types, and is also the total number of the flight-by-flight image sets in the total multi-channel image set;

[0121] SA24: traverse each of the k-th flying image sets, and obtain the Laplacian pyramid of each flying image from the 1st layer to the Nth layer according to the same method;

[0122] SA25: Select the Laplacian pyramid of all the flying images in the k-th flying image set at the i-th layer, obtain the maximum absolute value of all the selected Laplacian pyramids at each pixel point, and obtain the target pixel value of each pixel point of the k-th flying image set according to the maximum absolute value at each pixel point; and obtain the target pyramid of the k-th flying image set at the i-th layer according to the target pixel values ​​of all pixels of the k-th flying image set, which is recorded as

[0123] SA26: Traverse the Laplacian pyramid of all the flying images in the k-th flying image set at each layer, and obtain the target pyramid of the k-th flying image set at each layer in the same way, denoted as Wherein, respectively refer to the target pyramid of the kth flight image set at the 1st layer, the 2nd layer, ..., the Nth layer;

[0124] SA27: Calculate the mean of the Gaussian pyramid of all the flying images in the k-th flying image set at the N+1 layer to obtain a Gaussian pixel map of the k-th flying image set at the N+1 layer;

[0125] SA28: Obtain and output a fused image corresponding to the kth in-flight image set based on the Gaussian pixel map of the kth in-flight image set at the N+1 layer and all target pyramids;

[0126] SA29: Traverse each of the flying image sets in the total multi-channel image set, and obtain the fused image corresponding to each of the flying image sets in the same way and output it.

[0127] The pyramid of an image is a set of images arranged in a pyramid shape with gradually decreasing resolution and originating from the same original image. This embodiment implements image fusion based on the Laplacian pyramid fusion method. This method can reconstruct unsampled images from lower-level images in the pyramid, and achieve maximum image restoration through image segmentation. Furthermore, pixel information at different scales in each fly-by image set is fused to achieve multi-channel image super-depth-of-field fusion, thereby achieving compatibility with multi-channel light source detection modes for AOI detection, making it suitable for different application environments.

[0128] Specifically, in SA22, the j-th flying image is obtained in each Gaussian pyramid from the 1st layer to the Nth layer. i is a positive integer and satisfies 1≤i≤N. The Gaussian pyramid of the N+1th layer is recorded separately as The specific method of constructing the Gaussian pyramid is an existing technology, and the specific details are not repeated here.

[0129] In SA24, the Laplacian pyramid of each flying image in the k-th flying image set from the 1st layer to the Nth layer is obtained. i is a positive integer and satisfies 1≤i≤N, j is a positive integer and satisfies 1≤j≤M (M refers to the number of images in each fly-by-flight image set); in SA25, the j-th fly-by-flight image in the k-th fly-by-flight image set is in the Laplacian pyramid of the i-th layer The absolute value at the pixel point (x, y) is Pick The maximum absolute value (i.e., the maximum absolute value) at the pixel point (x, y) in the flying image corresponding to the maximum absolute value is taken as the corresponding target pixel value; the same is true for other pixel points. After obtaining the target pixel values ​​at all pixel points, the target pyramid of the k-th flying image set at the i-th layer is obtained, which is recorded as In SA26, the same method is used to obtain the target pyramid of each layer of the k-th flight image set, which is recorded as In SA27, before calculating the mean of all the flying images in the k-th flying image set at the N+1-th layer of the Gaussian pyramid, we first obtain the Gaussian pyramid of each flying image at the N+1-th layer, which is denoted as Then, the mean of all values ​​in the set is calculated (i.e., sum and average first), and the Gaussian pixel map of the k-th flight image set at the N+1 layer is obtained, which is recorded as

[0130] Preferably, SA28 includes:

[0131] Perform Gaussian upsampling on the Gaussian pixel map of the kth in-flight image set at the N+1 layer to obtain a Gaussian sampling map at the N+1 layer; and sum the Gaussian sampling map of the kth in-flight image set at the N+1 layer with the target pyramid at the N layer to obtain the target pixel map of the kth in-flight image set at the N layer; according to the same method, perform Gaussian upsampling on the target pixel map of the kth in-flight image set at the i+1 layer layer by layer, and sum it with the target pyramid at the i layer of the kth in-flight image set until i reaches 1, and obtain the target pixel map of the kth in-flight image set at the 1st layer, which is used as the fused image corresponding to the kth in-flight image set and outputted;

[0132] The specific formula for performing Gaussian upsampling on the target pixel map of the kth in-flight image set at the i+1 layer and summing it with the target pyramid of the kth in-flight image set at the i layer is:

[0133]

[0134] Among them, TOP i kis the target pixel map of the kth flying image set at layer i, is the target pixel map of the kth flying image set at the i+1th layer.

[0135] By performing Gaussian upsampling on the target pixel map of the previous layer layer by layer and summing it with the target pyramid of the current layer, it is possible to achieve gradual restoration from the top layer to the bottom layer with the maximum degree of restoration.

[0136] For i=N, the specific formula for Gaussian upsampling of the target pixel map of the k-th fly-by-flight image set at the i+1 layer and summing it with the target pyramid of the k-th fly-by-flight image set at the i-th layer is: That is, the Gaussian sampling image obtained by Gaussian upsampling the Gaussian pixel map at the N+1 layer of the k-th in-flight image set is summed with the target pyramid at the N layer. When i=N-1, the specific formula for Gaussian upsampling the target pixel map at the N layer of the k-th in-flight image set and summing it with the target pyramid at the N-1 layer of the k-th in-flight image set is: That is, the target sampling map obtained by Gaussian upsampling the target pixel map of the k-th in-flight image set at the Nth layer is summed with the target pyramid at the N-1th layer; ... and so on. When i = 2, the specific formula for Gaussian upsampling the target pixel map of the k-th in-flight image set at the 3rd layer and summing it with the target pyramid at the 2nd layer of the k-th in-flight image set is: That is, the target sampling map obtained by Gaussian upsampling the target pixel map of the k-th flying image set on the third layer is summed with the target pyramid on the second layer. When i = 1, the specific formula for Gaussian upsampling the target pixel map of the k-th flying image set on the second layer and summing it with the target pyramid on the first layer of the k-th flying image set is: That is, the target sampling map obtained by Gaussian upsampling the target pixel map of the second layer of the k-th flying image set is summed with the target pyramid of the first layer. The above process is the complete process of Gaussian upsampling the target pixel map of the previous layer layer by layer and summing it with the target pyramid of the current layer, and finally the TOP1 is obtained and output. k That is, the fused image corresponding to the kth flying image set in S28; in S29, the fused image corresponding to each flying image set is obtained according to the same method as S21 to S28. k}, k is a positive integer and satisfies 1≤k≤K (K is the total number of light source types).

[0137] Preferably, when the kth flying image set in the total multi-channel image set is selected, SA3 includes:

[0138] SA31: performing filtering processing and absolute value processing on each of the k-th flying image set in sequence to obtain a source image corresponding to each flying image;

[0139] SA32: Select the source image corresponding to the j-th flying image, perform block sampling on the j-th source image according to a preset block sampling size, and obtain a sampled image corresponding to the j-th flying image;

[0140] SA33: Perform a neighborhood summation operation on each pixel of the j-th sample image according to the preset neighborhood size, and fill the result of the neighborhood summation of each pixel into each corresponding pixel point to obtain the evaluation image corresponding to the j-th sample image;

[0141] Assuming that the preset neighborhood size is 2f+1 pixels, and f is a positive integer, the pixel value corresponding to the pixel point (x, y) in the jth evaluation image is:

[0142]

[0143] in, is the pixel value corresponding to the pixel point (x, y) of the j-th evaluation image in the k-th flight image set, is the pixel value corresponding to the pixel point (p, q) in the neighborhood of the pixel point (x, y) of the j-th sampled image;

[0144] SA34: traverse each of the k-th flying image sets, and obtain an evaluation image corresponding to each of the k-th flying image sets using the same method;

[0145] SA35: extracting the pixel value of each pixel in all evaluation images of the kth in-flight image set to form an evaluation array corresponding to each pixel; and extracting the height data of each pixel in all evaluation images of the kth in-flight image set to form a height array corresponding to each pixel; the data in the evaluation array corresponds to the data in the height array in a one-to-one manner;

[0146] SA36: Extract the maximum pixel value in the evaluation array and the maximum height data in the corresponding height array for each pixel point, and determine whether the corresponding pixel point is an abnormal point based on each maximum pixel value;

[0147] SA37: When the pixel point (x, y) is a normal point, extract the two adjacent pixel values ​​adjacent to the maximum pixel value in the evaluation array corresponding to the pixel point (x, y), and extract the height data corresponding to the two adjacent pixel values ​​in the height array corresponding to the pixel point (x, y); calculate the actual height of the kth in-flight image set at the pixel point (x, y) based on the extracted two adjacent pixel values, the height data corresponding to the two adjacent pixel values, the maximum pixel value, and the maximum height data;

[0148] SA38: According to the same method, the actual height of each normal point of the k-th in-flight image set is obtained; and based on all normal points and the actual heights at all normal points, the target height map corresponding to the k-th in-flight image set is obtained.

[0149] Each flight image in the original multi-channel data set carries height data. Through the height data optimization process described in SA31 to SA38 above, not only can images at different height layers be directly fused into one image so that the image is clear in every area, but the original accompanying height data can also be displayed in the form of an image, realizing 3D recovery of the height data and more comprehensively displaying the characteristics of the target at different height layers, thereby greatly improving the accuracy of subsequent defect detection.

[0150] Specifically, before SA31, the kth set of the flight image is selected for subsequent height data optimization. The selection can be based on actual conditions or can be manually set before the actual operation. In SA31, the image quality can be improved by filtering and taking the absolute value, which is convenient for the subsequent optimization of the height data. The 3D restoration of the height data is achieved in the form of a graph, and the source image corresponding to each flight image is recorded as 1≤j≤M (M refers to the number of images in each fly-by image set); in SA32, let the preset block sampling size BlockSize=t, then for the source image corresponding to the jth fly-by image, , starting from the pixel point (0,0), sampling is performed every (t,t), that is, each (t,t) pixel point shares one evaluation data, which can reduce the amount of calculation; when all pixels are block sampled according to this method, the sampled image corresponding to the j-th flying image is obtained, which is recorded as The neighborhood summation operation is used in SA33 to further reduce the complexity of the calculation, improve the processing speed, and save the space occupied during the calculation. In SA34, according to the method of SA33, the evaluation image corresponding to each flying image in the k-th flying image set is obtained, which is recorded as In SA35, by constructing the evaluation array and height array, it is convenient to use the maximum pixel value of each pixel in the evaluation array as the evaluation standard in SA36 to screen out abnormal points, and then screen out pixels with unclear height information (close to the pure color area); after the abnormal points are screened out by SA36, the remaining normal points are all considered to have valid height data. At this time, in SA37, for any normal pixel (x, y), its corresponding evaluation array is V[M] k , by extracting the two adjacent pixel values ​​adjacent to its maximum pixel value, and in the height array H[M] k The height data corresponding to these two adjacent pixel values ​​can be used to calculate the actual height of the pixel point (x, y) In SA38, the same principle applies to other normal points. The actual height of each pixel can be obtained by the same method. Finally, these normal points and the corresponding actual heights can be output as a height data in the form of a graph (i.e., target height graph).

[0151] Preferably, in SA37, for any pixel point (x, y) that is a normal point, the actual height of the k-th flying image set at the pixel point (x, y) is calculated by extracting two adjacent pixel values ​​adjacent to the maximum pixel value and the height data corresponding to these two adjacent pixel values ​​in the height array:

[0152]

[0153] in, is the actual height of the kth flying image set at the pixel point (x, y), T1 k,(x,y) and are the first intermediate parameter and the second intermediate parameter respectively, and They are the maximum pixel value of the pixel point (x, y) in the corresponding evaluation array and the maximum height data in the corresponding height array, and They are the two adjacent pixel values ​​adjacent to the maximum pixel value in the evaluation array corresponding to the pixel point (x, y), and They are the height data corresponding to the two adjacent pixel values ​​in the height array corresponding to the pixel point (x, y).

[0154] The above calculation formula uses parabolic approximation to calculate the actual height of the pixel point (x, y), which can effectively reduce the error of the height data and thus improve the accuracy of the target height map.

[0155] Preferably, in SA36, judging whether the corresponding pixel point is an abnormal point according to each maximum pixel value includes:

[0156] SA361: For a pixel point (x,y), calculate the pixel error between each pixel value in the evaluation array corresponding to the pixel point (x,y) and the maximum pixel value;

[0157] SA362: When all pixel errors meet the first outlier criterion, the pixel (x, y) is determined to be an outlier; otherwise, the pixel (x, y) is determined to be a normal point;

[0158] The first abnormal point criterion is:

[0159]

[0160] Among them, V l k,(x,y) is the lth pixel value in the evaluation array of the kth flight image set at the pixel point (x, y), is the error between the lth pixel value and the maximum pixel value in the evaluation array at the pixel point (x, y), u is the preset pixel error threshold, and abs(·) is the absolute value function;

[0161] SA363: Traverse each pixel and judge each pixel using the same method.

[0162] The above-mentioned outlier screening process compares the pixel error between the pixel value of each pixel and the maximum pixel value in the evaluation array with the preset pixel error threshold, and uses a simple and direct judgment method to screen out outliers of the pixels, thereby screening out pixels whose height information is not prominent (close to the pure color area).

[0163] Preferably, after SA38, it also includes:

[0164] SA39: For a pixel point (x, y) in the target height map corresponding to the kth in-flight image set, obtain a preset search size and a preset height error threshold, and perform outlier removal on the target height map again according to the preset search size and the preset height error threshold to obtain an outlier point set in the target height map of the kth in-flight image set;

[0165] SA310: Filling height data for all outliers in the outlier point set of the target height map of the kth in-flight image set, and covering the target height map corresponding to the kth in-flight image set based on the filled outlier point set and the corresponding height data, and performing Gaussian filtering on the covered target height map to obtain an optimized height map corresponding to the kth in-flight image set.

[0166] By further filtering out abnormal points in the target height map in the above-mentioned SA39, abnormalities caused by underexposure or overexposure of some pixels during image acquisition can be filtered out, further improving the data validity and accuracy of the target height map. By filling and covering the height data in SA310, abnormal points can be repaired based on valid height data, thereby achieving the repair of the entire target height map, ensuring data continuity in the final optimized height map, and enabling multi-channel image fusion to achieve the optimal 3D restoration effect.

[0167] SA4 includes:

[0168] The optimized height map and all the fused images are output to obtain the target fusion result.

[0169] Taking the optimized height map of any flight-captured image set and all the fused images after the fusion of all flight-captured image sets as the final target fusion result, it is possible to achieve the fusion of super-depth-of-field images of different channels, and to directly fuse images of different height layers into one image, so that the image is clear in every area. At the same time, the original accompanying height data can be displayed in the form of an image, realizing 3D restoration of multi-channel images, which is compatible with the current multi-channel light source detection mode of AOI detection.

[0170] Preferably, in SA39, outlier screening is performed again on the target height map according to the preset search size and the preset height error threshold to obtain an outlier point set of the target height map of the kth flight image set, including:

[0171] SA391: Select the height data of any pixel point (x, y) in the target height map, set the preset search size to v, and search the height data of four pixel points (xv, y), (x+v, y), (x, yv), and (x, y+v) in the target height map respectively;

[0172] SA392: Determine whether the height errors between the height data of the four pixel points (xv, y), (x+v, y), (x, yv), and (x, y+v) and the height data of the pixel point (x, y) are all greater than the preset height error threshold. If so, determine that the pixel point (x, y) is an abnormal point; otherwise, determine that the pixel point (x, y) is a normal point;

[0173] SA393: Traverse the height data of each pixel point (x, y) in the target height map, and judge each pixel point (x, y) according to the same method to obtain all abnormal points in the target height map after the abnormal points are screened out;

[0174] S394: Obtain an outlier point set corresponding to the target height map of the k-th flight image set based on all the obtained outlier points.

[0175] When capturing an image, if a pixel is underexposed or overexposed, the corresponding height data will have a large error with the height data of adjacent pixels. Therefore, this embodiment uses the above-mentioned abnormal point screening process to accurately screen out abnormal conditions caused by underexposure or overexposure of some pixels when capturing an image, and obtain an accurate set of abnormal points.

[0176] Preferably, in SA310, filling height data for all outliers in the outlier point set of the target height map of the kth flight image set includes:

[0177] SA3101: Select any non-edge outlier point (a1, b1) in the outlier point set of the k-th flight image set, and search the target height map for height data corresponding to four normal points adjacent to the non-edge outlier point (a1, b1) in the four search directions of up, down, left, and right, with the selected non-edge outlier point (a1, b1) as the center; calculate the mean of the height data of the four normal points found, and fill the height data obtained by the mean calculation into the height data of the non-edge outlier point (a1, b1) in the target height map to complete the filling of the height data of the non-edge outlier point (a1, b1);

[0178] SA3102: traverse each non-edge outlier point in the outlier point set of the k-th flight image set, and complete the height data filling of each non-edge outlier point using the same method;

[0179] SA3103: Select any edge outlier point (a2, b2) from the outlier point set of the k-th flight image set, and search the target height map for height data corresponding to a normal point that is closest to the edge outlier point (a2, b2) based on the selected edge outlier point (a2, b2); fill the height data corresponding to the searched normal point into the height data of the edge outlier point (a2, b2) in the target height map to complete the height data filling of the edge outlier point (a2, b2);

[0180] SA3104: traverse each edge outlier point in the outlier point set of the k-th flight image set, and complete the height data filling of each edge outlier point using the same method.

[0181] During the height data filling process for both non-edge outlier points and edge outlier points, normal points nearby are searched for and filled based on their height data. This not only restores the outlier points to normal after filling, ensuring the data validity of the height map, but also ensures that the data of the entire height map does not mutate, guaranteeing its data continuity. The order of SA3101-S3102 and SA3103-S3103 can be swapped.

[0182] In SA3101, when a non-edge outlier point (a1, b1) is selected from the outlier point set obtained after the target height map is filtered out again, the height data corresponding to the non-edge outlier point (a1, b1) in the target height map is Taking the non-edge abnormal point as the center, according to the four search directions of up, down, left and right, the height data corresponding to the four normal points adjacent to the non-edge abnormal point (a1, b1) in the target height map are searched respectively. and The formula for calculating the mean is:

[0183]

[0184] Calculate the mean Overwrite its original height data The height data filling of the non-edge outlier point can be achieved.

[0185] Similarly, in SA3102, the height data of each non-edge outlier point is filled in according to the same method.

[0186] In SA3103, when an edge outlier point (a2, b2) is selected from the outlier point set obtained after the target height map is filtered out again, the height data corresponding to the edge outlier point (a2, b2) in the target height map is Taking the edge abnormal point as the benchmark, search for the height data corresponding to the nearest normal point. Cover it to the original height data The height data filling of the edge abnormal point can be achieved.

[0187] Similarly, in SA3104, the height data of each edge outlier point is filled in according to the same method.

[0188] Finally, in SA310, the target height map after all outlier points are covered is Gaussian filtered to obtain the optimized height map. This is the height map that needs to be output in the end.

[0189] Example 2

[0190] like Figure 4 As shown, a semiconductor defect detection method includes:

[0191] SB1: According to the semiconductor super-depth of field image fusion method of Example 1, a total set of multi-channel images in the semiconductor processing process is obtained, and image fusion is performed on the total set of multi-channel images to obtain a target fusion result;

[0192] SB2: Perform image analysis based on the target fusion results to obtain semiconductor defect detection results.

[0193] In this embodiment, the target fusion result is obtained by the semiconductor super-depth of field image fusion method of Example 1. The target fusion result directly fuses images of different height layers into one image, so that the image is clear in each area, and at the same time, the original accompanying height data is displayed in the form of an image. Image analysis based on the target fusion result can significantly improve the accuracy of semiconductor surface defect detection (specifically, defect detection of wire bonding process and chip bonding process) to meet the high-precision requirements of defect detection and analysis in the semiconductor processing process.

[0194] The steps of the semiconductor super-depth image fusion method used in the semiconductor defect detection method described in this embodiment are the same as those of the semiconductor super-depth image fusion method in Example 1. For details not included in this embodiment, please refer to Example 1 and Figures 1 to 3 The detailed description is omitted here.

[0195] Example 3

[0196] like Figure 5 As shown, a semiconductor super-depth-of-field image fusion system is applied to the semiconductor super-depth-of-field image fusion method described in Example 1, comprising:

[0197] An image acquisition module is configured to acquire a multi-channel image collection; the multi-channel image collection includes a plurality of flight image collections, each of which includes flight images of the target under test at multiple altitude levels captured under illumination from a corresponding light source, wherein each flight image is accompanied by altitude data for each pixel at the altitude level at which it is located;

[0198] An image fusion module is configured to fuse all the fly-by-flight images in each of the fly-by-flight image sets based on a Laplace pyramid fusion method to obtain a fused image corresponding to each of the fly-by-flight image sets;

[0199] a height optimization module, which selects one of the flight image sets and optimizes all height data in the selected flight image set to obtain a target height map corresponding to the selected flight image set;

[0200] An output module is used to obtain a target fusion result based on the target height map and all the fused images.

[0201] The semiconductor super-depth-of-field image fusion system of this embodiment realizes the fusion of multi-channel super-depth-of-field images, and can directly fuse images of different height layers into one image, so that the image is clear in each area, and at the same time, the original accompanying height data is displayed in the form of an image, thereby realizing 3D restoration of multi-channel images, and can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis. The entire method is relatively simple to operate and is compatible with the current multi-channel light source detection mode of AOI detection of wire bonding and chip bonding, without the need for equipment programming and without taking up too much memory.

[0202] The functions of the various modules of the semiconductor super-depth image fusion system described in this embodiment correspond to the steps of the semiconductor super-depth image fusion method of Example 1. Therefore, for details not covered in this embodiment, please refer to Example 1 and Figures 1 to 3 The detailed description is omitted here.

[0203] Example 4

[0204] like Figure 6 As shown, a semiconductor defect detection system includes:

[0205] The semiconductor super-depth-of-field image fusion system in the third embodiment is used to obtain a total set of multi-channel images in a semiconductor processing process, and perform image fusion on the total set of multi-channel images to obtain a target fusion result; and

[0206] The image analysis module is used to perform image analysis based on the target fusion results to obtain semiconductor defect detection results.

[0207] This embodiment directly fuses images of different height layers into one image, making the image clear in every area, and at the same time displaying the original accompanying height data in the form of an image. Image analysis based on the target fusion result can significantly improve the accuracy of semiconductor surface defect detection (specifically, defect detection in wire bonding processes and chip bonding processes) to meet the high-precision requirements of defect detection and analysis in the semiconductor processing process.

[0208] Similarly, for details not included in this embodiment, please refer to Example 1, Example 2, Example 3 and Figures 1 to 5 The detailed description is omitted here.

[0209] Example 5

[0210] A semiconductor super-depth-of-field image fusion device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed, the method steps in the semiconductor super-depth-of-field image fusion method of embodiment 1 are implemented.

[0211] By storing a computer program on a memory and running it on a processor, the fusion of multi-channel super-depth-of-field images is achieved. Images at different height layers can be directly fused into one image, making the image clear in every area. At the same time, the original accompanying height data can be displayed in the form of an image, realizing 3D restoration of multi-channel images, which can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis. The entire method is relatively simple to operate and is compatible with the current multi-channel light source detection mode of AOI detection of wire bonding and chip bonding, without the need for device programming and without taking up too much memory.

[0212] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of a computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0213] The memory can be used to store computer programs and / or models. The processor implements various functions of the computer device by running or executing the computer programs and / or models stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, video data, etc.). In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0214] It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0215] These computer programs can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0217] This embodiment further provides a computer storage medium, which includes: at least one instruction, which implements the method steps in the semiconductor super-depth of field image fusion method of embodiment 1 when the instruction is executed.

[0218] By executing a computer storage medium containing at least one instruction, the fusion of multi-channel super-depth-of-field images is achieved. Images of different height layers can be directly fused into one image, making the image clear in every area. At the same time, the original accompanying height data can be displayed in the form of an image, realizing 3D restoration of multi-channel images, which can effectively improve the accuracy of subsequent semiconductor surface defect detection and analysis. The entire method is relatively simple to operate and is compatible with the current multi-channel light source detection mode of AOI detection of wire bonding and chip bonding, without the need for equipment programming and without taking up too much memory.

[0219] Similarly, for details not yet provided in this embodiment, please refer to the first and Figures 1 to 3 The detailed description is omitted here.

[0220] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A semiconductor super-depth of field image fusion method, characterized in that: include: Get the total set of multi-channel images; The multi-channel image collection includes a plurality of flight image sets, each of which includes flight images of the target to be measured at multiple altitude layers collected under the illumination of a corresponding light source, wherein each flight image is accompanied by altitude data of each pixel at the altitude layer; Based on the Laplacian pyramid fusion method, all the flight-captured images in each flight-captured image set are fused to obtain a fused image corresponding to each flight-captured image set; Selecting any one of the flight image sets, optimizing all height data in the selected flight image set, and obtaining a target height map corresponding to the selected flight image set; Obtaining a target fusion result according to the target height map and all the fused images; When the kth in-flight image set is selected from the total multi-channel image set, all height data in the selected in-flight image set are optimized to obtain a target height map corresponding to the selected in-flight image set, including: Performing filtering and absolute value processing on each of the k-th flying images in the flying image set in sequence to obtain a source image corresponding to each flying image; Select the source image corresponding to the j-th flying image, perform block sampling on the j-th source image according to a preset block sampling size, and obtain a sampled image corresponding to the j-th flying image; According to the preset neighborhood size, a neighborhood summation operation is performed on each pixel of the j-th sampling image, and the result of the neighborhood summation of each pixel is filled into each corresponding pixel point to obtain the evaluation image corresponding to the j-th sampling image; Assuming that the preset neighborhood size is 2f+1 pixels, and f is a positive integer, the pixel value corresponding to the pixel point (x, y) in the jth evaluation image is: in, is the pixel value corresponding to the pixel point (x, y) of the j-th evaluation image in the k-th flight image set, is the pixel value corresponding to the pixel point (p, q) in the neighborhood of the pixel point (x, y) of the j-th sampled image; Traversing each of the k-th flying images in the flying image set, and obtaining an evaluation image corresponding to each of the k-th flying images in the flying image set according to the same method; Extracting pixel values ​​of each pixel in all evaluation images of the kth in-flight image set to form an evaluation array corresponding to each pixel; extracting height data of each pixel in all in-flight images of the kth in-flight image set to form a height array corresponding to each pixel; the data in the evaluation array corresponds to the data in the height array in a one-to-one manner; Extract the maximum pixel value in the evaluation array and the maximum height data in the corresponding height array for each pixel point, and determine whether the corresponding pixel point is an abnormal point based on each maximum pixel value; When the pixel point (x, y) is a normal point, extract the two adjacent pixel values ​​adjacent to the maximum pixel value in the evaluation array corresponding to the pixel point (x, y), and extract the height data corresponding to the two adjacent pixel values ​​in the height array corresponding to the pixel point (x, y); calculate the actual height of the kth flying image set at the pixel point (x, y) based on the extracted two adjacent pixel values, the height data corresponding to the two adjacent pixel values, the maximum pixel value, and the maximum height data; According to the same method, the actual height of each normal point of the k-th flight image set is obtained; and based on all normal points and the actual heights at all normal points, the target height map corresponding to the k-th flight image set is obtained.

2. The semiconductor super-depth of field image fusion method according to claim 1, characterized in that: The obtaining of the total set of multi-channel images comprises: Obtaining an action sequence and an action trajectory corresponding to the light source camera used in a Z-axis flying photography scenario of the target to be measured; the action sequence includes a plurality of light source combinations that the light source camera needs to light up during the Z-axis flying photography and the lighting time of each light source in each light source combination; the action trajectory includes altitude data of the light source camera at multiple altitude layers during the Z-axis flying photography; According to the action sequence and the action trajectory, the light source camera is controlled to perform Z-axis flying photography of the target to be measured, so as to obtain a plurality of flying photography images of the target to be measured at each height data and arranged according to the lighting time of each light source combination; Clustering all the flight-by-flight images according to the light source types in the light source combination to obtain a plurality of flight-by-flight image sets; The total set of multi-channel images is obtained based on all the flight image sets.

3. The semiconductor super-depth of field image fusion method according to claim 1, characterized in that: The Laplacian pyramid fusion method is based on fusing all the fly-by-flight images in each of the fly-by-flight image sets to obtain a fused image corresponding to each of the fly-by-flight image sets, including: Selecting the kth flying image set from the total multi-channel image set; According to the preset pyramid layer N, constructing a Gaussian pyramid from the 1st layer to the N+1th layer for each of the kth flying images in the flying image set; Selecting the jth flying image in the kth flying image set, and constructing a Laplacian pyramid of the jth flying image in the first to Nth layers according to each Gaussian pyramid of the jth flying image in the first to Nth layers; The formula for constructing the Laplacian pyramid of the jth flying image in the kth flying image set at the i-th layer is: in, is the Laplacian pyramid of the jth flying image in the kth flying image set at the i-th layer, is the Gaussian pyramid of the jth flying image in the kth flying image set at the i-th layer, P yr Up(·) is the upsampling function, P yr Down(·) is a downsampling function; i is a positive integer and satisfies 1≤i≤N; j is a positive integer and satisfies 1≤j≤M, where M refers to the number of images in each of the flight-by-flight image sets; j is a positive integer and satisfies 1≤j≤M, where M refers to the number of images in each of the flight-by-flight image sets; k is a positive integer and satisfies 1≤k≤K, where K is the total number of light source types, and is also the total number of the flight-by-flight image sets in the total multi-channel image set; Traversing each of the k-th flying images in the flying image set, and obtaining a Laplacian pyramid of each flying image from the 1st layer to the Nth layer according to the same method; Select the Laplacian pyramid of all the flying images in the k-th flying image set at the i-th layer, obtain the maximum absolute value of all the selected Laplacian pyramids at each pixel point, and obtain the target pixel value of each pixel point of the k-th flying image set according to the maximum absolute value at each pixel point; and obtain the target pyramid of the k-th flying image set at the i-th layer according to the target pixel values ​​of all pixels of the k-th flying image set, which is recorded as Traverse the Laplacian pyramid of all the flying images in the k-th flying image set at each layer, and follow the same method to obtain the target pyramid of the k-th flying image set at each layer, which is recorded as Wherein, respectively refer to the target pyramid of the kth flight image set at the 1st layer, the 2nd layer, ..., the Nth layer; Performing mean calculation on the Gaussian pyramid of the kth flying image set at the N+1 layer to obtain a Gaussian pixel map of the kth flying image set at the N+1 layer; According to the Gaussian pixel map of the kth in-flight image set at the N+1 layer and all target pyramids, a fused image corresponding to the kth in-flight image set is obtained and output; Each of the flying image sets in the multi-channel image set is traversed, and the fused image corresponding to each of the flying image sets is obtained and output according to the same method.

4. The semiconductor super-depth of field image fusion method according to claim 3, characterized in that: For the kth in-flight image set, obtaining and outputting a fused image corresponding to the kth in-flight image set based on the Gaussian pixel map of the kth in-flight image set at the N+1 layer and all target pyramids, includes: Perform Gaussian upsampling on the Gaussian pixel map of the kth in-flight image set at the N+1 layer to obtain a Gaussian sampling map at the N+1 layer; and sum the Gaussian sampling map of the kth in-flight image set at the N+1 layer with the target pyramid at the N layer to obtain the target pixel map of the kth in-flight image set at the N layer; according to the same method, perform Gaussian upsampling on the target pixel map of the kth in-flight image set at the i+1 layer layer by layer, and sum it with the target pyramid at the i layer of the kth in-flight image set until i reaches 1, and obtain the target pixel map of the kth in-flight image set at the 1st layer, which is used as the fused image corresponding to the kth in-flight image set and outputted; The specific formula for performing Gaussian upsampling on the target pixel map of the kth in-flight image set at the i+1 layer and summing it with the target pyramid of the kth in-flight image set at the i layer is: Among them, TOP i k is the target pixel map of the kth flying image set at layer i, is the target pixel map of the kth flying image set at the i+1th layer.

5. The semiconductor super-depth of field image fusion method according to claim 1, characterized in that: When the pixel point (x, y) is a normal point, the calculation formula for the actual height of the kth flying image set at the pixel point (x, y) is: in, is the actual height of the kth flight image set at the pixel point (x, y), and are the first intermediate parameter and the second intermediate parameter respectively, and They are the maximum pixel value of the pixel point (x, y) in the corresponding evaluation array and the maximum height data in the corresponding height array, and They are the two adjacent pixel values ​​adjacent to the maximum pixel value in the evaluation array corresponding to the pixel point (x, y), and They are the height data corresponding to the two adjacent pixel values ​​in the height array corresponding to the pixel point (x, y).

6. The semiconductor super-depth of field image fusion method according to claim 5, characterized in that: The step of determining whether a corresponding pixel point is an abnormal point according to each maximum pixel value includes: For the pixel point (x, y), calculate the pixel error between each pixel value in the evaluation array corresponding to the pixel point (x, y) and the maximum pixel value; When all pixel errors meet the first outlier criterion, the pixel point (x, y) is determined to be an outlier; otherwise, the pixel point (x, y) is determined to be a normal point; The first abnormal point criterion is: Among them, V l k,(x,y) is the lth pixel value in the evaluation array of the kth flight image set at the pixel point (x, y), is the error between the lth pixel value and the maximum pixel value in the evaluation array at the pixel point (x, y), u is the preset pixel error threshold, and abs(·) is the absolute value function; Traverse each pixel and judge each pixel in the same way.

7. The semiconductor super-depth of field image fusion method according to claim 5, characterized in that: After obtaining the target height map corresponding to the k-th flight image set, the method further includes: For a pixel point (x, y) in the target height map corresponding to the kth in-flight image set, a preset search size and a preset height error threshold are obtained, and outlier points in the target height map are again screened out according to the preset search size and the preset height error threshold to obtain an outlier point set in the target height map of the kth in-flight image set; Filling all outliers in the outlier point set of the target height map of the kth in-flight image set with height data, overwriting the target height map corresponding to the kth in-flight image set based on the filled outlier point set and the corresponding height data, and performing Gaussian filtering on the overwritten target height map to obtain an optimized height map corresponding to the kth in-flight image set; Obtaining a target fusion result according to the target height map and all the fused images includes: The optimized height map and all the fused images are output to obtain the target fusion result.

8. The semiconductor super-depth of field image fusion method according to claim 7, characterized in that: The step of again filtering out outliers from the target height map according to the preset search size and the preset height error threshold to obtain an outlier set of the target height map of the kth flight image set includes: Select the height data of any pixel point (x, y) in the target height map, set the preset search size to v, and search the height data of four pixel points (xv, y), (x+v, y), (x, yv), and (x, y+v) in the target height map respectively; Determine whether the height errors between the height data of the four pixel points (xv, y), (x+v, y), (x, yv) and (x, y+v) and the height data of the pixel point (x, y) are all greater than the preset height error threshold. If so, determine that the pixel point (x, y) is an abnormal point; otherwise, determine that the pixel point (x, y) is a normal point; Traversing the height data of each pixel point (x, y) in the target height map, judging each pixel point (x, y) according to the same method, and obtaining all abnormal points in the target height map after the abnormal points are screened out; According to all the obtained abnormal points, a set of abnormal points corresponding to the target height map of the k-th flight image set is obtained.

9. The semiconductor super-depth of field image fusion method according to claim 7, characterized in that: Filling height data for all outliers in the outlier point set of the target height map of the kth flight image set includes: Select any non-edge outlier point (a1, b1) in the outlier point set of the k-th flight image set, and search for height data corresponding to four normal points adjacent to the non-edge outlier point (a1, b1) in the target height map in four search directions of up, down, left, and right, with the selected non-edge outlier point (a1, b1) as the center; calculate the mean of the height data of the four normal points found, and fill the height data obtained by the mean calculation into the height data of the non-edge outlier point (a1, b1) in the target height map to complete the filling of the height data of the non-edge outlier point (a1, b1); Traversing each non-edge outlier point in the outlier point set of the k-th flight image set, and completing the height data filling of each non-edge outlier point in the same manner; Select any edge outlier point (a2, b2) in the outlier point set of the k-th flight image set, and use the selected edge outlier point (a2, b2) as a reference to search for height data corresponding to a normal point that is closest to the edge outlier point (a2, b2) in the target height map; fill the height data corresponding to the searched normal point into the height data of the edge outlier point (a2, b2) in the target height map to complete the height data filling of the edge outlier point (a2, b2); Traverse each edge outlier point in the outlier point set of the k-th flight image set, and complete the height data filling of each edge outlier point in the same way.

10. A semiconductor defect detection method, characterized in that: include: According to the method according to any one of claims 1 to 9, a total set of multi-channel images in a semiconductor processing process is acquired, and image fusion is performed on the total set of multi-channel images to obtain a target fusion result; Image analysis is performed based on the target fusion results to obtain semiconductor defect detection results.

11. A semiconductor super-depth-of-field image fusion system, characterized in that: Applied to the method according to any one of claims 1 to 9, comprising: An image acquisition module is configured to acquire a multi-channel image collection; the multi-channel image collection includes a plurality of flight image collections, each of which includes flight images of the target under test at multiple altitude levels captured under illumination from a corresponding light source, wherein each flight image is accompanied by altitude data for each pixel at the altitude level at which it is located; An image fusion module is configured to fuse all the fly-by-flight images in each of the fly-by-flight image sets based on a Laplace pyramid fusion method to obtain a fused image corresponding to each of the fly-by-flight image sets; a height optimization module, which selects one of the flight image sets and optimizes all height data in the selected flight image set to obtain a target height map corresponding to the selected flight image set; An output module is used to obtain a target fusion result based on the target height map and all the fused images.

12. A semiconductor defect detection system, characterized in that: include: The semiconductor super-depth-of-field image fusion system according to claim 11 is used to obtain a total set of multi-channel images in a semiconductor processing process, and perform image fusion on the total set of multi-channel images to obtain a target fusion result; as well as The image analysis module is used to perform image analysis based on the target fusion results to obtain semiconductor defect detection results.

13. A computer storage medium, characterized in that The computer storage medium comprises: at least one instruction, which implements the method steps according to any one of claims 1 to 9 when the instruction is executed.

Images