An ultra-deep-field image fusion method, device, equipment and readable storage medium

Abstract

This application discloses a method, apparatus, device, and readable storage medium for super-depth-of-field image fusion, relating to the field of super-depth-of-field image fusion technology. It includes: acquiring several sample images with different focal lengths for multiple test object samples; determining the distribution characteristics of different height structures of the test object based on the sample images, dividing the sample images into multiple structural regions representing different height structures, and establishing a spatial position model of the structural regions in the coordinate system of the test object samples; acquiring test images of the test object with different focal lengths; locating the test object, and mapping the spatial position model to the image coordinate system of the test images based on the location results; evaluating the sharpness of each test image at the location corresponding to the set of structural regions, and determining the target focal length image that will enable each structural region to be clearly imaged; and performing image registration and image fusion processing on each target focal length image to generate a super-depth-of-field image.

Description

Technical Field

[0001] This invention relates to the field of ultra-depth-of-field image fusion technology, and more specifically to an ultra-depth-of-field image fusion method, apparatus, device, and readable storage medium. Background Technology

[0002] In current semiconductor defect detection processes, the surface of the chip under inspection typically exhibits significant three-dimensional morphological differences, such as different levels of circuit structures, localized protrusions or depressions, resulting in substantial height variations on the chip surface. Limited by the depth-of-field range of optical imaging systems, traditional detection methods that rely on cameras to capture single images struggle to simultaneously and clearly present product features at different height levels within the same focal plane, thus affecting the completeness and accuracy of defect detection.

[0003] To overcome the blurring problem caused by insufficient depth of field, existing technologies generally employ multifocal image fusion schemes. This involves acquiring multiple images with different focal lengths during the image acquisition phase, and then using an image fusion algorithm to generate a full depth-of-field image. For example, in practical applications, it is typically necessary to acquire 15 to 30 high-resolution images, such as 30 images with a resolution of 4096×2296 pixels. However, this type of scheme reveals significant real-time performance limitations in industrial production scenarios: fusing a set of multifocal images often takes several seconds, typically about 6 seconds, while semiconductor production lines have high requirements for inspection speed and cycle time, and existing processing speeds are insufficient to meet actual production needs.

[0004] Existing multifocal image fusion workflows generally include: acquiring images at all different focal lengths, performing global registration and alignment on the acquired images, and then using fusion algorithms such as weighted averaging and Laplacian pyramid to generate a full-depth image. Analysis reveals that existing fusion algorithms are relatively mature, and their performance is not the main bottleneck; the overall system efficiency is limited primarily due to inherent flaws in the processing workflow design.

[0005] On the one hand, existing methods generally adopt a full-processing strategy, uniformly registering and fusing all acquired images without distinguishing the actual contribution of different images to the final fusion result in different regions. In multifocal image sequences, usually only a few images contain clear and effective information in specific regions, while the remaining images not only contribute limitedly but may even interfere with the fusion effect, yet they still participate in the high-cost computation process, resulting in a large amount of redundant computation.

[0006] On the other hand, existing technologies do not fully utilize the characteristic of semiconductor chip surface structures having clearly defined planar layers and height regularities. For the same product model, the distribution of planes at different heights is consistent, and the clear areas in images at different focal lengths exhibit stable regularities. However, existing processes do not incorporate this type of prior information for process optimization, resulting in the need to repeat full image processing for each detection.

[0007] Furthermore, global registration calculations between multiple images require establishing multiple pairs of geometric correspondences, and their computational complexity increases linearly or even quadratically with the number of images. At high resolutions and with a large number of images, the registration step typically accounts for 60%–70% of the total processing time, becoming a major performance bottleneck that makes multifocal image fusion in semiconductor defect detection unable to meet real-time requirements. Therefore, a super-depth-of-field image fusion method that can overcome these shortcomings is urgently needed. Summary of the Invention

[0008] The purpose of this invention is to provide a method, apparatus, device, and readable storage medium for ultra-depth-of-field image fusion. During the system configuration phase, based on multiple test object samples of the same model, sample images with different focal lengths are pre-acquired, and the distribution characteristics of different height structures of the test object are determined. A spatial position model of the structural regions in the coordinate system of the test object samples is established, enabling the accurate localization of different height structures of the test object during subsequent online detection. During online detection, by locating the test object and mapping the spatial position model to the image coordinate system of the test image, the corresponding regions of each height structure in the test image are quickly determined. Sharpness evaluation is performed only on images with different focal lengths within the corresponding structural regions, selecting target focal length images that provide clear imaging of each structural region. Image registration and fusion processing are then performed only on the target focal length images. This avoids blind full registration and fusion of all focal length images, significantly reduces the proportion of invalid images involved in the calculation, greatly reduces the amount of computation in the registration and fusion process, and especially alleviates the performance bottleneck caused by the increase in the number of images in the registration calculation. Under the premise of ensuring clear imaging of structures at different heights, it effectively improves the processing efficiency and real-time performance of ultra-depth-of-field image generation, thereby better meeting the requirements of online semiconductor defect detection for detection cycle time and stability.

[0009] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a super-depth-of-field image fusion method, the method comprising: During the system configuration phase, for multiple test object samples of the same model, several sample images of each test object sample with different focal lengths are collected. Based on the sample images, the distribution characteristics of the structure at different heights of the object under test are determined, multiple structural regions representing the structure at different heights are divided, and a spatial position model of the structural regions in the coordinate system of the object under test is established. When performing online detection on the object under test, several images of the object under test with different focal lengths are collected. The object under test is located, and the spatial location model is mapped to the image coordinate system of the image under test based on the location result, so as to determine the corresponding position of the set of structural regions in the image under test; the set of structural regions contains multiple structural regions at different heights in the object under test. At the locations corresponding to the set of structural regions, the sharpness of each tested image is evaluated, and the target focal length image that makes each structural region clear is determined. Image registration and image fusion are performed on the images of each target focal length to generate a super depth-of-field image.

[0010] In some embodiments, the location of the object under test is determined, and the spatial location model is mapped to the image coordinate system of the image under test based on the location result, including: Perform visual coarse localization on the image under test to determine the position and angular offset of the object under test in the image under test; The spatial location model is mapped to the image coordinate system of the image under test based on the position and angle offset.

[0011] In some embodiments, visual coarse localization is performed on the image under test to determine the position and angular offset of the object under test in the image under test, including: The contour of the object being measured is extracted using an edge detection algorithm; Based on the minimum bounding rectangle of each contour, the position and angular offset of the measured object in the measured image are determined.

[0012] In some embodiments, mapping the spatial location model to the image coordinate system of the image under test based on position and angle offset includes: Based on position and angular offset, coordinate system one is established for all measured images; Map the spatial location model to the image coordinate system of the image being measured.

[0013] In some embodiments, the distribution characteristics of different height structures of the tested object are determined based on the sample image, and multiple structural regions representing different height structures are divided, including: Based on the sharpness distribution characteristics of the sample images, the surface of the object under test is divided into multiple structural regions that represent different height structures; For each structural region, if multiple structural regions have sample images with the same clarity, then the multiple structural regions are merged into one structural region.

[0014] In some embodiments, image registration and image fusion processing are performed on the images of each target focal length to generate a super-depth-of-field image, including: The target focal length image at the center position is selected as the reference image, and the transformation matrix of the non-reference image in the target focal length image relative to the reference image is calculated using the feature point matching method. Geometric correction of non-reference images is performed based on the transformation matrix to align the spatial dimensions of all target focal length images; The images of each target focal length are fused to obtain a super depth-of-field image fusion.

[0015] In some embodiments, the images of each target focal length are fused to obtain a super-depth-of-field image fusion, including: Based on the sharpness evaluation results of each target focal length image, fusion weights are assigned to each structural region in each target focal length image; The images of each target focal length are fused based on the fusion weights to obtain a super depth-of-field image fusion.

[0016] Secondly, the present invention also provides a super-depth-of-field image fusion apparatus, the apparatus comprising: The sample acquisition module is used during the system configuration phase to acquire several sample images of each test object sample with different focal lengths for multiple test object samples of the same model. The region segmentation module is used to determine the distribution characteristics of different height structures of the tested object based on the sample image, divide the object into multiple structural regions that represent different height structures, and establish a spatial position model of the structural regions in the sample coordinate system of the tested object. The image acquisition module is used to acquire several images of the object under test at different focal lengths when performing online detection on the object under test; The model mapping module is used to locate the position of the object under test. Based on the location result, the spatial position model is mapped to the image coordinate system of the image under test to determine the corresponding position of the set of structural regions in the image under test. The set of structural regions contains multiple structural regions at different heights in the object under test. The sharpness evaluation module is used to evaluate the sharpness of each tested image at the location corresponding to the set of structural regions, and to determine the target focal length image that makes each structural region sharp. The image fusion module is used to perform image registration and image fusion processing on images of various target focal lengths to generate super depth-of-field images.

[0017] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the super-depth-of-field image fusion method provided in the first aspect.

[0018] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the super-depth-of-field image fusion method provided in the first aspect.

[0019] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the ultra-depth-of-field image fusion method provided in the first aspect.

[0020] The beneficial effects of this invention are as follows: In the system configuration stage, the super-depth-of-field image fusion method of this invention acquires several sample images of each test object sample with different focal lengths for multiple test object samples of the same model; based on the sample images, the distribution characteristics of the structure at different heights of the test object are determined, and multiple structural regions representing different height structures are divided, and a spatial position model of the structural regions in the coordinate system of the test object sample is established; when performing online detection on the test object, several test images of the test object with different focal lengths are acquired; the test object is located, and based on the location result, the spatial position model is mapped to the image coordinate system of the test image to determine the corresponding position of the set of structural regions in the test image; the set of structural regions contains multiple structural regions at different heights in the test object; at the position corresponding to the set of structural regions, the sharpness of each test image is evaluated, and the target focal length image that makes each structural region clear is determined; image registration and image fusion processing are performed on each target focal length image to generate a super-depth-of-field image. By pre-collecting sample images of different focal lengths based on multiple test object samples of the same model during the system configuration phase and determining the distribution characteristics of structures of different heights of the test object, a spatial position model of the structural region in the coordinate system of the test object sample is established. This allows for the direct and accurate positioning of structures of different heights of the test object during subsequent online inspection. During online inspection, the test object is located and the spatial position model is mapped to the image coordinate system of the test image, thereby quickly determining the corresponding region of each height structure in the test image. Sharpness evaluation is performed only on images of different focal lengths within the corresponding structural region, selecting target focal length images that provide clear imaging of each structural region. Image registration and fusion processing are then performed only on the target focal length images. This avoids blind full registration and fusion of all focal length images, significantly reduces the proportion of invalid images involved in the calculation, and greatly reduces the computational load during registration and fusion. In particular, it alleviates the performance bottleneck caused by the increase in the number of images during registration calculation. While ensuring clear imaging of structures of different heights, it effectively improves the processing efficiency and real-time performance of ultra-depth-of-field image generation, thus better meeting the requirements of semiconductor online defect detection for inspection cycle time and stability.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0022] Figure 1This is a flowchart illustrating a super-depth-of-field image fusion method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating a process of mapping a spatial location model to the image coordinate system of a measured image according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating another super-depth-of-field image fusion method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a super-depth-of-field image fusion device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics; however, not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0025] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] In some embodiments, such as Figure 1 The diagram illustrates a flowchart of a super-depth-of-field image fusion method, which includes the following specific steps: S101, during the system configuration phase, for multiple test object samples of the same model, several sample images of each test object sample with different focal lengths are collected.

[0027] The object being tested can be a semiconductor chip.

[0028] Specifically, the process begins with hardware configuration and sample preparation. An image acquisition platform is used, comprising a high-resolution area array industrial camera, a high-precision servo-controlled Z-axis, a microscope lens, and a uniform illumination system. Three to five chips are selected as the test samples to eliminate individual differences. Subsequently, image sequence acquisition is performed: after fixing one sample chip, the Z-axis is controlled within a range covering the maximum height difference on the chip surface with a margin, performing precise scanning at equal steps (e.g., 3-4 micrometers) slightly smaller than the depth of field of the optical system. At each stop position, the camera triggers the capture of an image, thus obtaining a sequence of dozens (e.g., 30) clear images of the sample at different focal lengths, each image associated with its precise Z-axis position coordinates. This process is repeated on all samples to collect sufficient raw data.

[0029] The fundamental purpose of acquiring multifocal image sequences is to construct a "region-resolution" mapping model. To this end, the system configuration phase typically includes subsequent analysis and processing steps: first, sub-pixel-level registration is performed on the image sequences of each sample to ensure spatial alignment; then, by calculating the resolution index (such as Laplacian response variance) of each pixel in different images, a cluster of "resolution-focal length" curves for each location on the chip surface is generated. Analyzing the peak distribution patterns of these curves identifies continuous regions on the chip surface that simultaneously reach a resolution peak at the same or adjacent focal lengths. These regions correspond to structural planes at different physical heights (such as substrates, conductive layers, solder balls, etc.). Finally, the boundary information of these regions (preferably in the form of polygon vertex coordinates) and their corresponding optimal imaging focal lengths or image numbers are structurally saved as a configuration file specific to this model. This configuration file carries the "region division information" extracted from offline analysis, becoming the decision-making basis for achieving rapid image screening and fusion in the online detection phase.

[0030] S102, Based on the sample image, determine the distribution characteristics of the structure at different heights of the object under test, divide the structure into multiple structural regions that represent the structure at different heights, and establish a spatial position model of the structural regions in the coordinate system of the object under test sample.

[0031] Specifically, the sharpness of the registered and aligned sample image sequence is first calculated pixel-by-pixel, generating a sharpness variation curve for each point on the chip surface across different focal lengths. Observation reveals that regions belonging to the same physical height plane tend to have sharpness peaks at the same or adjacent focal lengths; however, the peak positions shift significantly between different height planes. Utilizing this pattern, combined with prior knowledge that the semiconductor chip surface structure has a clearly defined layered structure, the chip surface can be divided into regions. For example, the chip's packaging substrate, peripheral wires, central core circuitry, and raised solder ball array, due to their different physical heights, will each achieve optimal sharpness within specific focal length ranges in the image sequence. The analysis algorithm (or with the aid of manual annotation) can divide the chip surface into several structural regions corresponding to different height levels by identifying connected pixel sets with similar sharpness peak distribution characteristics.

[0032] To accurately describe the morphology and location of these structural regions, this invention preferably uses polygonal boundaries to define each region. This is because the circuit and structural outlines of a chip are usually not regular rectangles; polygons can better fit their true geometry and reduce interference from background or irrelevant areas. After determining the polygonal regions, a unified coordinate system associated with the characteristics of the sample under test—that is, the "sample coordinate system"—needs to be established. Typically, this coordinate system uses a stable feature of the chip (such as a specific contour corner or marker center) as its origin and its edge direction as its axis. Under this coordinate system, the coordinates of each vertex of each polygonal region are accurately recorded, thus forming a "spatial location model." This model is essentially a structured dataset that abstractly describes "where" (the location of the polygonal region) on the chip corresponds to "what height structure," laying the mathematical foundation for quickly mapping prior knowledge onto any image of the chip under test during the online detection phase. Finally, this spatial location model, along with the optimal focal length information corresponding to each region, is saved to a dedicated configuration file for this chip model, completing the construction of the offline knowledge base.

[0033] Optionally, determining the distribution characteristics of different height structures of the tested object based on the sample images and dividing them into multiple structural regions that represent different height structures may also include: dividing the surface of the tested object into multiple structural regions that represent different height structures based on the sharpness distribution characteristics of the sample images; and merging multiple structural regions into one structural region if the sample images corresponding to multiple structural regions have the same sharpness.

[0034] Specifically, in some chip designs, there may be multiple independent structures that are physically discontinuous but at the same height (e.g., test pads of the same height scattered throughout the chip). During initial segmentation, the algorithm might identify these as multiple independent structural regions. However, analysis reveals that these regions exhibit completely consistent or highly consistent sharpness variations across all sample images; that is, they are simultaneously sharp and blurry in the same image. This indicates that they not only have the same height but also consistent optical imaging characteristics. From the perspective of simplifying the model and improving online processing efficiency, the system merges these regions, recording them in the spatial location model as a composite region with multiple polygonal boundaries (or a complex polygon). Thus, in subsequent online screening, only one sharpness evaluation of this composite region is needed to represent all physically height-equivalent sub-parts, further reducing unnecessary computation and making the model more concise and efficient.

[0035] S103: When performing online detection on the object under test, acquire several images of the object under test with different focal lengths.

[0036] Specifically, after the chip under test is transferred to the vision inspection station and precisely positioned, the industrial control computer integrated into the system sends instructions to the motion control card, driving the high-precision servo Z-axis onboard camera and lens to execute a preset fast focusing and scanning program. This program does not require recalculating the scanning range; instead, it directly calls the acquisition parameters optimized and determined for the same model of device during the offline modeling stage: including the starting point, ending point, and fixed step distance. Subsequently, the Z-axis steps sequentially to each preset focal length position at a high and stable speed. After a brief pause at each position to allow vibration to dissipate, the camera performs exposure and captures a high-resolution image (e.g., 4096×2296 pixels) corresponding to that focal length under a precise trigger signal from the control card. This process is repeated continuously until all preset positions have been traversed, typically completing the acquisition of a complete image sequence for a device in a very short time (e.g., within 1 second), resulting in a set of "images under test" with identical content but different clear planes. The entire acquisition process is synchronized with the production cycle, ensuring real-time inspection. This set of images serves as the data foundation for rapid region analysis and filtering fusion in subsequent processes.

[0037] S104, locate the position of the object under test, and map the spatial position model to the image coordinate system of the image under test based on the location result, so as to determine the corresponding position of the set of structural regions in the image under test.

[0038] The set of structural regions includes multiple structural regions at different heights within the object being measured.

[0039] Specifically, the system first selects one image from the multiple acquired test images (usually the one with the middle focal length or the one estimated to be the clearest) for rapid visual coarse localization. Using algorithms such as edge detection combined with contour extraction, or fast matching algorithms based on chip standard templates, the precise translation (Δx, Δy) and rotation angle (Δθ) of the chip under test in that image are calculated. This coarse localization process only needs to process a single image and does not require high absolute accuracy; its core purpose is to efficiently obtain the geometric transformation relationship of the chip from its ideal standard position to its actual imaging position.

[0040] Subsequently, the system calls the configuration file pre-stored for this chip model and reads the "spatial location model" defined based on the chip's own characteristics—that is, the set of vertex coordinates for each polygonal structural region. Using the transformation parameters (Δx, Δy, Δθ) obtained in the previous step, the system performs a simple affine or rigid transformation to batch map the coordinates of all vertices in the model to the pixel coordinate system of the currently tested image. This mathematical mapping process is completed instantaneously, and its output is the precise pixel position range corresponding to each structural region to be evaluated (such as the substrate area, solder ball area, etc.) on the currently captured image. Thus, the abstract region segmentation knowledge obtained from offline analysis is successfully and dynamically "attached" to each real image acquired online, laying a solid foundation for efficient sharpness analysis and image selection only within these specific regions.

[0041] S105, at the location corresponding to the set of structural regions, the sharpness of each tested image is evaluated, and the target focal length image that makes each structural region clear is determined.

[0042] Specifically, the system processes each structural region mapped to the current image coordinate system sequentially. For each polygonal region, the algorithm traverses all acquired test images (e.g., 30 images) and performs the following operations in sequence: based on the polygon vertex coordinates of the region, it crops the corresponding image patch from each image; subsequently, it calculates a quantified sharpness evaluation value for the image patch. Sharpness evaluation typically employs computationally efficient spatial domain algorithms, such as calculating the variance of the image patch after convolution with the Laplacian operator; a larger value indicates a sharper image. Alternatively, the sum of gradient magnitudes or image variance can be used. After traversing all images, the system records the number of the image (or images with the highest sharpness evaluation value) for that region; this is the "target focal length image that makes the structural region sharp."

[0043] After all preset structural regions have undergone the aforementioned evaluation, the system summarizes the results. Since the clearest images in different regions may be the same or different, a deduplicated set of image numbers is obtained after summarizing. This set typically contains only 3 to 5 images, far fewer than the total number of images originally acquired (e.g., 30). These selected images are the "representative images" that each carry the clearest information from different height planes of the chip; they collectively contain all the information needed to reconstruct a complete and clear field of view. Through this targeted clarity evaluation based on prior regions, the system avoids the expensive full-screen, pixel-by-pixel calculations of traditional methods, thereby achieving an order-of-magnitude improvement in processing speed and laying the foundation for real-time detection.

[0044] S106 performs image registration and image fusion processing on the focal length images of each target to generate a super depth-of-field image.

[0045] Specifically, the process begins with high-precision image registration. Since the number of images involved in registration is significantly reduced (typically only 3 to 5), the system can employ more sophisticated algorithms to ensure alignment quality. In one embodiment, the system selects one image from the subset (e.g., an image with a centered sharp region or a centered number) as a reference. Subsequently, for each of the remaining images in the subset, stable feature points between the image and the reference image are extracted and matched using algorithms such as Scale Invariant Feature Transform (SIFT) or Oriented FAST and Rotated BRIEF (ORB). After filtering out mismatched points using the Random Sample Consensus (RANSAC) algorithm, an affine or projective transformation matrix describing the geometric relationship between the two images is calculated. Based on this matrix, the non-reference image is resampled and geometrically corrected to achieve spatial alignment with the reference image at the sub-pixel level, thereby ensuring that the same structural features from different focal lengths are precisely overlapped before fusion.

[0046] After registration, the multi-scale image fusion stage begins. This invention preferably employs a multi-band fusion strategy based on the Laplacian pyramid. First, a Laplacian pyramid is constructed for each registered image to decompose image details at different spatial frequencies. Simultaneously, based on the sharpness scores of each region obtained in the previous sharpness evaluation stage, a corresponding weight map is generated for each image. At each pixel location, the weight tends to be assigned to the sharpest source image at that location. A Gaussian pyramid is constructed for this weight map to achieve multi-scale smoothing. Subsequently, at each layer of the pyramid, the Laplacian coefficients of each source image are weighted and fused with the weights of their corresponding layers to obtain the fused pyramid coefficients. Finally, a reconstruction process is performed on the fused Laplacian pyramid. Through top-down upsampling and summation, the complete spatial image is recovered step by step, generating the final "super-depth-of-field image." This image integrates the sharpest parts of all target focal length images, achieving a clear full-depth-of-field presentation from the chip substrate to the highest protrusion structure, with the overall processing time controlled within milliseconds, meeting the stringent requirements of real-time inspection in semiconductor production lines.

[0047] Optionally, image registration and image fusion processing are performed on each target focal length image to generate a super depth-of-field image, including: selecting the target focal length image at the center as the reference image, calculating the transformation matrix of the non-reference image in the target focal length image relative to the reference image using the feature point matching method; performing geometric correction on the non-reference image based on the transformation matrix to spatially align all target focal length images; and fusing each target focal length image to obtain a fused super depth-of-field image.

[0048] Specifically, to reduce the geometric differences between the reference image and the image to be registered and improve global consistency, a spatially centered image is typically selected as the reference, with the spatial relationship determined by the order of image capture. Subsequently, for each non-reference image in the subset, the system employs algorithms such as Scale Invariant Feature Transform (SIFT) or Speeded Robust Feature Transform (SURF) to extract stable keypoints and descriptors between it and the reference image. Through nearest-neighbor matching of descriptors, a preliminary correspondence between feature points in the two images is established. To eliminate mismatches caused by noise or repetitive structures, the Random Sample Consensus (RANSAC) algorithm is further used to iteratively estimate the optimal geometric transformation model (usually an affine or perspective transformation matrix), while simultaneously filtering out outliers. Finally, an accurate transformation matrix is ​​calculated for each non-reference image, describing the spatial mapping between its pixel coordinate system and the reference image's pixel coordinate system. Based on their respective transformation matrices, the system resamples and geometrically corrects all non-reference images, ensuring strict spatial alignment at the sub-pixel level across all images in the subset, guaranteeing that the same physical feature points from different focal lengths are precisely overlapped before fusion.

[0049] After accurate registration, the multi-scale image fusion stage begins. This invention preferably employs a fusion strategy based on Laplacian pyramid decomposition and reconstruction. First, a Laplacian pyramid is constructed for each aligned source image, decomposing it into sets of coefficients across different spatial frequency bands, thereby separating the image's details and contour information. Simultaneously, based on the region weight map generated for each image during the online sharpness evaluation stage (this weight map represents which image each pixel originated from that is the sharpest), a corresponding Gaussian pyramid is constructed to achieve multi-scale smoothing of the weights, avoiding abrupt transitions at the fusion boundary. Next, at each layer of the pyramid, the Laplacian coefficients of each source image at that layer are weighted and superimposed with their corresponding Gaussian weights, fusing to generate new coefficients for that frequency band. Finally, a top-down pyramid reconstruction process is performed on the fused Laplacian coefficients of each layer. Through continuous upsampling and summation, the complete spatial image is gradually recovered, generating the final "super-depth-of-field fused image." This image integrates the sharpest parts of all source images, achieving a clear full-depth-of-field presentation from the chip substrate to the highest protrusion structure. Because the number of images involved in this computationally complex process has been drastically reduced, the entire registration and fusion process can be completed in a very short time, thus meeting the stringent real-time requirements of online semiconductor inspection.

[0050] Optionally, the images of each target focal length are fused to obtain a super-depth-of-field image fusion, including: assigning fusion weights to each structural region in each target focal length image based on the sharpness evaluation results of each target focal length image; and fusing the images of each target focal length image based on the fusion weights to obtain a super-depth-of-field image fusion.

[0051] Specifically, based on the quantization results calculated in the online sharpness determination stage (e.g., the Laplacian variance of each structural region in each candidate image), an initial weight map of the same size as the original image is constructed for each target focal length image. For each pixel in the image, its weight value is determined by the structural region it belongs to: if the region where the pixel is located is determined to be the sharpest in the image, then the pixel is assigned a higher weight value (e.g., close to 1.0) in the weight map; if the region is not sharp in the image, it is assigned a lower weight value (e.g., close to 0). This region-based weight allocation strategy ensures that in the final fusion result, the main information of each physical location comes from the source image that imaged it the sharpest.

[0052] However, directly using an initial weight map that is rigidly divided according to region boundaries for fusion can easily produce obvious seams or unnatural transitions at region boundaries. To address this issue, the system applies Gaussian filtering to the initial weight map or performs multi-scale smoothing by constructing a Gaussian pyramid. This smoothing operation creates a soft, gradual transition of weights near region boundaries, thereby achieving a natural blending of information from different image sources during fusion and avoiding the introduction of artificial artifacts.

[0053] After obtaining the smoothed final weight map, the system performs pixel-level weighted fusion. For each coordinate position in the output image, its pixel value is obtained by multiplying the pixel values ​​of all registered source images at that position by their corresponding weight values ​​in the weight map, and then summing the results. This process can be represented as: Where I_k is the k-th source image, and W_k is its normalized weight map (satisfying ΣW_k(x,y)=1 for any (x,y). Through this adaptive fusion guided by sharpness weights, the final generated ultra-depth image integrates all key features, such as the fine texture of the chip substrate, the intricate wiring of the intermediate layer circuit, and the bright outline of the solder balls at high positions, into a single image at their optimal sharpness, providing high-quality input for subsequent high-precision defect detection algorithms. The entire fusion process is computationally efficient because it only requires processing a few images and their corresponding weight maps, effectively supporting the operation of real-time detection systems.

[0054] In the above embodiments, the ultra-depth-of-field image fusion method, during the system configuration phase, acquires several sample images of each test object sample with different focal lengths for multiple test object samples of the same model; determines the distribution characteristics of different height structures of the test object based on the sample images, divides multiple structural regions representing different height structures, and establishes a spatial position model of the structural regions in the coordinate system of the test object sample; during online detection of the test object, acquires several test images of the test object with different focal lengths; performs position localization on the test object, and maps the spatial position model to the image coordinate system of the test image based on the localization result to determine the corresponding position of the set of structural regions in the test image; the set of structural regions contains multiple structural regions of different heights in the test object; at the position corresponding to the set of structural regions, performs sharpness evaluation on each test image, and determines the target focal length image that makes each structural region clear; performs image registration and image fusion processing on each target focal length image to generate an ultra-depth-of-field image. By pre-collecting sample images of different focal lengths based on multiple test object samples of the same model during the system configuration phase and determining the distribution characteristics of structures of different heights of the test object, a spatial position model of the structural region in the coordinate system of the test object sample is established. This allows for the direct and accurate positioning of structures of different heights of the test object during subsequent online inspection. During online inspection, the test object is located and the spatial position model is mapped to the image coordinate system of the test image, thereby quickly determining the corresponding region of each height structure in the test image. Sharpness evaluation is performed only on images of different focal lengths within the corresponding structural region, selecting target focal length images that provide clear imaging of each structural region. Image registration and fusion processing are then performed only on the target focal length images. This avoids blind full registration and fusion of all focal length images, significantly reduces the proportion of invalid images involved in the calculation, and greatly reduces the computational load during registration and fusion. In particular, it alleviates the performance bottleneck caused by the increase in the number of images during registration calculation. While ensuring clear imaging of structures of different heights, it effectively improves the processing efficiency and real-time performance of ultra-depth-of-field image generation, thus better meeting the requirements of semiconductor online defect detection for inspection cycle time and stability.

[0055] In another embodiment, such as Figure 2 As shown, a flowchart illustrating the process of mapping a spatial location model to the image coordinate system of the measured image is also provided. The specific method includes: S201, Perform visual coarse localization on the image under test to determine the position and angular offset of the object under test in the image under test.

[0056] Specifically, one image (e.g., the one with the best overall contrast) is selected from multiple acquired images at different focal lengths for processing to avoid unnecessary repetitive calculations. The core of coarse localization lies in quickly extracting the overall contour of the chip. A typical implementation is as follows: First, adaptive thresholding or Canny edge detection is performed on the selected image to separate the chip region from the background, obtaining a set of continuous edge pixels. Next, a contour-finding algorithm is used to identify the largest closed contour from these edges; this contour corresponds to the outer boundary of the chip. Then, geometric analysis is performed on this contour, such as fitting its minimum bounding rectangle or using polygon approximation to determine its principal direction. By calculating the offset of the center coordinates of this bounding rectangle from the ideal center of the chip in the standard template, the translation parameters (Δx, Δy) of the chip can be obtained; by calculating the rotation angle of this rectangle relative to the horizontal axis of the image, the rotation parameters (Δθ) of the chip can be obtained. This series of image processing operations all use optimized classic algorithms, ensuring robustness while minimizing computational overhead, and can be completed in milliseconds, thus ensuring the real-time performance of the entire detection process. The translation and rotation parameters obtained in this way fully define the rigid body transformation of the current measured object from the ideal standard position to the actual imaging position, providing a key coordinate transformation basis for the next step of accurately mapping the predefined structural region.

[0057] Optionally, the process can also involve using an edge detection algorithm to extract the contour of the object under test; and determining the position and angular offset of the object under test in the image under test based on the minimum bounding rectangle of each contour.

[0058] Specifically, the selected image under test is first preprocessed, possibly including Gaussian filtering to suppress sensor noise while preserving chip edge information. Then, the Canny edge detection operator is applied, using gradient calculation, non-maximum suppression, and double-threshold hysteresis to obtain a clear binary edge map. In this edge map, the chip's physical boundaries, package edges, and certain high-contrast internal structures are likely highlighted. Next, the system executes a contour search algorithm to retrieve all closed contours from the edge map and sort them according to the area enclosed by the contours. Typically, the outer contour of the chip's package, due to its largest size, is identified as the contour with the largest area and is thus reliably extracted.

[0059] After obtaining the main contour, the system calculates its minimum bounding rectangle. This rectangle is the one that completely encloses the contour and has the smallest area, with its direction aligned with the principal axis of the contour. By directly reading the pixel coordinates of the center point of this rectangle, the chip's position (Δx, Δy) in the current image can be obtained. Simultaneously, the angle formed by this rectangle relative to the image's horizontal coordinate axis represents the chip's planar rotation angle offset (Δθ). This method fully utilizes the global geometric characteristics of the chip contour and exhibits good robustness to local defects, uneven illumination, or slight noise on the surface. The entire process involves minimal computation, requiring only basic image processing operators, and can be completed in a very short time. It provides fast and reliable initial positioning information for the entire real-time processing chain, ensuring accurate and efficient subsequent region mapping and image filtering steps.

[0060] S202, map the spatial position model to the image coordinate system of the image under test based on the position and angle offset.

[0061] Specifically, the system loads a pre-generated configuration file for this chip model from storage and reads the "spatial position model" recorded within it. This model is essentially a set of coordinates for each structural region in the coordinate system of the polygon vertices within the sample coordinate system of the object under test (usually with specific corners of the chip as the origin and the edges as the axes). Simultaneously, the system acquires the rigid body transformation parameters calculated by the visual coarse localization step, including the horizontal translation Δx, the vertical translation Δy, and the rotation angle Δθ.

[0062] Based on these parameters, the system constructs a coordinate transformation relationship. In a typical embodiment, a rigid transformation model is used. For any vertex coordinates (X_model, Y_model) in the model, its coordinates mapped to the current measured image pixel coordinate system (X_image, Y_image) are calculated using the following formula: First, rotate: ; ; Then, a translation is performed: X_image = X_rot + Δx; Y_image = Y_rot + Δy; The entire calculation process is completed in batches through matrix operations, resulting in extremely high efficiency. Ultimately, the polygon vertices of all structural regions are transformed to their actual pixel positions in the current image, thus "defining" the precise range of each critical region that needs to be evaluated within the current image. This mapping result is directly used to guide the next step—the system will extract image patches and perform sharpness analysis only within these dynamically determined polygonal regions that strictly correspond to the chip's actual orientation, thereby completely avoiding the time-consuming global image processing of traditional methods and achieving a leap in processing speed.

[0063] Optionally, the above process can also involve establishing a coordinate system for all measured images based on position and angle offsets, and mapping the spatial position model to the image coordinate system of the measured images.

[0064] Specifically, we first establish a premise: during the image acquisition phase, although the camera's focal length (Z-axis position) constantly changes, the chip under test itself does not move within the plane (XY plane). Therefore, the translation (Δx, Δy) and rotation angle (Δθ) calculated from a single image (e.g., the one used for coarse localization) are applicable to all other focal length images acquired in the same batch. Based on this, the system performs "Coordinate System 1" on all images under test. This is not resampling or deforming the image content, but rather logically associating and applying the same set of rigid body transformation parameters to each image. This ensures that when discussing a physical point on the chip, regardless of which focal length image it is in, its ideal position relative to the chip's own coordinate system can be transformed to the pixel coordinate system of that image using this same set of parameters. This establishes a unified mathematical benchmark for locating the same physical region in all subsequent images.

[0065] Subsequently, the system performs the core mapping operation. It loads a "spatial location model"—a list of coordinates of the polygon vertices of each structural region of the chip in its own coordinate system—from an offline configuration file. For each vertex coordinate (X_model, Y_model) in the model, the system uses predefined parameters (Δx, Δy, Δθ) and a uniform rigid transformation formula (e.g., rotation followed by translation) to calculate the corresponding pixel coordinates (X_image, Y_image) of that vertex in any tested image. This calculation process is completed in batches through efficient matrix operations, generating the same mapping result for all images at once. Thus, the polygon boundaries of each predefined structural region are synchronously and consistently located in images with different focal lengths. This step cleverly avoids the heavy computational burden of performing complex image registration for dozens of images one by one, achieving rapid deployment of the model across the entire image set with only a single coordinate transformation, paving the way for efficient and parallel sharpness evaluation in specific regions of each image in the next step.

[0066] To more comprehensively demonstrate this solution, this embodiment presents an optional approach to ultra-depth-of-field image fusion, such as... Figure 3 As shown: S301, during the system configuration phase, collects several sample images of different focal lengths for each of the multiple test object samples of the same model.

[0067] S302, based on the sharpness distribution characteristics of the sample image, divides the surface of the object under test into multiple structural regions that represent different height structures.

[0068] S303: For each structural region, if multiple structural regions have sample images with the same clarity, then the multiple structural regions are merged into one structural region.

[0069] S304, Establish the spatial position model of the structural region in the coordinate system of the sample of the measured object.

[0070] S305: When performing online detection on the object under test, it acquires several images of the object under test at different focal lengths.

[0071] S306 uses an edge detection algorithm to extract the contour of the object being measured.

[0072] S307, based on the minimum bounding rectangle of each contour, determines the position and angular offset of the measured object in the measured image.

[0073] S308, coordinate system unification is performed on all measured images based on position and angle offset.

[0074] S309, map the spatial location model to the image coordinate system of the image under test to determine the corresponding position of the set of structural regions in the image under test.

[0075] The set of structural regions includes multiple structural regions at different heights within the object being measured.

[0076] S310, at the location corresponding to the set of structural regions, the sharpness of each tested image is evaluated, and the target focal length image that makes each structural region clear is determined.

[0077] S311, Select the target focal length image at the center as the reference image, and use the feature point matching method to calculate the transformation matrix of the non-reference image in the target focal length image relative to the reference image.

[0078] S312 performs geometric correction on the non-reference image based on the transformation matrix, so that all target focal length images are spatially aligned.

[0079] S313, based on the sharpness evaluation results of each target focal length image, assign fusion weights to each structural region in each target focal length image.

[0080] S314, based on the fusion weight, fuse the focal length images of each target to obtain the super depth-of-field image fusion.

[0081] The specific processes of S301-S314 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0082] Based on the same inventive concept, this application also provides a super-depth-of-field image fusion apparatus for implementing the super-depth-of-field image fusion method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more super-depth-of-field image fusion apparatus embodiments provided below can be found in the limitations of the super-depth-of-field image fusion method described above, and will not be repeated here.

[0083] In one embodiment, such as Figure 4 As shown, a super-depth-of-field image fusion apparatus is provided, the apparatus comprising: The sample acquisition module 40 is used to acquire several sample images of different focal lengths for each of the multiple test object samples of the same model during the system configuration phase. The region division module 41 is used to determine the distribution characteristics of different height structures of the tested object based on the sample image, divide the object into multiple structural regions that represent different height structures, and establish a spatial position model of the structural regions in the sample coordinate system of the tested object. The image acquisition module 42 is used to acquire several images of the object under test with different focal lengths when performing online detection on the object under test; The model mapping module 43 is used to locate the position of the object under test and to map the spatial position model to the image coordinate system of the image under test based on the location result, so as to determine the corresponding position of the set of structural regions in the image under test; the set of structural regions contains multiple structural regions at different heights in the object under test. The sharpness evaluation module 44 is used to evaluate the sharpness of each tested image at the location corresponding to the set of structural regions, and to determine the target focal length image that makes each structural region clear. The image fusion module 45 is used to perform image registration and image fusion processing on each target focal length image to generate a super depth-of-field image.

[0084] In another embodiment, the above Figure 4 The model mapping module 43 includes: The visual positioning unit is used to perform coarse visual positioning of the image under test and determine the position and angular offset of the object under test in the image under test. The model mapping unit is used to map the spatial position model to the image coordinate system of the image under test based on the position and angle offset.

[0085] In another embodiment, the visual positioning unit in the above embodiment is specifically used to: extract the contour of the object under test using an edge detection algorithm; and determine the position and angular offset of the object under test in the image under test based on the minimum bounding rectangle of each contour.

[0086] In another embodiment, the model mapping unit in the above embodiment is specifically used to: perform coordinate system matching on all measured images according to position and angle offset; and map the spatial position model to the image coordinate system of the measured image.

[0087] In another embodiment, the above Figure 4 The region division module 41 is specifically used to: divide the surface of the object under test into multiple structural regions representing different height structures based on the sharpness distribution characteristics of the sample images; for each structural region, if there are multiple structural regions whose corresponding sample images have the same sharpness, then the multiple structural regions are merged into one structural region.

[0088] In another embodiment, the above Figure 4 The image fusion module 45 includes: The matrix determination unit is used to select the target focal length image at the center position as the reference image, and to calculate the transformation matrix of the non-reference image in the target focal length image relative to the reference image using the feature point matching method. Image alignment units are used to perform geometric correction on non-reference images based on the transformation matrix, so that all target focal length images are spatially aligned. The image fusion unit is used to fuse the focal length images of each target to obtain a super depth-of-field image fusion.

[0089] In another embodiment, the image fusion unit in the above embodiment is specifically used to: assign fusion weights to each structural region in each target focal length image based on the sharpness evaluation results of each target focal length image; and fuse each target focal length image based on the fusion weights to obtain a super depth-of-field image fusion.

[0090] This application also provides an electronic device, in some embodiments, referring to... Figure 5As shown, the electronic device 700 includes an input unit 710, a memory 720, a processor 730, and an output unit 740. The memory 720 stores program instructions that can be executed on the processor 730. The processor 730 can execute the super-depth image fusion method and / or technical solution based on the foregoing embodiments by calling the program instructions. The electronic device 700 can be a mobile terminal device such as a mobile phone or a computer.

[0091] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program that performs a super-depth-of-field image fusion method. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions that invoke the methods of this application may be stored in a fixed or removable storage medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in a storage medium that operates according to the program instructions.

[0092] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0093] The technical features of the above embodiments can be arbitrarily integrated. For the sake of brevity, not all possible integrations of the technical features in the above embodiments are described. However, as long as the integration of these technical features does not contradict each other, they should be considered to be within the scope of this specification.

[0094] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A super-zoom image fusion method, characterized in that, The method includes: During the system configuration phase, for multiple test object samples of the same model, several sample images of each test object sample with different focal lengths are collected. Based on the sample images, the distribution characteristics of the structure at different heights of the tested object are determined, multiple structural regions representing different heights are divided, and a spatial position model of the structural regions in the sample coordinate system of the tested object is established. When performing online detection on the object under test, several test images of the object under test with different focal lengths are acquired; The object under test is located, and the spatial location model is mapped to the image coordinate system of the image under test based on the location result, so as to determine the corresponding position of the set of structural regions in the image under test; the set of structural regions includes multiple structural regions at different heights in the object under test. At the locations corresponding to the set of structural regions, the sharpness of each of the tested images is evaluated, and the target focal length image that makes each structural region clear is determined. Image registration and image fusion are performed on each of the target focal length images to generate a super depth-of-field image.

2. The super-zoom image fusion method of claim 1, wherein, The process includes locating the object under test and mapping the spatial location model to the image coordinate system of the image under test based on the location result, including: Visual coarse localization is performed on the image under test to determine the position and angular offset of the object under test in the image under test; The spatial position model is mapped to the image coordinate system of the measured image based on the position and angle offset.

3. The super-depth image fusion method as described in claim 2, characterized in that, Visual coarse localization is performed on the image under test to determine the position and angular offset of the object under test in the image under test, including: The contour of the object under test is extracted using an edge detection algorithm; Based on the minimum bounding rectangle of each of the aforementioned contours, the position and angular offset of the object under test in the image under test are determined.

4. The super-depth image fusion method as described in claim 2, characterized in that, Mapping the spatial position model to the image coordinate system of the measured image based on the position and angle offset includes: Based on the position and angle offset, coordinate system one is established for all measured images; The spatial location model is mapped to the image coordinate system of the image being measured.

5. The super-depth image fusion method as described in claim 1, characterized in that, Based on the sample images, the distribution characteristics of the structure at different heights of the tested object are determined, and multiple structural regions representing different heights are delineated, including: Based on the sharpness distribution characteristics of the sample images, the surface of the object under test is divided into multiple structural regions that characterize different height structures. If multiple structural regions have sample images with the same clarity, then these multiple structural regions are merged into one structural region.

6. The super-depth image fusion method as described in claim 1, characterized in that, Image registration and image fusion are performed on each of the target focal length images to generate a super depth-of-field image, including: The target focal length image at the center position is selected as the reference image, and the transformation matrix of the non-reference image in the target focal length image relative to the reference image is calculated using the feature point matching method; Geometric correction is performed on the non-reference image based on the transformation matrix to align the spatial images of all target focal lengths. The various target focal length images are fused to obtain the super depth-of-field image fusion.

7. The super-depth image fusion method as described in claim 6, characterized in that, The fusion of the various target focal length images to obtain the super-depth-of-field image fusion includes: Based on the sharpness evaluation results of each of the target focal length images, fusion weights are assigned to each structural region in each of the target focal length images; The target focal length images are fused based on the fusion weights to obtain the super depth-of-field image fusion.

8. A super-depth-of-field image fusion device, characterized in that, The device includes: The sample acquisition module is used during the system configuration phase to acquire several sample images of each test object sample with different focal lengths for multiple test object samples of the same model. The region division module is used to determine the distribution characteristics of different height structures of the tested object based on the sample image, divide the object into multiple structural regions that represent different height structures, and establish a spatial position model of the structural regions in the sample coordinate system of the tested object. The image acquisition module is used to acquire several test images of the test object at different focal lengths when performing online detection on the test object; The model mapping module is used to locate the object under test and map the spatial location model to the image coordinate system of the image under test based on the location result, so as to determine the corresponding position of the set of structural regions in the image under test; the set of structural regions includes multiple structural regions at different heights in the object under test. The sharpness evaluation module is used to evaluate the sharpness of each of the tested images at the locations corresponding to the set of structural regions, and to determine the target focal length image that makes each structural region clear. The image fusion module is used to perform image registration and image fusion processing on each of the target focal length images to generate a super depth-of-field image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the ultra-depth image fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the super-depth-of-field image fusion method according to any one of claims 1 to 7.

Images