Wafer image splicing method, device and equipment

Abstract

The invention relates to the technical field of wafer images, and discloses a wafer image splicing method, device and equipment, and the method comprises the steps: obtaining a plurality of microscopic images of a wafer according to a preset collection track and a preset collection number, and constructing a microscopic image data set; constructing a pyramid image corresponding to each microscopic image; for any two adjacent microscopic images, respectively extracting regions of interest of the two adjacent microscopic images based on the two pyramid images corresponding to the two adjacent microscopic images; in each microscopic image, Fourier-Mellin transformation is carried out on a region-of-interest image of the microscopic image, an offset parameter, a rotation parameter and a scaling parameter of each microscopic image are obtained, relative position coordinates of each microscopic image in the panoramic image of the wafer are calculated, and the size of the panoramic image is calculated; and based on the size of the panoramic image and the relative position coordinates of each microscopic image, fusing all adjacent microscopic images to obtain a fused panoramic image, and completing splicing of the wafer images.

Description

Technical Field

[0001] The present invention relates to the field of wafer image technology, and in particular to a wafer image stitching method, device and equipment. Background Art

[0002] The imaging of a microscope with a high-power objective lens is usually a small local field of view, resulting in a small amount of information in a single image, and the full picture of the sample cannot be obtained. Although a low-power objective lens can obtain a larger field of view information, it will lose the high-resolution detail features. Therefore, it is an urgent need in the industrial field to obtain panoramic information of the entire sample by stitching a series of adjacent high-resolution images. Before stitching multiple high-resolution images, the images need to be registered first. Common registration methods and defects are as follows:

[0003] (1) Traditional spatial domain registration methods, such as template matching, select appropriate similarity criteria based on the grayscale value of the image. The amount of calculation in this method increases as the size of the image increases, and the execution speed is slow, which cannot meet the speed requirements of registration. In addition, it is strongly affected by factors such as illumination and noise, and it is difficult to effectively suppress image geometric distortion.

[0004] (2) Feature-based registration methods are generally based on the unchanged geometric features in the image. By establishing a matching relationship between them, the transformation model parameter values ​​between the images to be registered are obtained. This type of method has little impact on illumination and noise, and has high accuracy and robustness. However, for wafer images to be spliced, due to the small overlap area, the number of features is small, and it often takes a lot of time to extract enough features. In addition, the periodicity of texture will directly affect the accuracy of feature matching, resulting in reduced precision and serious distortion. Therefore, this type of method is generally less effective in wafer image splicing.

[0005] (3) Registration methods based on transform domain, such as phase correlation method, are based on discrete Fourier transform of images to obtain image power spectrum and phase spectrum information. Since illumination and edge characteristics are distributed in the low-frequency and high-frequency parts of the spectrum, this type of method is robust to the influence of shooting environment to a certain extent. Since noise is uncorrelated with image signals, this type of algorithm can suppress the influence of noise on registration results. However, when the phase correlation method is used directly on images with small overlapping areas, the peak of the pulse function is weak, which seriously affects the offset result. Therefore, the general algorithm system will require the user to judge the size of the overlapping area in advance. However, for images with varying overlapping area ratios, this method is difficult to guarantee accuracy, and the phase correlation method cannot calculate the rotation angle and scaling of the image.

[0006] In summary, for wafer images in the semiconductor field, the wafer images to be stitched under a microscope generally have the characteristics of small overlap area, simple image texture, texture periodicity, and obvious brightness difference. When stitching a large number of images, serious cumulative errors will occur due to inaccurate registration results, which will directly lead to stitching failure. In addition, the existing wafer stitching method is easily affected by factors such as light and noise, and is highly dependent on fixed system parameters, making it difficult to achieve automatic stitching, and it is impossible to simultaneously guarantee the real-time and accuracy of wafer stitching, which leads to inaccurate images obtained by stitching. Summary of the invention

[0007] To this end, the technical problem to be solved by the present invention is to overcome the problems in the prior art that due to the disappearance of overlapping areas of wafer microscopic images and simple image texture, stitching of wafer microscopic images is prone to failure and the stitching effect is greatly affected by image noise.

[0008] In order to solve the above technical problems, the present invention provides a wafer image stitching method, comprising:

[0009] Acquire multiple microscopic images of the wafer according to a preset acquisition trajectory and a preset acquisition quantity, and construct a microscopic image data set;

[0010] For each microscopic image, the image itself is used as the bottom image, and the bottom image is scaled according to the preset number of pyramid layers and the preset scaling ratio corresponding to each layer, so as to construct a pyramid image corresponding to each microscopic image;

[0011] For any two adjacent microscopic images, based on their corresponding two pyramid images, the regions of interest of the two adjacent microscopic images are extracted respectively, including:

[0012] Initialize k=K, where K represents the total number of layers of the pyramid image, and the Kth layer image represents the top layer image of the pyramid image;

[0013] Within the preset ratio range, according to the kth preset step length, traverse and obtain the structural similarity of the kth layer image in the two pyramid images under multiple overlapping ratios, obtain the maximum value and the second maximum value of the structural similarity, and the corresponding first overlapping ratio and second overlapping ratio;

[0014] Based on the first overlap ratio and the second overlap ratio, updating the preset ratio range interval;

[0015] Let k=k-1, calculate the structural similarity of the k-th layer image at different overlapping ratios in the update preset ratio range, until k=1, obtain the maximum value of the first layer image structural similarity of the two pyramid images and its corresponding overlapping ratio;

[0016] Taking the overlap ratio of the first layer of the pyramid image as the optimal overlap ratio, the region of interest images are extracted from the corresponding two microscopic images respectively;

[0017] In each microscopic image, a Fourier-Mellin transform is performed on the image of the region of interest, the offset parameter, rotation parameter and scaling parameter of each microscopic image are obtained, and the relative position coordinates of each microscopic image in the panoramic image of the wafer are calculated;

[0018] Calculate the size of the panoramic image based on the size of the lower right corner image in the panoramic image and the maximum value of the horizontal coordinate and the maximum value of the vertical coordinate in all relative position coordinates;

[0019] Based on the size of the panoramic image and the relative position coordinates of each microscopic image, all adjacent microscopic images are fused to obtain a fused panoramic image, thereby completing the stitching of the wafer image.

[0020] Preferably, the structural similarity is expressed as:

[0021]

[0022] in, and Respectively represent two microscopic images to be registered; Representing images and The brightness relationship is expressed as Representing images and The contrast relationship is expressed as Representing images and The structural relationship of and Represent the microscopic images to be registered and The mean of and Represent the microscopic images to be registered and The variance of Represents the microscopic image to be registered and The covariance between them; c1, c2 and c3 are three constants used to prevent the denominator from being 0, respectively expressed as The default values ​​of k1 and k2 are 0.01 and 0.03. It is the image pixel value, usually 255.

[0023] Preferably, updating the preset ratio range based on the first overlap ratio and the second overlap ratio includes:

[0024] First overlap ratio As the middle point, construct the first range interval Indicates the preset interval length;

[0025] With the second overlap ratio As the middle point, construct the second range interval

[0026] If the first range interval does not overlap with the second range interval, a union of the first range interval and the second range interval is used as the updated preset ratio range interval;

[0027] If the first range interval overlaps with the second range interval, the first range interval is used as the updated preset ratio range interval.

[0028] Preferably, in each microscopic image, a Fourier-Mellin transform is performed on the image of the region of interest to obtain the offset parameter, rotation parameter and scaling parameter of each microscopic image, including:

[0029] The region of interest image r1(x, y) of any microscopic image is expressed as:

[0030] r1(x, y)=r2(λ(x cosΔθ+y sinΔθ)-Δx, λ(-x sinΔθ+y cosΔθ)-Δy);

[0031] Perform Fourier transform on the image of the region of interest to obtain the corresponding frequency domain image R1(u, v), which is expressed as:

[0032]

[0033] After high-pass filtering the frequency domain image, the amplitude spectrum of the frequency domain image is calculated It is expressed as:

[0034]

[0035] Perform log polar coordinate transformation on the amplitude spectrum, convert the rotation and scaling of the image in the region of interest into the translation in the log polar coordinate system, and obtain the amplitude spectrum in the log polar coordinate system. It is expressed as:

[0036]

[0037] Based on the amplitude spectrum in the logarithmic polar coordinate system, the polar angle direction coordinates and the polar longitude direction coordinates of the image of the region of interest are obtained by using the phase correlation method as the rotation parameter Δθ and the scaling parameter λ of the image of the region of interest;

[0038] The image of the region of interest is rotated and scaled based on the rotation parameter and scaling parameter of the image of the region of interest to obtain a corrected image; based on the amplitude spectrum of the corrected image, the offset parameters (Δx, Δy) of the image of the region of interest are extracted using a phase correlation method;

[0039] Wherein, r1(x, y) represents an image of a region of interest of a first microscopic image, and r2(x, y) represents an image of a region of interest of a second microscopic image adjacent to the first microscopic image.

[0040] Preferably, calculating the relative position coordinates of each microscopic image in the panoramic image of the wafer includes:

[0041] For each microscopic image in the panoramic image, the microscopic image is rotated based on its corresponding rotation parameter to obtain a corresponding rotated microscopic image;

[0042] The coordinates of the upper left corner of the rotating microscopic image in the first row and first column of the panoramic image (0, 0) are used as the zero point coordinates;

[0043] For each rotational microscopic image in the first column of the panoramic image, its region of interest (W) is subtracted from the rotational microscopic image in the previous row. ROI , H ROI ) and the coordinates of the lower left corner after the rotation and the corresponding offset parameters (Δx1, Δy1) are used as the coordinates of the upper left corner of the rotated microscopic image (Δx1, H ROI +Δy1);

[0044] For each rotation microscopic image in each row of the panoramic image except the first column, its region of interest (W ROI , H ROI ) and the corresponding offset parameters (Δx2, Δy2) as the coordinates of the upper left corner of the rotated microscopic image (W ROI +Δx2,Δy2).

[0045] Preferably, all adjacent microscopic images are fused using a distance-weighted fusion method to obtain a fused panoramic image.

[0046] Preferably, the fused panoramic image is expressed as:

[0047]

[0048] Wherein, f(x, y) represents the fused panoramic image, m(x, y) and n(x, y) represent two adjacent microscopic images, and α represents the distance weight set based on the overlapping area.

[0049] Preferably, the distribution of any two adjacent microscopic images includes: up-and-down distribution and left-and-right distribution.

[0050] This embodiment provides a wafer image stitching device, including:

[0051] A microscopic image acquisition module is used to acquire multiple microscopic images of the wafer according to a preset acquisition trajectory and a preset acquisition quantity, and to construct a microscopic image data set;

[0052] A pyramid image construction module is used for scaling the underlying image according to a preset number of pyramid layers and a preset scaling ratio corresponding to each layer for each microscopic image, so as to construct a pyramid image corresponding to each microscopic image;

[0053] The region of interest extraction module is used to extract the regions of interest of any two adjacent microscopic images based on their corresponding two pyramid images, including: initializing k=K, K represents the total number of layers of the pyramid image, and the K-th layer image represents the top layer image of the pyramid image; within the preset ratio range, according to the k-th preset step length, traversing to obtain the structural similarity of the k-th layer image in the two pyramid images under multiple overlapping ratios, and obtaining the maximum value and the second maximum value of the structural similarity, and the corresponding first overlapping ratio and second overlapping ratio; updating the preset ratio range based on the first overlapping ratio and the second overlapping ratio; setting k=k-1, calculating the structural similarity of the k-th layer image under different overlapping ratios in the updated preset ratio range, until k=1, obtaining the maximum value of the first layer image structural similarity of the two pyramid images and the corresponding overlapping ratio; taking the overlapping ratio of the first layer image of the pyramid image as the optimal overlapping area ratio, respectively extracting the region of interest images from the corresponding two microscopic images;

[0054] An image stitching module is used to perform Fourier-Mellin transformation on the image of the region of interest in each microscopic image, obtain the offset parameter, rotation parameter and scaling parameter of each microscopic image, and calculate the relative position coordinates of each microscopic image in the panoramic image of the wafer; calculate the size of the panoramic image based on the size of the lower right corner image in the panoramic image and the maximum values ​​of the abscissa and ordinate in all relative position coordinates;

[0055] The fused panoramic image acquisition module is used to fuse all adjacent microscopic images based on the size of the panoramic image and the relative position coordinates of each microscopic image, obtain a fused panoramic image, and complete the stitching of the wafer image.

[0056] This embodiment provides a wafer image stitching device, including:

[0057] A collection device, used for collecting multiple microscopic images of the wafer based on a preset collection trajectory and a preset collection quantity;

[0058] A wafer image stitching device is connected to the acquisition device for acquiring a microscopic image of the wafer and using the wafer image stitching method as described above to acquire a fused panoramic image;

[0059] A display is communicatively connected to the wafer image stitching device and is used to display the fused panoramic image.

[0060] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0061] The wafer image stitching method of the present invention constructs a microscopic image of a wafer image into a pyramid image, optimizes the overlapping ratio layer by layer by using the pyramid image and the structural similarity, and updates the preset ratio range interval, so as to quickly search for the overlapping area between any two adjacent microscopic images, accelerate the extraction of the region of interest, and can prevent the misjudgment caused by the texture periodicity of the wafer image, and further improve the accuracy of image registration; the present invention uses Fourier-Mellin transformation to convert the image of the region of interest from the spatial domain to the frequency domain and the logarithmic polar coordinate system, and in the logarithmic polar coordinates, the phase spectrum is used to quickly calculate the offset parameters, rotation parameters and scaling parameters between two adjacent microscopic images, adjust the microscopic image, weaken the influence of noise on the image, and reduce the influence of illumination changes in the low-frequency region on the image through high-pass filtering, thereby improving the accuracy of the stitched fused panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0063] Figure 1 is a flowchart of the steps of the wafer image stitching method provided by the present invention;

[0064] Figure 2 It is a flowchart of obtaining the overlap ratio provided by the present invention;

[0065] Figure 3 It is a schematic diagram of the distribution mode of adjacent images provided by the present invention;

[0066] Figure 4 It is a schematic diagram of the relative position coordinate calculation principle provided by the present invention. DETAILED DESCRIPTION

[0067] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0068] Reference Figure 1As shown, the step flow chart of the wafer image stitching method provided by the present invention includes the following steps:

[0069] S101: Acquire multiple microscopic images of the wafer according to a preset acquisition trajectory and a preset acquisition quantity, and construct a microscopic image data set;

[0070] S102: for each microscopic image, use itself as the bottom image, scale the bottom image according to the preset number of pyramid layers and the preset scaling ratio corresponding to each layer, and construct a pyramid image corresponding to each microscopic image;

[0071] S103: for any two adjacent microscopic images, based on their corresponding two pyramid images, respectively extracting the regions of interest of the two adjacent microscopic images, including:

[0072] S103-1: Initialize k=K, where K represents the total number of layers of the pyramid image, and the Kth layer image represents the top layer image of the pyramid image;

[0073] S103-2: within a preset ratio range, traverse and obtain the structural similarities of the k-th layer image in the two pyramid images at multiple overlapping ratios according to the k-th preset step length, and obtain the maximum value and the second maximum value of the structural similarity, and the corresponding first overlapping ratio and second overlapping ratio;

[0074] S103-3: updating a preset ratio range based on the first overlap ratio and the second overlap ratio;

[0075] S103-4: Let k=k-1, calculate the structural similarity of the k-th layer image at different overlapping ratios in the update preset ratio range, until k=1, obtain the maximum value of the first layer image structural similarity of the two pyramid images and the corresponding overlapping ratio;

[0076] S103-5: taking the overlap ratio of the first layer of the pyramid image as the optimal overlap ratio, respectively extracting the region of interest images from the corresponding two microscopic images;

[0077] S104: in each microscopic image, perform Fourier-Mellin transformation on the image of the region of interest, obtain the offset parameter, rotation parameter and scaling parameter of each microscopic image, and calculate the relative position coordinates of each microscopic image in the panoramic image of the wafer;

[0078] S105: Calculating the size of the panoramic image based on the size of the lower right corner image in the panoramic image and the maximum value of the horizontal coordinate and the maximum value of the vertical coordinate in all relative position coordinates;

[0079] S106: Based on the size of the panoramic image and the relative position coordinates of each microscopic image, all adjacent microscopic images are fused to obtain a fused panoramic image, thereby completing the stitching of the wafer image.

[0080] Specifically, this embodiment calculates the structural similarity between two images based on the brightness, contrast and structural relationship of the images, which is expressed as:

[0081]

[0082] in, and Respectively represent two microscopic images to be registered; Representing images and The brightness relationship is expressed as Representing images and The contrast relationship is expressed as Representing images and The structural relationship of and Represent the microscopic images to be registered and The mean of and Represent the microscopic images to be registered and The variance of Represents the microscopic image to be registered and The covariance between them; c1, c2 and c3 are three constants used to prevent the denominator from being 0, respectively expressed as The default values ​​of k1 and k2 are 0.01 and 0.03. It is the image pixel value, usually 255.

[0083] Specifically, in step S103-3, this implementation updates the preset ratio range based on the first overlap ratio and the second overlap ratio, including:

[0084] First overlap ratio As the middle point, construct the first range interval Indicates the preset interval length;

[0085] With the second overlap ratio As the middle point, construct the second range interval

[0086] If the first range interval does not overlap with the second range interval, a union of the first range interval and the second range interval is used as the updated preset ratio range interval;

[0087] If the first range interval overlaps with the second range interval, the first range interval is used as the updated preset ratio range interval.

[0088] In this embodiment, if the first overlap ratio is 0.15 and the second overlap ratio is 0.1, both are returned, and a fine search is performed near 0.15 and 0.1 to find the maximum value when searching at the next level; if the first overlap ratio is 0.13 and the second overlap ratio is 0.12, then only 0.13 needs to be returned; by updating the returned preset ratio range, time can be saved and misjudgment caused by the periodicity of wafer image texture can be prevented.

[0089] Specifically, in step S104, in each microscopic image, the image of the region of interest is subjected to Fourier-Mellin transformation to obtain the offset parameters, rotation parameters and scaling parameters of each microscopic image, including:

[0090] S104-1: The region of interest image r1(x, y) of any microscopic image is expressed as:

[0091] r1(x, y)=r2(λ(x cosΔθ+y sinΔθ)-Δx, y(-x sinΔθ+y cosΔθ)-Δy);

[0092] S104-2: Perform Fourier transform on the image of the region of interest to obtain the corresponding frequency domain image R1(u, v), which is expressed as:

[0093]

[0094] S104-3: After high-pass filtering the frequency domain image, calculate the amplitude spectrum of the frequency domain image It is expressed as:

[0095]

[0096] S104-4: Perform log polar coordinate conversion on the amplitude spectrum, convert the rotation and scaling of the image in the region of interest into the translation in the log polar coordinate system, and obtain the amplitude spectrum in the log polar coordinate system It is expressed as:

[0097]

[0098] S104-5: Based on the amplitude spectrum in the logarithmic polar coordinate system, using the phase correlation method, the polar angle direction coordinates and the polar longitude direction coordinates of the image of the region of interest are obtained as the rotation parameter Δθ and the scaling parameter λ of the image of the region of interest;

[0099] S104-6: rotating and scaling the image of the region of interest based on the rotation parameter and scaling parameter of the image of the region of interest to obtain a corrected image; extracting the offset parameters (Δx, Δy) of the image of the region of interest using a phase correlation method based on the amplitude spectrum of the corrected image;

[0100] Wherein, r1(x, y) represents an image of a region of interest of a first microscopic image, and r2(x, y) represents an image of a region of interest of a second microscopic image adjacent to the first microscopic image.

[0101] Specifically, in step S104, after obtaining the offset parameter, rotation parameter and scaling parameter of each microscopic image, the relative position coordinates of each microscopic image in the panoramic image of the wafer are calculated, including:

[0102] S104-7: rotating each microscopic image in the panoramic image based on its corresponding rotation parameter to obtain a corresponding rotated microscopic image;

[0103] S104-8: taking the coordinates (0, 0) of the upper left corner of the rotating microscopic image in the first row and first column of the panoramic image as the zero point coordinates;

[0104] S104-9: For each rotating microscopic image in the first column of the panoramic image, subtract its region of interest (W) from the rotating microscopic image in the previous row. ROI , H ROI ) and the coordinates of the lower left corner after the rotation and the corresponding offset parameters (Δx1, Δy1) are used as the coordinates of the upper left corner of the rotated microscopic image (Δx1, H ROI +Δy1);

[0105] S104-10: For each rotating microscopic image in each row of the panoramic image except the first column, subtract the region of interest (W ROI , H ROI ) and the corresponding offset parameters (Δx2, Δy2) as the coordinates of the upper left corner of the rotated microscopic image (W ROI +Δx2,Δy2).

[0106] Specifically, in step S105, the length and width of the lower right corner image in the panoramic image are added to the maximum values ​​of the abscissa and ordinate in the relative position coordinates, respectively, as the length and width of the panoramic image, to obtain the size of the panoramic image.

[0107] Specifically, in step S106, all adjacent microscopic images are fused using a distance-weighted fusion method to obtain a fused panoramic image, which is expressed as:

[0108]

[0109] Wherein, f(x, y) represents the fused panoramic image, m(x, y) and n(x, y) represent two adjacent microscopic images, and α represents the distance weight set based on the overlapping area.

[0110] The wafer image stitching method of the present invention constructs a microscopic image of a wafer image into a pyramid image, optimizes the overlapping ratio layer by layer by using the pyramid image and the structural similarity, and updates the preset ratio range interval, so as to quickly search for the overlapping area between any two adjacent microscopic images, accelerate the extraction of the region of interest, and can prevent the misjudgment caused by the texture periodicity of the wafer image, and further improve the accuracy of image registration; the present invention uses Fourier-Mellin transformation to convert the image of the region of interest from the spatial domain to the frequency domain and the logarithmic polar coordinate system, and in the logarithmic polar coordinates, the phase spectrum is used to quickly calculate the offset parameters, rotation parameters and scaling parameters between two adjacent microscopic images, adjust the microscopic image, weaken the influence of noise on the image, and reduce the influence of illumination changes in the low-frequency region on the image through high-pass filtering, thereby improving the accuracy of the stitched fused panoramic image.

[0111] Based on the above embodiment, in an embodiment of the present invention, the wafer image stitching method provided by the present invention is used to stitch wafer images, and the specific steps include:

[0112] S201: Acquire wafer images according to set parameters such as a preset acquisition trajectory and a preset acquisition quantity, and make corresponding containers according to these parameters to store wafer image data sets;

[0113] S202: Using the pyramid image and structural similarity to determine the overlapping area of ​​the images to be stitched, refer to Figure 2 As shown in FIG. 1 , it is a flowchart for obtaining the overlap ratio, and the specific steps include:

[0114] S202-1: setting the original image as the bottom layer of the pyramid, scaling the bottom layer image according to the determined number of pyramid layers and scaling ratio, and gradually obtaining a pyramid image;

[0115] By obtaining the pyramid image corresponding to each image in the image dataset, the amount of calculation is reduced and the algorithm is accelerated.

[0116] S202-2: Obtain two pyramid top-level images corresponding to adjacent images to be registered, and traverse different overlapping ratios within a preset ratio range according to a k-th preset step size;

[0117] In the preset ratio range, the bottom image generally does not exceed 20%, and the top image is reduced proportionally; for example, when calculating between the k-th layer images, the step size is set to 0.01, and the structural similarities with different overlapping ratios in the range of 0 to 0.2 (the 0 to 20% interval here is preset) are traversed, and the ratio corresponding to the maximum value and the second maximum value is returned; at the k-1 layer, the ratio of the maximum value and the second maximum value returned in the upper layer is traversed with a step size of 0.005, and the same applies to the others.

[0118] Reference Figure 3 As shown, it is a schematic diagram of the distribution of adjacent images; when the two images to be registered are distributed left and right, the right boundary of the left image is the starting position, and the left boundary of the right image is the starting position, and the corresponding sub-images are obtained by smoothing according to different proportions; similarly, when the two images to be registered are distributed up and down, the lower boundary of the upper image is the starting position, and the upper boundary of the lower image is the starting position, and the corresponding sub-images are obtained by smoothing according to different proportions;

[0119] Calculate the structural similarity corresponding to each ratio respectively, retain the maximum and second maximum values, and their corresponding overlap ratios;

[0120] The calculation formula of structural similarity is expressed as:

[0121] Image brightness:

[0122] Image Contrast:

[0123] Image structure:

[0124] Structural similarity:

[0125] Among them, x and y represent two adjacent images to be registered, u x with u y Respectively represent the mean of the two images to be registered, σ x With σ y Respectively represent the variance of the two images to be registered, σ xy represents the covariance between the two images to be registered; c1, c2 and c3 are three constants used to prevent the denominator from being 0, respectively expressed as The default values ​​of k1 and k2 are 0.01 and 0.03. It is the image pixel value, usually 255.

[0126] S202-3: Enter the next layer of the pyramid, and according to the maximum and second maximum values ​​of the structural similarity obtained in the previous step, perform a fine search with a small step size in a small range of the corresponding overlapping ratio to obtain the maximum and second maximum values ​​of the structural similarity of the current layer, and their corresponding overlapping area ratios;

[0127] S202-4: Determine whether the obtained overlap area ratios corresponding to the maximum and second maximum values ​​of the structural similarity are within the same interval;

[0128] If yes, the corresponding ratios are considered to be the same value, and only the maximum value and its corresponding overlapping area ratio are returned;

[0129] If not, then return the maximum and second maximum values ​​and their corresponding overlapping area ratios;

[0130] S202-5: Determine whether the current layer is the bottom layer of the pyramid. If yes, directly output the overlapping area ratio result corresponding to the maximum value. If no, return to step S202-3.

[0131] S203: Based on the overlap area ratio corresponding to the maximum value of the bottom layer of the pyramid, the Fourier-Mellin transform is used to calculate the offset, angle and size difference between the adjacent images to be stitched; the scale difference between the wafer images is almost zero, so it can be directly ignored in subsequent calculations. The specific steps include:

[0132] S203-1: extracting a ROI sub-image from the image to be registered according to the optimal overlapping area ratio;

[0133] S203-2: Perform Fourier transform on each ROI sub-image to convert it from the spatial domain to the frequency domain;

[0134] S203-3: Use high-pass filtering to retain the edge, filter out factors such as illumination, and calculate the amplitude spectrum;

[0135] S203-4: Perform a polar-pair coordinate system transformation on the amplitude spectrum, and convert the rotation and scaling of the ROI sub-image into a translation in the coordinate system;

[0136] S203-5: Calculate rotation and scaling parameters using a phase correlation method, and adjust the ROI sub-image;

[0137] S203-6: Calculate the offset parameter using the phase correlation method for the adjusted image;

[0138] Specifically, the Fourier-Mellin transform principle is: Assume that the ROI images in the above steps are r1(x, y) and r2(x, y), and the translation, rotation and scaling parameters between the two are (Δx, Δy), Δθ and λ, that is:

[0139] r1(x, y)=r2(λ(x cosΔθ+y sinΔθ)-Δx, λ(-x sinΔθ+y cosΔθ)-Δy);

[0140] After Fourier transform, it is expressed as:

[0141]

[0142] Calculate the amplitude spectrum, expressed as:

[0143]

[0144] Finally, the amplitude spectrum is transformed into polar coordinates, which is expressed as:

[0145]

[0146] S204: Estimate the size of the stitched large image and update the pose of the small image;

[0147] Reference Figure 4 The figure below is a schematic diagram of the relative position coordinate calculation principle; based on the translation parameters and rotation parameters, the relative position of each small image in the stitched large image is calculated. First, the zero point of the upper left corner of the first row and first column image is used as the starting coordinate. The image in the first row and second column is rotated at an angle, and then its upper left corner coordinate is set to the upper right corner coordinate of the first row and first column image minus the above ROI area plus the calculated translation coordinate. The same is true for the remaining images in the first row.

[0148] After adjusting the rotation angle, the upper left corner coordinates of the second row and first column picture are set to the lower left corner coordinates of the first row and first column picture minus the above ROI area plus the calculated translation coordinates, and the same applies to the remaining pictures;

[0149] Based on this, the relative coordinates of all updated pose images are obtained, and then the lower right corner image is used to calculate the size of the entire image; if there are negative values ​​in the relative coordinates, all relative coordinates are updated to retain some pixels corresponding to the negative values.

[0150] S205: Image fusion based on distance weighting;

[0151] The corresponding panoramic picture is generated by using the relative coordinates and the full picture size in step S204, and then adjacent pictures are fused.

[0152] In the process of fusion of two images, they are divided into overlapping areas and non-overlapping areas. For the images starting from the second row and the second column, the overlapping areas not only overlap left and right but also overlap up and down when fusion is performed. Therefore, when using the distance-weighted fusion method, the weights need to be considered separately. The mathematical expression of image fusion is as follows:

[0153]

[0154] Among them, f(x, y) represents the fused panoramic image, m(x, y) and n(x, y) represent two adjacent microscopic images; α represents the distance weight set based on the overlapping area, which needs to be calculated separately according to the width of the left and right overlapping areas and the height of the upper and lower overlapping areas.

[0155] S206: The automatic stitching system ends and outputs the fused panoramic image.

[0156] The present invention uses pyramid images and the structural similarity SSIM (Structural Similarity Image Measurement) algorithm to quickly find the overlapping areas between adjacent images, improve the accuracy of image registration, and achieve automatic stitching; at the same time, the Fourier-Mellin transform is used to convert the image from the spatial domain to the frequency domain and the logarithmic polar coordinate system, and the position, angle and scale differences between adjacent images to be stitched are quickly calculated through the phase spectrum. Due to the irrelevance between noise and image signals, the phase spectrum can weaken the influence of noise, and reduce the influence of illumination changes in low-frequency areas on the system through high-pass filtering. The embodiment of the present invention improves the real-time performance of the stitching algorithm while ensuring accuracy, reduces the influence of factors such as illumination and noise on the stitching system, and reduces the dependence on fixed system parameters, thereby achieving automatic stitching of wafer images.

[0157] Based on the above embodiments, an embodiment of the present invention further provides a wafer image stitching device, including:

[0158] A microscopic image acquisition module is used to acquire multiple microscopic images of the wafer according to a preset acquisition trajectory and a preset acquisition quantity, and to construct a microscopic image data set;

[0159] A pyramid image construction module is used for scaling the underlying image according to a preset number of pyramid layers and a preset scaling ratio corresponding to each layer for each microscopic image, so as to construct a pyramid image corresponding to each microscopic image;

[0160] The region of interest extraction module is used to extract the regions of interest of any two adjacent microscopic images based on their corresponding two pyramid images, including: initializing k=K, K represents the total number of layers of the pyramid image, and the K-th layer image represents the top layer image of the pyramid image; within the preset ratio range, according to the k-th preset step length, traversing to obtain the structural similarity of the k-th layer image in the two pyramid images under multiple overlapping ratios, and obtaining the maximum value and the second maximum value of the structural similarity, and the corresponding first overlapping ratio and second overlapping ratio; updating the preset ratio range based on the first overlapping ratio and the second overlapping ratio; setting k=k-1, calculating the structural similarity of the k-th layer image under different overlapping ratios in the updated preset ratio range, until k=1, obtaining the maximum value of the first layer image structural similarity of the two pyramid images and the corresponding overlapping ratio; taking the overlapping ratio of the first layer image of the pyramid image as the optimal overlapping area ratio, respectively extracting the region of interest images from the corresponding two microscopic images;

[0161] An image stitching module is used to perform Fourier-Mellin transformation on the image of the region of interest in each microscopic image, obtain the offset parameter, rotation parameter and scaling parameter of each microscopic image, and calculate the relative position coordinates of each microscopic image in the panoramic image of the wafer; calculate the size of the panoramic image based on the size of the lower right corner image in the panoramic image and the maximum values ​​of the abscissa and ordinate in all relative position coordinates;

[0162] The fused panoramic image acquisition module is used to fuse all adjacent microscopic images based on the size of the panoramic image and the relative position coordinates of each microscopic image, obtain a fused panoramic image, and complete the stitching of the wafer image.

[0163] The wafer image stitching device of this embodiment is used to implement the aforementioned wafer image stitching method. Therefore, the specific implementation method of the wafer image stitching device can be seen in the embodiment part of the wafer image stitching method in the previous text. For example, the microscopic image acquisition module, the pyramid image construction module, and the region of interest extraction module are respectively used to implement steps S101, S102 and S103 in the above-mentioned wafer image stitching method; the image stitching module is used to implement steps S104 and S105 in the above-mentioned wafer image stitching method; the fused panoramic image acquisition module is used to implement step S106 in the above-mentioned wafer image stitching method. Therefore, its specific implementation method can refer to the description of the corresponding various parts of the embodiment, which will not be repeated here.

[0164] Based on the above embodiments, an embodiment of the present invention further provides a wafer image stitching device, including:

[0165] A collection device, used for collecting multiple microscopic images of the wafer based on a preset collection trajectory and a preset collection quantity;

[0166] A wafer image stitching device is connected to the acquisition device for acquiring a microscopic image of the wafer and using the wafer image stitching method as described above to acquire a fused panoramic image;

[0167] A display is communicatively connected to the wafer image stitching device and is used to display the fused panoramic image.

[0168] Specifically, in this embodiment, the acquisition device is a high-resolution camera, so as to obtain high-resolution wafer microscopic images based on a preset acquisition trajectory and a preset acquisition quantity for subsequent stitching.

[0169] The wafer image stitching method, device and equipment of the present invention construct the microscopic image of the wafer image into a pyramid image, optimize the overlapping ratio layer by layer by using the pyramid image and structural similarity, and update the preset ratio range interval, so as to quickly search for the overlapping area between any two adjacent microscopic images, accelerate the extraction of the region of interest, and prevent the misjudgment caused by the texture periodicity of the wafer image, and further improve the accuracy of image registration; the present invention uses Fourier-Mellin transformation to convert the image of the region of interest from the spatial domain to the frequency domain and the logarithmic polar coordinate system, and in the logarithmic polar coordinates, the phase spectrum is used to quickly calculate the offset parameters, rotation parameters and scaling parameters between the two adjacent microscopic images, adjust the microscopic image, weaken the influence of noise on the image, and reduce the influence of illumination changes in the low-frequency area on the image through high-pass filtering, thereby improving the accuracy of the stitched fused panoramic image.

[0170] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0174] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A wafer image stitching method, characterized in that: include: Acquire multiple microscopic images of the wafer according to a preset acquisition trajectory and a preset acquisition quantity, and construct a microscopic image data set; For each microscopic image, the image itself is used as the bottom image, and the bottom image is scaled according to the preset number of pyramid layers and the preset scaling ratio corresponding to each layer, so as to construct a pyramid image corresponding to each microscopic image; For any two adjacent microscopic images, based on their corresponding two pyramid images, the regions of interest of the two adjacent microscopic images are extracted respectively, including: Initialize k=K, where K represents the total number of layers of the pyramid image, and the Kth layer image represents the top layer image of the pyramid image; Within the preset ratio range, according to the kth preset step length, traverse and obtain the structural similarity of the kth layer image in the two pyramid images under multiple overlapping ratios, obtain the maximum value and the second maximum value of the structural similarity, and the corresponding first overlapping ratio and second overlapping ratio; Based on the first overlap ratio and the second overlap ratio, updating the preset ratio range interval; Let k=k-1, calculate the structural similarity of the k-th layer image at different overlapping ratios in the update preset ratio range, until k=1, obtain the maximum value of the first layer image structural similarity of the two pyramid images and its corresponding overlapping ratio; Taking the overlap ratio of the first layer of the pyramid image as the optimal overlap ratio, the region of interest images are extracted from the corresponding two microscopic images respectively; In each microscopic image, a Fourier-Mellin transform is performed on the image of the region of interest, the offset parameter, rotation parameter and scaling parameter of each microscopic image are obtained, and the relative position coordinates of each microscopic image in the panoramic image of the wafer are calculated; Calculate the size of the panoramic image based on the size of the lower right corner image in the panoramic image and the maximum value of the horizontal coordinate and the maximum value of the vertical coordinate in all relative position coordinates; Based on the size of the panoramic image and the relative position coordinates of each microscopic image, all adjacent microscopic images are fused to obtain a fused panoramic image, thereby completing the stitching of the wafer image.

2. The wafer image stitching method according to claim 1, characterized in that: Structural similarity, expressed as: in, and Respectively represent two microscopic images to be registered; Representing images and The brightness relationship is expressed as Representing images and The contrast relationship is expressed as Representing images and The structural relationship of and Represent the microscopic images to be registered and The mean of and Represent the microscopic images to be registered and The variance of Represents the microscopic image to be registered and The covariance between them; c1, c2 and c3 are three constants used to prevent the denominator from being 0, respectively expressed as The default values ​​of k1 and k2 are 0.01 and 0.

03. It is the image pixel value, usually 255.

3. The wafer image stitching method according to claim 1, characterized in that: The updating of the preset ratio range based on the first overlap ratio and the second overlap ratio includes: First overlap ratio As the middle point, construct the first range interval Indicates the preset interval length; With the second overlap ratio As the middle point, construct the second range interval If the first range interval does not overlap with the second range interval, a union of the first range interval and the second range interval is used as the updated preset ratio range interval; If the first range interval overlaps with the second range interval, the first range interval is used as the updated preset ratio range interval.

4. The wafer image stitching method according to claim 1, characterized in that ,In each microscopic image, the image of the region of interest is subjected to Fourier-Mellin transformation to obtain the offset parameters, rotation parameters and scaling parameters of each microscopic image, including: The region of interest image r1(x, y) of any microscopic image is expressed as: r1(x, y)=r2(λ(xcosΔθ+ysinΔθ)-Δx, λ(-xsinΔθ+ycosΔθ)-Δy); Perform Fourier transform on the image of the region of interest to obtain the corresponding frequency domain image R1(u, v), which is expressed as: After high-pass filtering the frequency domain image, the amplitude spectrum of the frequency domain image is calculated It is expressed as: Perform log polar coordinate transformation on the amplitude spectrum, convert the rotation and scaling of the image in the region of interest into the translation in the log polar coordinate system, and obtain the amplitude spectrum in the log polar coordinate system. It is expressed as: Based on the amplitude spectrum in the logarithmic polar coordinate system, the polar angle direction coordinates and the polar longitude direction coordinates of the image of the region of interest are obtained by using the phase correlation method as the rotation parameter Δθ and the scaling parameter λ of the image of the region of interest; The image of the region of interest is rotated and scaled based on the rotation parameter and scaling parameter of the image of the region of interest to obtain a corrected image; based on the amplitude spectrum of the corrected image, the offset parameters (Δx, Δy) of the image of the region of interest are extracted using a phase correlation method; Wherein, r1(x, y) represents an image of a region of interest of a first microscopic image, and r2(x, y) represents an image of a region of interest of a second microscopic image adjacent to the first microscopic image.

5. The wafer image stitching method according to claim 4, characterized in that: Calculate the relative position coordinates of each microscopic image in the panoramic image of the wafer, including: For each microscopic image in the panoramic image, the microscopic image is rotated based on its corresponding rotation parameter to obtain a corresponding rotated microscopic image; The coordinates of the upper left corner of the rotating microscopic image in the first row and first column of the panoramic image (0, 0) are used as the zero point coordinates; For each rotational microscopic image in the first column of the panoramic image, its region of interest (W) is subtracted from the rotational microscopic image in the previous row. ROI , H ROI ) and the coordinates of the lower left corner after the rotation and the corresponding offset parameters (Δx1, Δy1) are used as the coordinates of the upper left corner of the rotated microscopic image (Δx1, H ROI +Δy1); For each rotation microscopic image in each row of the panoramic image except the first column, its region of interest (W ROI , H ROI ) and the corresponding offset parameters (Δx2, Δy2) as the coordinates of the upper left corner of the rotated microscopic image (W ROI +Δx2,Δy2).

6. The wafer image stitching method according to claim 1, characterized in that: All adjacent microscopic images are fused using a distance-weighted fusion method to obtain a fused panoramic image.

7. The wafer image stitching method according to claim 6, characterized in that: The fused panoramic image is expressed as: Wherein, f(x, y) represents the fused panoramic image, m(x, y) and n(x, y) represent two adjacent microscopic images, and α represents the distance weight set based on the overlapping area.

8. The wafer image stitching method according to claim 1, characterized in that: The distribution modes of any two adjacent microscopic images include: up-and-down distribution and left-and-right distribution.

9. A wafer image stitching device, characterized in that: include: A microscopic image acquisition module is used to acquire multiple microscopic images of the wafer according to a preset acquisition trajectory and a preset acquisition quantity, and to construct a microscopic image data set; A pyramid image construction module is used for scaling the underlying image according to a preset number of pyramid layers and a preset scaling ratio corresponding to each layer for each microscopic image, so as to construct a pyramid image corresponding to each microscopic image; The region of interest extraction module is used to extract the regions of interest of any two adjacent microscopic images based on their corresponding two pyramid images, including: initializing k=K, K represents the total number of layers of the pyramid image, and the K-th layer image represents the top layer image of the pyramid image; within the preset ratio range, according to the k-th preset step length, traversing to obtain the structural similarity of the k-th layer image in the two pyramid images under multiple overlapping ratios, and obtaining the maximum value and the second maximum value of the structural similarity, and the corresponding first overlapping ratio and second overlapping ratio; updating the preset ratio range based on the first overlapping ratio and the second overlapping ratio; setting k=k-1, calculating the structural similarity of the k-th layer image under different overlapping ratios in the updated preset ratio range, until k=1, obtaining the maximum value of the first layer image structural similarity of the two pyramid images and the corresponding overlapping ratio; taking the overlapping ratio of the first layer image of the pyramid image as the optimal overlapping area ratio, respectively extracting the region of interest images from the corresponding two microscopic images; An image stitching module is used to perform Fourier-Mellin transformation on the image of the region of interest in each microscopic image, obtain the offset parameter, rotation parameter and scaling parameter of each microscopic image, and calculate the relative position coordinates of each microscopic image in the panoramic image of the wafer; calculate the size of the panoramic image based on the size of the lower right corner image in the panoramic image and the maximum values ​​of the abscissa and ordinate in all relative position coordinates; The fused panoramic image acquisition module is used to fuse all adjacent microscopic images based on the size of the panoramic image and the relative position coordinates of each microscopic image, obtain a fused panoramic image, and complete the stitching of the wafer image.

10. A wafer image stitching device, characterized in that: include: A collection device, used for collecting multiple microscopic images of the wafer based on a preset collection trajectory and a preset collection quantity; A wafer image stitching device, which is in communication with the acquisition device, obtains a microscopic image of the wafer, and obtains a fused panoramic image using the wafer image stitching method according to any one of claims 1 to 8; A display is communicatively connected to the wafer image stitching device and is used to display the fused panoramic image.

Images