Image processing method, system, electronic device and storage medium
By adjusting the integrated stages of the TDI camera and registering the image according to the surface scattering rate of the semiconductor product, the problem of inconsistent exposure of mechanical splicing TDI cameras in different areas is solved, and the accuracy and consistency of defect detection are improved.
Patent Information
- Application Number
- CN202411035315.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-31
AI Technical Summary
When using multiple TDI cameras to detect defects in semiconductor products, multiple mechanically spliced TDI sensors are prone to overexposed or underexposed when shooting different areas of the product, resulting in inconsistent exposure of images and affecting the accuracy of defect detection.
By adjusting the number of integration steps of the time delay integration camera corresponding to each area according to the scattering rate parameters of the target object, the exposure degree of the area image is controlled so that it is as consistent as possible, and the acquired area image is used during the data processing stage. Register and splice to form a complete defect detection image.
The consistency of exposure degree of images in different regions is achieved, the accuracy of subsequent defect detection results is improved, the problem of excessive exposure differences is alleviated, and the reliability of defect detection results of semiconductor products is ensured.
Smart Images

Figure CN118590764B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and in particular relates to an image processing method, system, electronic device and storage medium. Background Art
[0002] Time Delay Integration (TDI) is a special image acquisition technology. TDI cameras designed based on TDI technology can significantly increase signal strength (especially in low light conditions) when acquiring images and improve image signal-to-noise ratio. Due to the above characteristics, TDI cameras can be used for defect detection in semiconductor-related products (such as wafers and masks).
[0003] When using TDI cameras for defect inspection of semiconductor products, in order to match higher magnification and equipment yield requirements, multiple TDI cameras can be spliced together for image acquisition to obtain a larger imaging field of view.
[0004] There are two types of stitching: sensor stitching and mechanical stitching. Sensor stitching refers to stitching the TDI sensors of multiple TDI cameras to form a larger TDI sensor. Multiple TDI sensors share a circuit to output data. When acquiring images in this stitching method, the acquisition speed will be limited by the data transmission speed. Mechanical stitching is to directly fix multiple TDI sensors and their respective output circuits into a whole through a mechanical structure. Each TDI sensor uses its own output circuit to output data.
[0005] However, multiple mechanically spliced TDI sensors are prone to overexposure or underexposure when photographing different areas of a product. Summary of the invention
[0006] To this end, this application discloses the following technical solutions:
[0007] The first aspect of the present application provides an image processing method, comprising:
[0008] Obtaining scattering rate parameters of multiple regions of the target object;
[0009] According to the scattering rate parameters of each of the regions, the integration level of the time-delayed integration camera corresponding to each of the regions when collecting images is configured, and each of the time-delayed integration cameras is used to collect a regional image of the corresponding region;
[0010] The regional images captured by each of the time-delayed integration cameras are combined and processed to obtain a defect detection image of the target object.
[0011] Optionally, the manner in which each of the time-delayed integration cameras collects the regional image includes:
[0012] Determining a trigger shooting electrical signal synchronized with the moving speed of the target object;
[0013] Sending the trigger shooting electrical signal to each of the time delay integration cameras through a trigger signal circuit to trigger each of the time delay integration cameras to obtain an image signal of the target object;
[0014] The image signal output by each of the time-delayed integration cameras is modulated according to the trigger shooting electrical signal to obtain the regional image.
[0015] Optionally, configuring the integration level of the time-delayed integration camera corresponding to each of the regions when acquiring images according to the scattering rate parameters of each of the regions includes:
[0016] For each of the regions, the integration order of the time-delayed integration camera corresponding to the region when acquiring images is configured according to a preset inverse relationship and a scattering rate parameter of the region.
[0017] Optionally, the merging and processing of the regional images captured by each of the time-delayed integration cameras to obtain the defect detection image of the target object includes:
[0018] Registering the regional images captured by each of the time-delayed integration cameras to obtain a registered regional image;
[0019] The registered regional images are stitched together to obtain a defect detection image of the target object.
[0020] Optionally, registering the regional images captured by the time-delayed integration cameras to obtain the registered regional images includes:
[0021] Determining whether the target object belongs to the first category or the second category;
[0022] If the target object belongs to the first category, for every two frames of regional images acquired by two adjacent time-delayed integration cameras, register the two frames of regional images based on a first registration strategy to obtain a registered regional image;
[0023] If the target object belongs to the second category, for every two frames of regional images acquired by two adjacent time-delay integration cameras, the two frames of regional images are registered based on a second registration strategy to obtain a registered regional image.
[0024] Optionally, the target objects of the first category include semiconductor products whose surface graphic features have a significance greater than a certain threshold;
[0025] The target objects of the second category include semiconductor products whose surface graphic features have a significance level less than or equal to the threshold.
[0026] Optionally, registering the two frames of the regional images based on the first registration strategy to obtain the registered regional image includes:
[0027] Determine a displacement parameter representing the degree of misalignment between the two frames of the regional images according to a correlation coefficient of the two frames of the regional images;
[0028] The two frames of the regional images are registered according to the displacement parameters of the two frames of the regional images to obtain two frames of registered regional images.
[0029] Optionally, determining the displacement parameter characterizing the degree of misalignment between the two frames of the regional images according to the correlation coefficient of the two frames of the regional images includes:
[0030] Determine a reference image and an image to be registered that overlap in position between two frames of the regional images according to overlapping pixels between two adjacent time-delay integration cameras;
[0031] Moving the image to be registered according to a plurality of displacement amounts respectively, and determining a normalized correlation coefficient corresponding to each displacement amount according to the reference image and the image to be registered after each movement;
[0032] The displacement amount whose corresponding normalized correlation coefficient satisfies the first condition is determined as a displacement parameter representing the degree of misalignment between the two frames of the regional images.
[0033] Optionally, the first condition includes that the normalized correlation coefficient corresponding to the displacement parameter is the maximum value of a plurality of normalized correlation coefficients corresponding to the plurality of displacement amounts.
[0034] Optionally, registering the two frames of the regional images based on the second registration strategy to obtain the registered regional image includes:
[0035] Determine a reference image and an image to be registered that overlap in position between two frames of the regional images according to overlapping pixels between two adjacent time-delay integration cameras;
[0036] Determining at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image;
[0037] The two frames of the regional images are registered according to the reference feature points and the feature points to be registered to obtain two frames of registered regional images.
[0038] Optionally, performing registration processing on the two frames of the regional images according to the reference feature points and the feature points to be registered to obtain two frames of registered regional images includes:
[0039] Determine a feature point pair from the reference feature points and the feature points to be registered, the feature point pair comprising one of the reference feature points and one of the feature points to be registered, and a similarity measure between the feature points included in the feature point pair satisfies a second condition;
[0040] The two frames of the regional images are registered according to the reference feature points and the feature points to be registered contained in the feature point pairs to obtain two frames of registered regional images.
[0041] Optionally, the determining at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image includes:
[0042] Obtaining a reference gradient image corresponding to the reference image and a gradient image to be registered corresponding to the image to be registered, wherein the pixel value of each pixel in the reference gradient image represents the gradient value of the pixel at the corresponding position in the reference image, and the pixel value of each pixel in the gradient image to be registered represents the gradient value of the pixel at the corresponding position in the image to be registered;
[0043] At least one reference feature point of the reference image is determined according to the reference gradient image, and at least one to-be-registered feature point of the to-be-registered image is determined according to the to-be-registered gradient image.
[0044] Optionally, determining at least one reference feature point of the reference image according to the reference gradient image includes:
[0045] Processing the reference gradient image according to a two-dimensional Gaussian function to obtain a reference Gaussian scale image;
[0046] Determining at least one reference feature point of the reference image according to a scale parameter of each pixel in the reference Gaussian scale image;
[0047] The step of determining at least one feature point to be registered of the image to be registered according to the gradient image to be registered comprises:
[0048] Processing the gradient image to be registered according to a two-dimensional Gaussian function to obtain a Gaussian scale image to be registered;
[0049] At least one to-be-registered feature point of the to-be-registered image is determined according to the scale parameter of each pixel in the to-be-registered Gaussian scale image.
[0050] Optionally, the manner of determining the similarity measure between the reference feature point and the feature point to be registered includes:
[0051] Determine a plurality of reference descriptors corresponding to the reference feature points, and determine a plurality of to-be-registered descriptors corresponding to the to-be-registered feature points;
[0052] A similarity measure between the reference feature points and the feature points to be registered is determined according to the multiple reference descriptors and the multiple descriptors to be registered.
[0053] Optionally, the determining a plurality of reference descriptors corresponding to the reference feature points includes:
[0054] Determining a main direction of the reference feature point according to a scale parameter of the reference feature point, and rotating the reference image according to the main direction of the reference feature point;
[0055] Determining a reference neighborhood corresponding to the reference feature point in the rotated reference image, and determining a plurality of reference descriptors of the reference feature point according to a scale parameter of each pixel in the reference neighborhood;
[0056] The determining of a plurality of descriptors to be registered corresponding to the feature points to be registered comprises:
[0057] Determine the main direction of the feature point to be registered according to the scale parameter of the feature point to be registered, and rotate the image to be registered according to the main direction of the feature point to be registered;
[0058] A neighborhood to be registered corresponding to the feature point to be registered is determined in the rotated image to be registered, and a plurality of descriptors to be registered of the feature point to be registered are determined according to scale parameters of each pixel in the neighborhood to be registered.
[0059] A second aspect of the present application provides an image processing system, comprising at least an acquisition component and a data processing component;
[0060] The acquisition assembly includes a plurality of time-delayed integration cameras, a first beam splitter prism and a second beam splitter prism bonded in an interlaced manner;
[0061] The first beam splitter prism has a coating that enhances transmission capability;
[0062] The second beam splitter prism has a coating that enhances reflection capability;
[0063] Each of the time delay integration cameras corresponds to one of the beam splitting prisms;
[0064] Wherein, the photosensitive surface of the time-delayed integration camera corresponding to the first beam splitter prism corresponds to the transmission image surface of the first beam splitter prism;
[0065] The photosensitive surface of the time delay integration camera corresponding to the second beam splitter prism corresponds to the reflection image surface of the second beam splitter prism;
[0066] The transmission image plane of the first beam splitter prism and the reflection image plane of the second beam splitter prism are orthogonal;
[0067] Part of the pixels of the time delay integration cameras corresponding to every two adjacent beam splitting prisms overlap with each other.
[0068] A third aspect of the present application provides an electronic device, including a processor and a memory;
[0069] The memory is used to store computer programs;
[0070] The processor is used to execute the computer program, and when the computer program is executed, it is specifically used to implement the image processing method provided in any one of the first aspects of the present application.
[0071] The fourth aspect of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the image processing method provided in any one of the first aspects of the present application.
[0072] The beneficial effects of this application are:
[0073] The integration level of the TDI cameras corresponding to different areas is adjusted according to the scattering rates of different areas, thereby changing the exposure levels of the regional images obtained by the TDI cameras in different areas, making the exposure levels of multiple frames of regional images as consistent as possible, alleviating the problem of excessive differences in image exposure levels in different areas, and is conducive to improving the accuracy of subsequent defect detection results obtained based on regional images. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0075] Figure 1 is a structural schematic diagram of a mechanically spliced TDI camera provided in an embodiment of the present application;
[0076] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;
[0077] Figure 3 is a flow chart of an image registration method provided in an embodiment of the present application;
[0078] Figure 4 is a schematic diagram of determining a displacement parameter provided in an embodiment of the present application;
[0079] Figure 5 is a flow chart of another image registration method provided in an embodiment of the present application;
[0080] Figure 6 is a schematic diagram of determining a descriptor provided in an embodiment of the present application;
[0081] Figure 7 is a schematic diagram of an image registration based on feature points provided in an embodiment of the present application;
[0082] Figure 8 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;
[0083] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0084] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0085] The image processing method provided in this embodiment can be applied to a semiconductor product defect detection system (i.e., an image processing system) based on a TDI camera. The system can collect images of semiconductor products and analyze whether the semiconductor product has defects, as well as the location and size of the defects based on the images. The semiconductor product can be a wafer particle (or crystal grain, die).
[0086] The system may include an illumination assembly for providing illumination, an optical assembly for microscopic imaging, an acquisition assembly for acquiring images, and a data processing assembly for image processing.
[0087] The data processing component can be any electronic device with data processing capability. The acquisition component can include a plurality of TDI cameras spliced together by a mechanical structure.
[0088] The acquisition assembly includes a plurality of time-delay integration cameras, a first beam splitter prism and a second beam splitter prism bonded in an interlaced manner;
[0089] The first beam splitter prism has a coating that enhances transmission capability;
[0090] The second beam splitter prism has a coating that enhances reflection capability;
[0091] Each time delay integration camera corresponds to a beam splitter prism;
[0092] Wherein, the photosensitive surface of the time-delayed integration camera corresponding to the first beam splitting prism corresponds to the transmission image surface of the first beam splitting prism;
[0093] The photosensitive surface of the time delay integration camera corresponding to the second beam splitter prism corresponds to the reflection image surface of the second beam splitter prism;
[0094] The transmission image plane of the first beam splitter prism and the reflection image plane of the second beam splitter prism are orthogonal;
[0095] Part of the pixels between the time delay integration cameras corresponding to every two adjacent beam splitting prisms overlap with each other.
[0096] As an example, the acquisition component may have Figure 1 The structure shown in (1) of the invention. The acquisition component includes two cross-bonded beam splitters, one of which has a coating that enhances transmission capability (i.e., the first beam splitter), and the other beam splitter has a coating that enhances reflection capability (i.e., the second beam splitter). The cross-bonded beam splitters correspond to multiple time-delay integration cameras, for example Figure 1 (1) of TDI camera 1 to TDI camera 3, wherein the photosensitive surface of the time delay integration camera corresponding to the first beam splitter prism corresponds to the transmission image plane of the first beam splitter prism, and the photosensitive surface of the time delay integration camera corresponding to the second beam splitter prism corresponds to the reflection image plane of the second beam splitter prism.
[0097] The above correspondence can be achieved by mounting the photosensitive array of the TDI camera corresponding to the first beam splitter prism on the transmission image plane of the first beam splitter prism by gluing (or other mounting methods), for example Figure 1 In (1), the photosensitive arrays of the odd number of TDI cameras, namely TDI camera 1 and TDI camera 3 (ie, photosensitive array 1 and photosensitive array 3) are located on the transmission image plane of the first beam splitter prism (ie, Figure 1 The even number of TDI cameras, namely the photosensitive array of TDI camera 2 (ie, photosensitive array 2), is located on the reflection image plane of the second beam splitter prism (ie, Figure 1 The photosensitive arrays of the odd-numbered TDI cameras and the photosensitive arrays of the even-numbered TDI cameras are located on orthogonal transmission and reflection image planes, respectively. This stitching method can ensure that the optical path from the semiconductor product to the two image planes is completely equal, thereby avoiding the optical path difference and field of view misalignment problems of conventional misaligned stitching methods.
[0098] The photosensitive array of a TDI camera consists of multiple pixels arranged in a matrix.
[0099] Further, such as Figure 1As shown in (1), to ensure the continuity of the stitched image, there is a certain pixel overlap between adjacent TDI cameras (or adjacent photosensitive arrays), for example Figure 1 In (1), some pixels of TDI camera 1 and TDI camera 2 overlap with each other (i.e. Figure 1 (1) The shaded area of TDI camera 1 and TDI camera 2) When the acquisition component acquires the image of the wafer, this part of the overlapping wafer can acquire the image of the same area on the wafer. The partial overlap of pixels of adjacent TDI cameras can effectively expand the number of pixels scanned by the TDI camera, meet the needs of wide field imaging, and improve the speed and yield of defect detection. Generally speaking, the width of the overlapping part of the pixels of adjacent TDI cameras should be at least 50 pixels to meet the accuracy requirements of image registration.
[0100] Figure 1 When the acquisition component shown in (1) is working, the wafer being photographed can move at a certain target motion speed, and then the acquisition component uses a push-scan method to expose and image the wafer being photographed, thereby obtaining an image of the wafer surface. During the exposure and imaging process, the exposure time of the TDI camera needs to match the target motion speed, so when the acquisition component mechanically spliced with multiple TDI cameras photographs wafers or other semiconductor products, the multiple TDI cameras need to maintain a uniform exposure time.
[0101] Different areas of the wafer surface being photographed may have different surface scattering rates. In the captured image, areas with too high surface scattering rates may appear overexposed and the grayscale of the pixels may be too high, while areas with too low surface scattering rates may appear underexposed and the grayscale of the pixels may be too low. This situation will affect the accuracy of data analysis.
[0102] In order to solve the above problems, the present application provides an image processing method. Figure 2 , the method may include the following steps.
[0103] S201, obtaining scattering rate parameters of multiple regions of a target object.
[0104] The target object may be any semiconductor product being photographed, for example, a wafer that needs to be inspected for defects.
[0105] The scattering rate parameters of multiple regions can be obtained from the process recipe of the target object. The process recipe of the semiconductor product describes the various process steps for preparing the semiconductor product, as well as the process requirements and parameters of each step.
[0106] In step S201, the recipe of the target object may be directly read from a memory storing the recipe, and the scattering rate parameters of each area on the surface of the target object may be read from the recipe of the target object.
[0107] S202, configuring the integration level of the time-delayed integration camera corresponding to each region when acquiring images according to the scattering rate parameters of each region, each time-delayed integration camera is used to acquire a regional image of the corresponding region.
[0108] Multiple time delay integration cameras corresponding to each area can be Figure 1 (1) shows a plurality of TDI cameras mechanically spliced together. Figure 1 When the acquisition device shown in (1) photographs the target object, each TDI camera can respectively receive the light signal reflected by the corresponding area on the target object to form a regional image of the area.
[0109] In S202, the method of determining the integral order according to the scattering rate parameter may be:
[0110] For each region, according to the preset inverse relationship and the scattering rate parameter of the region, the integration level of the time delay integration camera corresponding to the region when collecting images is configured. That is to say, there is an inverse relationship between the integration level of the time delay integration camera corresponding to each region and the scattering rate parameter of the region. For any region of the target object, the larger the value of the scattering rate parameter of the region, the smaller the integration level of the TDI camera corresponding to the region, and the smaller the value of the scattering rate parameter of the region, the larger the integration level of the TDI camera corresponding to the region. In other words, when configuring the integration level, a smaller integration level can be configured for the TDI camera corresponding to the region with a higher scattering rate, and a larger integration level can be configured for the TDI camera corresponding to the region with a lower scattering rate.
[0111] As an example, a constant may be preset, and for any area on the surface of the target object, the constant may be divided by the scattering rate parameter of the area, and the result obtained may be used as the integral series of the TDI camera corresponding to the area.
[0112] S203, merging and processing the regional images captured by each time-delayed integration camera to obtain a defect detection image of the target object.
[0113] The manner in which each time delay integration camera collects regional images may be:
[0114] Determine a trigger shooting electrical signal synchronized with the moving speed of the target object;
[0115] Sending a trigger shooting electrical signal to each time delay integration camera through a trigger signal circuit to trigger each time delay integration camera to obtain an image signal of the target object;
[0116] The image signals output by each time-delay integration camera are modulated according to the trigger shooting electrical signal to obtain a regional image.
[0117] The image signal obtained by the time delay integration camera can be a photoelectric signal obtained after the camera's pixel converts the light signal reflected from the surface of the target object.
[0118] A plurality of time delay integration cameras can be connected to the data processing component through independent trigger signal circuits, and output image signals to the data processing component through independent output circuits.
[0119] The advantage of acquiring the regional image in the above manner is that it ensures that the multiple mechanically spliced TDI cameras can synchronously photograph various regions of the target object, thereby obtaining an accurate regional image.
[0120] In step S203, multiple frames of regional images captured by multiple TDI cameras may be spliced together according to the positional relationship between the multiple TDI cameras, and the spliced images may be used as defect detection images for performing defect detection on the target object.
[0121] by Figure 1 (2) is an example. Assume that the width of the overlapping part between TDI camera 1 and TDI camera 2 is 50 pixels, and the width of the overlapping part between TDI camera 2 and TDI camera 3 is 50 pixels, corresponding to 50 pixels of the image. Then in S203, according to the positional relationship of each TDI camera, the regional image 1 taken by TDI camera 1 can be placed on the left side of the regional image 2 taken by TDI camera 2, and the regional image 3 taken by TDI camera 3 can be placed on the right side of the regional image 2 taken by TDI camera 2. In addition, the 50-pixel-wide areas on the right side of regional image 1 and on the left side of regional image 2 overlap with each other, and the 50-pixel-wide areas on the left side of regional image 3 and on the right side of regional image 2 overlap with each other. Finally, the three frames of regional images are merged into one frame of image according to this positional relationship, and the merged image is used as the defect detection image of the target object.
[0122] The beneficial effects of the image processing method of this embodiment are described below:
[0123] The photoreceptor of a TDI camera consists of multiple rows of pixels, each of which is a light sensor element (such as a charge-coupled device CCD), and each row of pixels forms a line array. When a TDI camera captures an image, each line array independently captures the light signal and converts the captured light signal into an electric charge, which can flow along the sensor to the next line array as the target object moves. By matching the flow speed of the charge with the speed of the target movement, the signals captured by multiple line arrays are precisely aligned and accumulated in time, thereby significantly increasing the signal strength and improving the image signal-to-noise ratio.
[0124] The integration level of a TDI camera is equivalent to the number of line arrays used by the TDI camera when capturing images. For example, if the integration level of a TDI camera is configured as 64, the TDI camera can use 64 line arrays when capturing images, that is, use 64 rows of pixels. If the integration level of a TDI camera is configured as 128, the TDI camera can use 128 line arrays when capturing images, that is, use 128 rows of pixels.
[0125] From the working principle of the above-mentioned TDI camera, it can be seen that the more integration levels are used during shooting, the greater the intensity of the light signal captured by the TDI camera, and the fewer integration levels are used during shooting, the smaller the intensity of the light signal captured by the TDI camera.
[0126] When collecting regional images, the image processing method of this embodiment configures a smaller integration level for the TDI camera corresponding to the area with a higher scattering rate and stronger scattered light, thereby reducing the intensity of the light signal cumulatively captured by the TDI camera in the area; and configures a larger integration level for the TDI camera corresponding to the area with a lower scattering rate and weaker scattered light, thereby increasing the intensity of the light signal cumulatively captured by the TDI camera in the area. In this way, the intensities of light signals captured by multiple TDI cameras from corresponding areas can be controlled to approach each other, so that the exposure levels of the multi-frame regional images finally outputted are as consistent as possible, alleviating the problem of excessive differences in image exposure levels in different areas, which is beneficial to improving the accuracy of subsequent defect detection results obtained based on regional images.
[0127] In some optional embodiments, step S203 may include:
[0128] Registering the regional images captured by each time-delayed integration camera to obtain a registered regional image;
[0129] The images of each registered area are stitched together to obtain the defect detection image of the target object.
[0130] That is, before stitching the regional images, the regional images may be registered first, and then the registered regional images may be stitched together to form a defect detection image.
[0131] For any two frames of regional images, registering the two can be understood as adjusting the relative position relationship between the two frames so that the relative position relationship between the two frames of images meets specific conditions. The two frames of images in a specific relative position relationship after adjustment can be regarded as two registered frames of images.
[0132] For any two registered image frames, stitching the two registered image frames together can be regarded as merging the two image frames into one image frame while maintaining a specific relative position relationship between the two frames. The merged image still maintains the specific relative position relationship determined during the registration.
[0133] The beneficial effects of this embodiment are:
[0134] Multiple TDI cameras spliced together by mechanical structures are prone to pixel misalignment and data transmission delays during long-term use. This will cause the positional relationship between the regional images captured by each TDI camera to be inconsistent with the positional relationship between the TDI cameras. However, registering the regional images before splicing them can correct this deviation and ensure that the defect detection image formed after splicing can accurately reflect the surface condition of the target object.
[0135] Optionally, when registering the regional images, different registration strategies may be used for different categories of target objects. In other words, the aforementioned step of registering the regional images captured by each time-delayed integration camera may include:
[0136] Determine whether the target object belongs to the first category or the second category;
[0137] If the target object belongs to the first category, for every two frames of regional images acquired by two adjacent time-delayed integration cameras, register the two frames of regional images based on a first registration strategy to obtain a registered regional image;
[0138] If the target object belongs to the second category, for every two frames of regional images acquired by two adjacent time-delayed integration cameras, the two frames of regional images are registered based on the second registration strategy to obtain a registered regional image.
[0139] The first category of target objects may include semiconductor products whose surface graphic features are more significant than a certain threshold. In other words, the first category of target objects may be understood as having obvious graphic features, such as wafers that have been etched with graphics.
[0140] The second category of target objects may include semiconductor products whose surface graphic features have a significance greater than a certain threshold. In other words, the second category of target objects may be understood as having no obvious graphic features, such as wafers without etched graphics.
[0141] The surface of the target object of the second category may have a small amount of significant particles. The significant particles may be understood as some tiny areas on the surface of the target object that are significantly different from the surrounding areas, for example, may be understood as areas where tiny defects are located.
[0142] Whether the target object belongs to the first category or the second category can be determined based on the process recipe of the target object. If the process recipe includes the step of etching the graphics, it can be determined that the target object belongs to the first category. If the process recipe does not include the step of etching the graphics, it can be determined that the target object belongs to the second category.
[0143] The beneficial effects of this embodiment are:
[0144] Selecting appropriate registration strategies for registration according to different target object categories is helpful to improve the accuracy of the registration results.
[0145] See also Figure 3 , is a flow chart of a method for registering two frames of regional images based on a first registration strategy provided in this embodiment. The method may include the following steps.
[0146] S301, determining a displacement parameter representing the degree of misalignment between two frames of regional images according to a correlation coefficient of the two frames of regional images.
[0147] The registration method provided in this embodiment can be used to register two frames of regional images captured by any two adjacent TDI cameras.
[0148] In step S301, the positional relationship between the two frame area images can be adjusted according to different displacement amounts, and the correlation coefficient between the two adjusted frame area images can be calculated after each adjustment, thereby determining the correlation coefficient corresponding to each displacement amount, and then a displacement parameter can be determined from multiple displacement amounts based on the correlation coefficients corresponding to each displacement amount.
[0149] S302, performing registration processing on the two frames of regional images according to the displacement parameters of the two frames of regional images to obtain two frames of registered regional images.
[0150] The registration method may be to adjust the relative position between the two frames of regional images according to the displacement parameter determined in S301, and the adjusted images are used as the two frames of registered regional images.
[0151] Combination Figure 1 Assume that the positional relationship between region image 1 and region image 2 before registration is as follows: Figure 1 As shown in (2), when the registration is performed according to the method of this embodiment, the displacement parameter between the region image 1 and the region image 2 is determined to be 10 pixels in step S301, so Figure 1In (2), regional image 2 is moved to the direction of regional image 1, that is, to the left by another 10 pixels. The moved regional image 1 and regional image 2 are used as two frames of registered regional images.
[0152] The beneficial effects of this embodiment are:
[0153] The registration can be performed directly according to the correlation coefficient between the images. The registration speed is fast and the registration can be completed more quickly to obtain the defect detection image.
[0154] The process of determining the displacement parameter in step S301 may include:
[0155] S3011, determining a reference image and an image to be registered that overlap in position between two frames of regional images according to overlapping pixels between two adjacent time delay integration cameras.
[0156] Each pixel of the TDI camera corresponds to a pixel in the captured image, so in step S3011, the number of pixels overlapping in the width direction between two adjacent time delay integration cameras can be determined first, and then the area of corresponding width (expressed in the number of pixels) between the two frames of regional images is determined as the overlapping area, and the parts of the two frames of regional images located in this area are determined as the reference image and the image to be registered.
[0157] by Figure 1 (1) and Figure 4 As an example, assume that the area image captured by TDI camera 1 is Figure 4 In image 1 of (1), the regional image captured by TDI camera 2 is Figure 4 In image 2 of (1), TDI camera 1 is located to the left of TDI camera 2, and there is an overlap of 50 pixels between the two. Figure 4 The 50-pixel-wide area on the right side of image 1 in (1) is determined as the reference image. Figure 4 The 50-pixel-wide area on the left side of image 2 in (1) is determined as the image to be registered (as shown by the shaded part in the figure).
[0158] S3012, respectively moving the image to be registered according to the multiple displacement amounts, and determining a normalized correlation coefficient corresponding to each displacement amount according to the reference image and the image to be registered after each movement.
[0159] The displacement can be expressed in pixels and can range from 0 to N, where N is the width of the reference image (also the width of the image to be registered).
[0160] In combination with the above example, assuming that the width of the reference image and the image to be registered is 50 pixels, then the above multiple displacements may include 0 pixels to 50 pixels. Correspondingly, in step S3012, the normalized correlation coefficient corresponding to the displacement of 1 pixel, the normalized correlation coefficient corresponding to the displacement of 2 pixels... and the normalized correlation coefficient corresponding to the displacement of 50 pixels can be calculated in sequence.
[0161] See also Figure 4 (2), when calculating the displacement n, the image to be registered can be first moved according to the displacement, for example, the image to be registered is moved in the direction of image 1, that is, moved n pixels to the left. After the movement, a part of the image to be registered will exceed the boundary of the reference image, that is, it will no longer overlap with the reference image (such as the part beyond the boundary marked in the figure). At this time, this part can be cropped out and moved to the right side of the image to be registered, thereby obtaining the image to be registered after being moved by the displacement n, and then the normalized correlation coefficient (Normalized Correlation Coefficients, NCC) between the reference image and the image to be registered after being moved by the displacement n can be calculated, and the calculation result is used as the normalized correlation coefficient corresponding to the displacement n.
[0162] For any displacement n, the normalized correlation coefficient of the above-mentioned calculated displacement n can be expressed by the following formula (1).
[0163]
[0164] Among them, NCC n represents the normalized correlation coefficient corresponding to the displacement n, and mean(P) represents the normalized correlation coefficient between P1 and P N The average value of P1 to P N The accumulated result is divided by N, and mean(Q) represents the average value from Q1 to Q. N The average value of .
[0165] For any integer i between 1 and N, P i It can represent the average grayscale value of the pixels in the i-th column of the reference image, Q i It can represent the average grayscale value of the pixels in the i-th column of the image to be registered, where the pixels in the first column of the reference image and the image to be registered are Figure 4 The leftmost column of pixels in (1).
[0166] It can be seen that after the image to be registered is moved by the displacement amount n, the grayscale average value of each column of pixels in the reference image is multiplied by the grayscale average value of a column of pixels at the corresponding position in the image to be registered after the movement, and the result of the accumulation of N products is equivalent to the numerator of formula (1).
[0167] S3013: Determine the displacement amount whose corresponding normalized correlation coefficient satisfies the first condition as a displacement parameter representing the degree of misalignment between the two frames of regional images.
[0168] The first condition can be set according to actual conditions. For example, the first condition can be that the normalized correlation coefficient corresponding to the displacement parameter is the maximum value among multiple normalized correlation coefficients corresponding to multiple displacement amounts. In other words, in step S3013, the displacement amount with the largest corresponding normalized correlation coefficient can be used as the displacement parameter representing the degree of misalignment between the two frame regional images.
[0169] For example, assuming that N is equal to 50, the displacement range is 0 to 50 pixels. After calculating the normalized correlation coefficient according to step S3012, the normalized correlation coefficient corresponding to the displacement n0 = 30 pixels is greater than the normalized correlation coefficients corresponding to the other 49 displacements, that is, the normalized correlation coefficient corresponding to the displacement n0 is the maximum value among the normalized correlation coefficients corresponding to all the displacements. Therefore, the displacement n0 can be determined as the displacement parameter, that is, the displacement parameter is determined to be equal to 30 pixels.
[0170] The beneficial effects of this embodiment are:
[0171] When registering regional images of different regions, due to the different integration levels of the TDI cameras that shoot the regional images, there may be some differences in the grayscale distribution of pixels between the regional images. For example, the grayscale of the pixels in image 1 is between 50 and 100, and the grayscale of the pixels in image 2 may be between 80 and 130. This difference may have an adverse effect on the registration result. This embodiment determines the displacement parameter for registration based on the normalized correlation coefficient of the image, which is conducive to eliminating the impact of such grayscale differences on the registration result and obtaining a more accurate registration result.
[0172] See also Figure 5 , is a flow chart of a method for registering two frames of regional images based on a second registration strategy provided in this embodiment. The method may include the following steps.
[0173] S501 : determining a reference image and an image to be registered whose positions overlap between two frames of regional images according to overlapping pixels between two adjacent time delay integration cameras.
[0174] The implementation of step S501 may refer to step S301 and will not be described in detail.
[0175] S502: Determine at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image.
[0176] Among them, the feature points can be determined as follows:
[0177] A0, obtain a reference gradient image corresponding to the reference image and a gradient image to be registered corresponding to the image to be registered, the pixel value of each pixel in the reference gradient image represents the gradient value of the pixel at the corresponding position in the reference image, and the pixel value of each pixel in the gradient image to be registered represents the gradient value of the pixel at the corresponding position in the image to be registered;
[0178] A1, determining at least one reference feature point of the reference image according to the reference gradient image, and determining at least one to-be-registered feature point of the to-be-registered image according to the to-be-registered gradient image.
[0179] In step A1, the method of determining the reference feature points according to the reference gradient image can refer to steps A2 and A3, and the method of determining the feature points to be registered according to the gradient image to be registered can refer to steps A4 and A5.
[0180] A2, process the reference gradient images according to the two-dimensional Gaussian function to obtain the reference Gaussian scale images;
[0181] A3, determining at least one reference feature point of the reference image according to the scale parameter of each pixel in the reference Gaussian scale image;
[0182] A4, respectively processing the gradient images to be registered according to the two-dimensional Gaussian function to obtain the Gaussian scale images to be registered.
[0183] A5: Determine at least one feature point to be registered of the image to be registered according to the scale parameter of each pixel in the Gaussian scale image to be registered.
[0184] In step A0, for the reference image, the first-order gradient value of each pixel in the reference image can be calculated, and the calculation result is used as the grayscale value of the corresponding pixel in the reference gradient image. For example, for the pixel at the coordinate position (x0, y0) in the reference image, the first-order gradient value of the pixel is calculated, and the calculation result is used as the grayscale value of the pixel at the position (x0, y0) in the reference gradient image. After traversing each pixel in the reference image, a reference gradient image with the same size as the reference image can be obtained.
[0185] For the image to be registered, the first-order gradient value of each pixel in the reference image can be calculated, and the calculation result can be used as the grayscale value of the corresponding pixel in the gradient image to be registered. After traversing each pixel in the image to be registered, the gradient image to be registered with the same size as the image to be registered can be obtained.
[0186] In step A2, the reference gradient image can be processed using the two-dimensional Gaussian function G(x, y, σ) according to the following formula (2) to obtain a corresponding reference Gaussian scale image, and the gradient image to be registered can be processed using the function to obtain a corresponding Gaussian scale image to be registered.
[0187]
[0188] Among them, when used to process the reference gradient image, I(x, y) represents the grayscale value of the pixel at the (x, y) position in the reference gradient image, and L(x, y, σ) represents the scale parameter of the pixel at the corresponding position in the reference Gaussian scale image obtained after processing; when used to process the gradient image to be registered, I(x, y) represents the grayscale value of the pixel at the (x, y) position in the gradient image to be registered, and L(x, y, σ) represents the scale parameter of the pixel at the corresponding position in the Gaussian scale image to be registered after processing. It represents the standard deviation of the Gaussian function and also the scale factor of the scale space.
[0189] It should be noted that the reference Gaussian scale image, the reference gradient image and the reference image have the same size, and the pixels of the three correspond one to one; the Gaussian scale image to be registered, the gradient image to be registered and the image to be registered have the same size, and the pixels of the three correspond one to one.
[0190] In step A3, any point feature extraction operator may be used to process the scale parameters of each pixel in the reference Gaussian scale image to obtain feature parameters of each pixel, and then the reference feature points are determined based on the feature parameters.
[0191] As some examples, the point feature extraction operator used may be any one of the Harris operator, the SUSAN operator and the ORB operator.
[0192] For example, when the Harris operator is used, for any pixel K of the reference Gaussian scale image, the characteristic parameter R (K, σ) of the pixel can be calculated using the following formula (3): n ).
[0193]
[0194] Where M is a matrix, det means calculating the determinant of the matrix, tr means calculating the trace of the matrix, and D is a preset constant used as a parameter, which can usually be set to 0.04.
[0195] For any pixel K in the reference Gaussian scale image, if the feature parameter R(K, σ n ) is greater than the preset Harris global response threshold, and the feature parameter of the pixel K is a maximum value in the local range centered on K (that is, the feature parameter of K is greater than the feature parameters of other pixels in the local range), it can be determined that the pixel corresponding to pixel K in the reference image belongs to the reference feature point.
[0196] The local range may be a range with a preset scale centered on K, for example, a square range with a size of 7*7 centered on K.
[0197] The matrix M can be expressed as follows:
[0198]
[0199] Where, formula (5) indicates that the standard deviation is √2σ n Gaussian function.
[0200]
[0201] σ n is the scale of pixel K, i.e. σ in the above formula (2). L1 n represents the horizontal derivative of the reference Gaussian scale image at pixel point K, L2 n Represents the vertical derivative of the reference Gaussian scale image at pixel point K.
[0202] In steps A4 and A5, the methods of obtaining the Gaussian scale image to be registered and determining the feature points to be registered can refer to the methods of obtaining the reference Gaussian scale image and determining the reference feature points in steps A2 and A3, and will not be repeated here.
[0203] The advantages of determining the reference feature points and the feature points to be registered in the above manner are:
[0204] On the one hand, by using the Harris operator to determine feature points, pixels with obvious changes in grayscale values in the two-dimensional direction can be determined as feature points, which has good anti-noise performance, and the extracted feature points are stable and evenly and reasonably distributed in the image.
[0205] On the other hand, before using the Harris operator to calculate the feature parameters, the reference image and the image to be registered are first converted into corresponding first-order gradient images, and the first-order gradient image is used to generate the corresponding Gaussian scale image, which is conducive to eliminating the image produced by the grayscale difference between images in different regions and can make the determined feature points scale invariant.
[0206] After the feature points are determined, the two frames of regional images can be registered according to the reference feature points and the feature points to be registered to obtain two frames of registered regional images.
[0207] Please continue to see Figure 5 The process of performing registration processing according to the reference feature points and the feature points to be registered may include the following steps S503 and S504.
[0208] S503 , determining a feature point pair from the reference feature points and the feature points to be registered, the feature point pair including a reference feature point and a feature point to be registered, and the similarity measure between the feature points included in the feature point pair meets the second condition.
[0209] In step S503, the reference feature points and the feature points to be registered can be combined in pairs, and each combination includes a reference feature point and a feature point to be registered; then, for each combination, the similarity measure between the reference feature points and the feature points to be registered contained in the combination can be calculated; finally, a combination that meets the second condition is determined among all the combinations, and this combination is determined as the feature point pair in S503, that is, it is determined that the reference feature points and the feature points to be registered in the combination match each other.
[0210] For any reference feature point and any feature point to be registered, the method of determining the similarity measure between the two may include:
[0211] Determine a plurality of reference descriptors corresponding to the reference feature points, and determine a plurality of to-be-registered descriptors corresponding to the to-be-registered feature points;
[0212] A similarity measure between the reference feature points and the feature points to be registered is determined according to the plurality of reference descriptors and the plurality of descriptors to be registered.
[0213] The method for determining the reference descriptor may refer to steps B1 and B3, and the method for determining the descriptor to be registered may refer to steps B3 and B4.
[0214] B1, determining the main direction of the reference feature point according to the scale parameter of the reference feature point, and rotating the reference image according to the main direction of the reference feature point;
[0215] B2, determining a reference neighborhood corresponding to the reference feature point in the rotated reference image, and determining a plurality of reference descriptors of the reference feature point according to a scale parameter of each pixel in the reference neighborhood;
[0216] B3, determining the main direction of the feature point to be registered according to the scale parameter of the feature point to be registered, and rotating the image to be registered according to the main direction of the feature point to be registered;
[0217] B4, determining a neighborhood to be registered corresponding to the feature point to be registered in the rotated image to be registered, and determining a plurality of descriptors to be registered of the feature point to be registered according to the scale parameter of each pixel in the neighborhood to be registered;
[0218] B5, determining a similarity measure between the reference feature points and the feature points to be registered according to the multiple reference descriptors and the multiple descriptors to be registered.
[0219] In step B1, the main direction θ of the reference feature point K can be calculated according to the following formula (6): K .
[0220]
[0221] Among them, L(x, y+1), L(x, y-1), L(x+1, y), and L(x-1, y) respectively represent the scale parameters corresponding to the pixels at positions (x, y+1), (x, y-1), (x+1, y), and (x-1, y) in the reference image in the reference Gaussian scale image, and (x, y) is the position of the reference feature point K.
[0222] It can be seen that the main direction θ K is equivalent to an angle parameter. Therefore, after obtaining the main direction, the reference image can be rotated by the corresponding angle. For example, if the main direction θ is calculated according to formula (6): K If the value is 30 degrees, the reference image can be rotated 30 degrees clockwise.
[0223] In step B2, the reference neighborhood refers to a square neighborhood of 16*16 pixels centered on the reference feature point.
[0224] See also Figure 6 After the reference neighborhood is determined, the reference neighborhood can be divided into 16 4*4 seed points, each of which contains 4*4 pixels in the reference neighborhood.
[0225] For a seed point, the vectors of each pixel in the seed point in 8 directions can be calculated, and the amplitudes of these vectors can be combined to obtain a 128-dimensional (16*8) descriptor vector of the seed point, which is a reference descriptor of the reference feature point. By processing each seed point in the reference neighborhood in this way, 16 reference descriptors corresponding to the reference feature point can be obtained.
[0226] The calculation method of the pixel vector in each direction can be found in the Scale-invariant feature transform (SIFT) algorithm, which will not be described in detail.
[0227] The specific implementation method of B3 and B4 can refer to B1 and B2. It only needs to replace the data related to the reference image in B1 and B2 with the data related to the image to be registered, which will not be repeated here.
[0228] In step B5, the similarity measure can be represented by any index such as the minimum Euclidean distance, correlation coefficient, covariance function, sum of squared differences, sum of absolute values of differences, etc. between the multiple reference descriptors and the multiple descriptors to be registered.
[0229] As an example, in this embodiment, a correlation coefficient C between multiple reference descriptors and multiple to-be-registered descriptors may be used to represent a correlation measure. The correlation coefficient C may be calculated using the following formula (7).
[0230]
[0231] Among them, the calculated C is the correlation coefficient between the multiple reference descriptors of the reference feature points and the multiple descriptors to be registered of the feature points to be registered, which can also be considered as a similarity measure between the two. i represents the i-th reference descriptor, q i represents the i-th descriptor to be registered, where the reference descriptor and the descriptor to be registered with the same label have the same corresponding seed point position in the neighborhood. For example, p1 represents the reference descriptor corresponding to the seed point in the upper left corner of the reference neighborhood, and q1 represents the descriptor to be registered corresponding to the seed point in the upper left corner of the neighborhood to be registered.
[0232] N is the total number of reference descriptors, combined with Figure 6 For example, N can be equal to 16.
[0233] represents the average value of 16 reference descriptors corresponding to the reference feature points, Represents the average value of the 16 descriptors to be registered corresponding to the feature points to be registered.
[0234] By determining the descriptors of the feature points in the above manner, the influence of the grayscale difference between images of different regions can be avoided, and the registration of images of regions with different grayscale distributions can be accurately achieved.
[0235] S504, performing registration processing on the two frames of regional images according to the reference feature points and the feature points to be registered contained in the feature point pairs, to obtain two frames of registered regional images.
[0236] In step S504, the main direction of the reference feature point contained in the feature point pair and the main direction of the feature point to be registered can be determined first; then, a frame of the regional image to which the reference image belongs (for example Figure 7 The image 1) is rotated clockwise by a corresponding angle according to the main direction of the reference feature point, and at the same time, another frame area image (for example Figure 7 2) Rotate the image clockwise by a corresponding angle according to the main direction of the feature point to be registered.
[0237] After the rotation is complete, see Figure 7 , the relative position between image 1 to which the reference feature point belongs and image 2 to which the feature point to be registered belongs can be adjusted so that the feature point to be registered on image 2 overlaps with the reference feature point on image 1, that is, the two feature points belonging to the two frames of images are on the same plane coordinates. At this point, the registration process of the two frames of regional images is completed. At this time, the two frames of regional images are two frames of registered regional images.
[0238] The beneficial effect of this embodiment is that, compared with the first registration strategy, the registration method of this embodiment does not rely on the graphic features of the target object surface during registration, and only a small number of feature points are required to achieve accurate image registration. It has a wider range of applications and is particularly suitable for registering images of semiconductor products whose surfaces have not yet been etched with graphics.
[0239] This embodiment also provides an image processing device. Figure 8 , the device may include the following units.
[0240] The obtaining unit 801 obtains scattering rate parameters of multiple regions of the target object.
[0241] The configuration unit 802 configures the integration level of the time delay integration camera corresponding to each area when collecting images according to the scattering rate parameters of each area, and each time delay integration camera is used to collect the regional image of the corresponding area.
[0242] The merging unit 803 merges the regional images collected by the various time-delayed integration cameras to obtain a defect detection image of the target object.
[0243] Optionally, the merging unit 803 triggers each time delay integration camera to collect the regional image in the following manner:
[0244] Determine a trigger shooting electrical signal synchronized with the moving speed of the target object;
[0245] Sending a trigger shooting electrical signal to each time delay integration camera through a trigger signal circuit to trigger each time delay integration camera to obtain an image signal of the target object;
[0246] The image signals output by each time-delay integration camera are modulated according to the trigger shooting electrical signal to obtain a regional image.
[0247] Optionally, the configuration unit 802 configures the integration level of the time delay integration camera corresponding to each area when collecting images according to the scattering rate parameters of each area, including:
[0248] For each region, the integration level of the time-delayed integration camera corresponding to the region when acquiring images is configured according to the preset inverse relationship and the scattering rate parameter of the region.
[0249] Optionally, the merging unit 803 merges the regional images captured by each time-delayed integration camera to obtain a defect detection image of the target object, including:
[0250] Registering the regional images captured by each time-delayed integration camera to obtain a registered regional image;
[0251] The images of each registered area are stitched together to obtain the defect detection image of the target object.
[0252] Optionally, the merging unit 803 registers the regional images captured by each time delay integration camera to obtain a registered regional image, including:
[0253] Determine whether the target object belongs to the first category or the second category;
[0254] If the target object belongs to the first category, for every two frames of regional images acquired by two adjacent time-delayed integration cameras, register the two frames of regional images based on a first registration strategy to obtain a registered regional image;
[0255] If the target object belongs to the second category, for every two frames of regional images acquired by two adjacent time-delayed integration cameras, the two frames of regional images are registered based on the second registration strategy to obtain a registered regional image.
[0256] Optionally, the target objects of the first category include semiconductor products with patterns etched on their surfaces;
[0257] The second category of target objects includes semiconductor products whose surfaces are not etched with images.
[0258] Optionally, the merging unit 803 registers the two frames of regional images based on the first registration strategy to obtain a registered regional image, including:
[0259] Determine a displacement parameter representing the degree of misalignment between the two frames of regional images according to a correlation coefficient of the two frames of regional images;
[0260] The two frames of regional images are registered according to their displacement parameters to obtain two frames of registered regional images.
[0261] Optionally, the merging unit 803 determines a displacement parameter representing the degree of misalignment between the two frames of regional images according to a correlation coefficient of the two frames of regional images, including:
[0262] Determine the reference image and the image to be registered that overlap in position between two frames of regional images according to the overlapping pixels between two adjacent time-delay integration cameras;
[0263] The images to be registered are moved according to the multiple displacement amounts respectively, and a normalized correlation coefficient corresponding to each displacement amount is determined according to the reference image and the image to be registered after each movement;
[0264] The displacement amount whose corresponding normalized correlation coefficient satisfies the first condition is determined as a displacement parameter representing the degree of misalignment between the two frames of regional images.
[0265] Optionally, the first condition includes that a normalized correlation coefficient corresponding to the displacement parameter is a maximum value among multiple normalized correlation coefficients corresponding to multiple displacement amounts.
[0266] Optionally, the merging unit 803 registers the two frames of regional images based on the second registration strategy to obtain a registered regional image, including:
[0267] Determine the reference image and the image to be registered that overlap in position between two frames of regional images according to the overlapping pixels between two adjacent time-delay integration cameras;
[0268] Determine at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image;
[0269] The two frames of regional images are registered according to the reference feature points and the feature points to be registered, so as to obtain two frames of registered regional images.
[0270] Optionally, the merging unit 803 performs registration processing on the two frames of regional images according to the reference feature points and the feature points to be registered to obtain two frames of registered regional images, including:
[0271] Determine a feature point pair from the reference feature points and the feature points to be registered, the feature point pair includes a reference feature point and a feature point to be registered, and the similarity measure between the feature points included in the feature point pair satisfies the second condition;
[0272] The two frames of regional images are registered according to the reference feature points and the feature points to be registered contained in the feature point pair to obtain two frames of registered regional images.
[0273] Optionally, the merging unit 803 determines at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image, including:
[0274] Obtaining a reference gradient image corresponding to the reference image and a gradient image to be registered corresponding to the image to be registered, wherein the pixel value of each pixel in the reference gradient image represents the gradient value of the pixel at the corresponding position in the reference image, and the pixel value of each pixel in the gradient image to be registered represents the gradient value of the pixel at the corresponding position in the image to be registered;
[0275] At least one reference feature point of the reference image is determined according to the reference gradient image, and at least one to-be-registered feature point of the to-be-registered image is determined according to the to-be-registered gradient image.
[0276] Optionally, the merging unit 803 determines at least one reference feature point of the reference image according to the reference gradient image, including:
[0277] Processing the reference gradient image according to the two-dimensional Gaussian function to obtain a reference Gaussian scale image;
[0278] Determine at least one reference feature point of the reference image according to the scale parameter of each pixel in the reference Gaussian scale image;
[0279] Determining at least one feature point to be registered of the image to be registered according to the gradient image to be registered, comprising:
[0280] Processing the gradient image to be registered according to the two-dimensional Gaussian function to obtain the Gaussian scale image to be registered;
[0281] At least one to-be-registered feature point of the to-be-registered image is determined according to the scale parameter of each pixel in the to-be-registered Gaussian scale image.
[0282] Optionally, the merging unit 803 determines the similarity measure between the reference feature point and the feature point to be registered in the following manner:
[0283] Determine a plurality of reference descriptors corresponding to the reference feature points, and determine a plurality of to-be-registered descriptors corresponding to the to-be-registered feature points;
[0284] A similarity measure between the reference feature points and the feature points to be registered is determined according to the plurality of reference descriptors and the plurality of descriptors to be registered.
[0285] Optionally, the merging unit 803 determines multiple reference descriptors corresponding to the reference feature points, including:
[0286] Determine the main direction of the reference feature point according to the scale parameter of the reference feature point, and rotate the reference image according to the main direction of the reference feature point;
[0287] Determine a reference neighborhood corresponding to the reference feature point in the rotated reference image, and determine multiple reference descriptors of the reference feature point according to the scale parameter of each pixel in the reference neighborhood;
[0288] The merging unit 803 determines a plurality of descriptors to be registered corresponding to the feature points to be registered, including:
[0289] Determine the main direction of the feature point to be registered according to the scale parameter of the feature point to be registered, and rotate the image to be registered according to the main direction of the feature point to be registered;
[0290] A neighborhood to be registered corresponding to the feature point to be registered is determined in the rotated image to be registered, and a plurality of descriptors to be registered of the feature point to be registered are determined according to the scale parameter of each pixel in the neighborhood to be registered.
[0291] The specific working principle and beneficial effects of the image processing device provided in the embodiment of the present application can be found in the image processing method provided in the embodiment of the present application, and will not be described in detail.
[0292] An embodiment of the present application also provides a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the image processing method provided in any embodiment of the present application.
[0293] The present application also provides an electronic device. Fig. 9 , is a structural diagram of the electronic device, which may include a memory 901 and a processor 902; the memory 901 is used to store computer programs; the processor 902 is used to execute computer programs, specifically for implementing the image processing method provided in any embodiment of the present application.
[0294] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0295] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0296] Through the description of the above implementation methods, it can be known that the technicians in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.
[0297] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0298] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. An image processing method, characterized in that: include: Obtaining scattering rate parameters of multiple regions of the target object; According to the scattering rate parameters of each of the regions, the integration levels of the multiple mechanically spliced time-delay integration cameras corresponding to each of the regions when collecting images are configured, and each of the time-delay integration cameras is used to collect a regional image of the corresponding region; Registering the regional images captured by each of the time-delayed integration cameras to obtain a registered regional image; splicing the registered regional images to obtain a defect detection image of the target object; Wherein, each of the plurality of time-delayed integration cameras corresponds to a beam splitter prism, and the plurality of beam splitter prisms include a first beam splitter prism and a second beam splitter prism that are staggered and bonded; The first beam splitter prism has a coating that enhances transmission capability; The second beam splitter prism has a coating that enhances reflection capability; Wherein, the photosensitive surface of the time-delayed integration camera corresponding to the first beam splitter prism corresponds to the transmission image surface of the first beam splitter prism; The photosensitive surface of the time delay integration camera corresponding to the second beam splitter prism corresponds to the reflection image surface of the second beam splitter prism; The transmission image plane of the first beam splitter prism and the reflection image plane of the second beam splitter prism are orthogonal; Part of the pixels of the time delay integration cameras corresponding to every two adjacent beam splitting prisms overlap with each other.
2. The method according to claim 1, characterized in that The manner in which each of the time-delayed integration cameras collects the regional image includes: Determining a trigger shooting electrical signal synchronized with the moving speed of the target object; Sending the trigger shooting electrical signal to each of the time delay integration cameras through a trigger signal circuit to trigger each of the time delay integration cameras to obtain an image signal of the target object; The image signal output by each of the time-delayed integration cameras is modulated according to the trigger shooting electrical signal to obtain the regional image.
3. The method according to claim 1, characterized in that: The step of configuring the integration level of the time-delayed integration camera corresponding to each of the regions when acquiring images according to the scattering rate parameters of each of the regions comprises: For each of the regions, the integration order of the time-delayed integration camera corresponding to the region when acquiring images is configured according to a preset inverse relationship and a scattering rate parameter of the region.
4. The method according to claim 1, characterized in that: The registering the regional images captured by the time-delayed integration cameras to obtain the registered regional images includes: Determining whether the target object belongs to the first category or the second category; If the target object belongs to the first category, for every two frames of regional images acquired by two adjacent time-delayed integration cameras, register the two frames of regional images based on a first registration strategy to obtain a registered regional image; If the target object belongs to the second category, for every two frames of regional images acquired by two adjacent time-delay integration cameras, the two frames of regional images are registered based on a second registration strategy to obtain a registered regional image.
5. The method according to claim 4, characterized in that The target objects of the first category include semiconductor products whose surface graphic features have a significance greater than a certain threshold; The target objects of the second category include semiconductor products whose surface graphic features have a significance level less than or equal to the threshold.
6. The method according to claim 4, characterized in that The registering the two frames of the regional images based on the first registration strategy to obtain the registered regional image includes: Determine a displacement parameter representing the degree of misalignment between the two frames of the regional images according to a correlation coefficient of the two frames of the regional images; The two frames of the regional images are registered according to the displacement parameters of the two frames of the regional images to obtain two frames of registered regional images.
7. The method according to claim 6, characterized in that The step of determining the displacement parameter representing the degree of misalignment between the two frames of the regional images according to the correlation coefficient of the two frames of the regional images comprises: Determine a reference image and an image to be registered that overlap in position between two frames of the regional images according to overlapping pixels between two adjacent time-delay integration cameras; Moving the image to be registered according to a plurality of displacement amounts respectively, and determining a normalized correlation coefficient corresponding to each displacement amount according to the reference image and the image to be registered after each movement; The displacement amount whose corresponding normalized correlation coefficient satisfies the first condition is determined as a displacement parameter representing the degree of misalignment between the two frames of the regional images.
8. The method according to claim 7, characterized in that The first condition includes that the normalized correlation coefficient corresponding to the displacement parameter is a maximum value among a plurality of normalized correlation coefficients corresponding to the plurality of displacement amounts.
9. The method according to claim 4, characterized in that The registering the two frames of the regional images based on the second registration strategy to obtain the registered regional image includes: Determine a reference image and an image to be registered that overlap in position between two frames of the regional images according to overlapping pixels between two adjacent time-delay integration cameras; Determining at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image; The two frames of the regional images are registered according to the reference feature points and the feature points to be registered to obtain two frames of registered regional images.
10. The method according to claim 9, characterized in that The registering process of the two frames of the regional images according to the reference feature points and the feature points to be registered to obtain two frames of registered regional images includes: Determine a feature point pair from the reference feature points and the feature points to be registered, the feature point pair comprising one of the reference feature points and one of the feature points to be registered, and a similarity measure between the feature points included in the feature point pair satisfies a second condition; The two frames of the regional images are registered according to the reference feature points and the feature points to be registered contained in the feature point pairs to obtain two frames of registered regional images.
11. The method according to claim 9, characterized in that The determining of at least one reference feature point of the reference image and at least one to-be-registered feature point of the to-be-registered image comprises: Obtaining a reference gradient image corresponding to the reference image and a gradient image to be registered corresponding to the image to be registered, wherein the pixel value of each pixel in the reference gradient image represents the gradient value of the pixel at the corresponding position in the reference image, and the pixel value of each pixel in the gradient image to be registered represents the gradient value of the pixel at the corresponding position in the image to be registered; At least one reference feature point of the reference image is determined according to the reference gradient image, and at least one to-be-registered feature point of the to-be-registered image is determined according to the to-be-registered gradient image.
12. The method according to claim 11, characterized in that The determining at least one reference feature point of the reference image according to the reference gradient image comprises: Processing the reference gradient image according to a two-dimensional Gaussian function to obtain a reference Gaussian scale image; Determining at least one reference feature point of the reference image according to a scale parameter of each pixel in the reference Gaussian scale image; The step of determining at least one feature point to be registered of the image to be registered according to the gradient image to be registered comprises: Processing the gradient image to be registered according to a two-dimensional Gaussian function to obtain a Gaussian scale image to be registered; At least one to-be-registered feature point of the to-be-registered image is determined according to the scale parameter of each pixel in the to-be-registered Gaussian scale image.
13. The method according to claim 11, characterized in that The method of determining the similarity measure between the reference feature point and the feature point to be registered includes: Determine a plurality of reference descriptors corresponding to the reference feature points, and determine a plurality of to-be-registered descriptors corresponding to the to-be-registered feature points; A similarity measure between the reference feature points and the feature points to be registered is determined according to the multiple reference descriptors and the multiple descriptors to be registered.
14. The method according to claim 13, characterized in that The determining a plurality of reference descriptors corresponding to the reference feature points comprises: Determining a main direction of the reference feature point according to a scale parameter of the reference feature point, and rotating the reference image according to the main direction of the reference feature point; Determining a reference neighborhood corresponding to the reference feature point in the rotated reference image, and determining a plurality of reference descriptors of the reference feature point according to a scale parameter of each pixel in the reference neighborhood; The determining of a plurality of descriptors to be registered corresponding to the feature points to be registered comprises: Determine the main direction of the feature point to be registered according to the scale parameter of the feature point to be registered, and rotate the image to be registered according to the main direction of the feature point to be registered; A neighborhood to be registered corresponding to the feature point to be registered is determined in the rotated image to be registered, and a plurality of descriptors to be registered of the feature point to be registered are determined according to scale parameters of each pixel in the neighborhood to be registered.
15. An image processing system, characterized in that: At least includes an acquisition component and a data processing component; The acquisition assembly includes a plurality of time-delayed integration cameras, a first beam splitter prism and a second beam splitter prism bonded in an interlaced manner; The first beam splitter prism has a coating that enhances transmission capability; The second beam splitter prism has a coating that enhances reflection capability; Each of the time delay integration cameras corresponds to one of the beam splitting prisms; Wherein, the photosensitive surface of the time-delayed integration camera corresponding to the first beam splitter prism corresponds to the transmission image surface of the first beam splitter prism; The photosensitive surface of the time delay integration camera corresponding to the second beam splitter prism corresponds to the reflection image surface of the second beam splitter prism; The transmission image plane of the first beam splitter prism and the reflection image plane of the second beam splitter prism are orthogonal; Part of the pixels of the time delay integration cameras corresponding to every two adjacent beam splitting prisms overlap with each other; The data processing component is used to configure the integration series of multiple time-delay integration cameras of the acquisition component according to the method described in any one of claims 1 to 14, and process the regional images acquired by the multiple time-delay integration cameras to obtain the defect detection image of the target object.
16. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program, and when the computer program is executed, it is specifically used to implement the image processing method according to any one of claims 1 to 14.
17. A computer storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the image processing method according to any one of claims 1 to 14.
Citation Information
Patent Citations
Detection method and device based on TDI line-scan digital camera
CN113533343A
