Image stitching method and system in IC reverse engineering
By using sequence template matching method and selective stitching method in chip reverse analysis, and fusion using weighted averaging method, the problems of IC microscope image stitching accuracy and speed are solved, and efficient image stitching is achieved and shortened The chip design cycle is completed.
Patent Information
- Application Number
- CN202210552984.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-05-19
AI Technical Summary
In the existing chip reverse analysis technology, the IC microscope image splicing algorithm has poor accuracy and slow speed, resulting in miss splicing, extending the overall IC reverse engineering cycle and reducing the chip production speed.
Image registration is carried out based on sequence template matching method, and appropriate stitching paths are selected in combination with selective stitching method, and image fusion is realized through weighted averaging method to improve stitching accuracy and speed.
It improves the accuracy and speed of IC microscopic image splicing, shortens the overall IC reverse engineering cycle, speeds up the chip design, reduces the phenomenon of miss splicing, and achieves seamless splicing between images.
Smart Images

Figure CN114881859B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of chip reverse analysis engineering, and in particular relates to an image splicing method and system in IC reverse engineering. Background Art
[0002] Electronic products are everywhere in all fields of today's world. As the "heart" of electronic products, the demand for chips is increasing sharply, but the development and production speed of the chip industry still cannot meet the demand, and some high-end chips (such as CPUs) still rely on imports. As the core of the chip industry, chip design is the key to promoting the development of the chip industry. In order to improve corporate competitiveness, many companies adopt chip dissection and reverse analysis to learn advanced technologies and improve their own design capabilities.
[0003] For miniaturized chips, in order to obtain high-definition microscopic images, it is necessary to use an optical microscope to collect images. Due to the limitation of the field of view of the objective lens, the camera can only obtain a partial image of the chip each time. For each chip, it is necessary to collect multiple times in succession, and finally obtain the overall microscopic image of the chip through image stitching technology. The key technologies involved in image stitching mainly include image registration, image stitching and image fusion. Image registration is the most critical step, and the accuracy of registration directly affects the quality of stitching. Image fusion can eliminate stitching marks and obtain better stitching effects.
[0004] Chip reverse analysis engineering requires precise stitching of hundreds or thousands of chip microscopic images. IC microscopic images are large in scale, complex in scenes, and have small overlapping areas. Currently, there are few stitching algorithms directly for chip microscopic images in this field, and the algorithms have poor accuracy and slow speed. Direct use of common traditional image stitching algorithms will lead to incorrect stitching of IC microscopic images, which will undoubtedly prolong the overall IC reverse engineering cycle and reduce the chip production speed. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides an image stitching method and system in IC reverse engineering, which improves the stitching speed of IC microscopic images in IC reverse engineering while ensuring the stitching accuracy.
[0006] The present invention provides the following technical solutions:
[0007] The intelligent image stitching method in IC reverse engineering includes the following steps:
[0008] S1, taking the collected multiple IC microscopic images as input;
[0009] S2, image registration based on sequence template matching method;
[0010] S3, using the selective stitching method to select a suitable stitching path and perform image stitching;
[0011] S4, using weighted average method to perform image fusion and achieve seamless splicing between images;
[0012] S5. Output the stitched whole image.
[0013] Further, image registration is performed based on a sequence template matching method, including the following steps:
[0014] S21, selecting a template: A region as large as possible can be selected from the prior overlapping region of the selected reference image as a template.
[0015] S22, generate sequence templates. Perform grid segmentation on the template image T to generate sequence templates, that is, divide the template image T into a×b L template sub-images {T i} 0<i≤L .
[0016] S23, perform sequence template matching, and measure similarity between the sequence template and the corresponding window of the image to be registered. Map the obtained sequence template image to the search area S of the image to be registered to find matching points, and perform template matching multiple times. The criterion used for similarity measurement is the normalized product correlation criterion NCC.
[0017] S24: Obtain the best matching point.
[0018] Furthermore, the grayscale histogram variance method is used to first evaluate the information content of the template image, and then different splicing paths are selected according to the amount of information combined with the breadth-first search algorithm BFS and the union-find idea.
[0019] Furthermore, the image fusion using the weighted average method includes: for two IC microscopic images to be fused, each image only needs to be multiplied by the corresponding fusion weight, the sum of the two fusion weights is equal to one, and then the results are added to obtain the fused image.
[0020] An image stitching system in IC reverse engineering, the system comprising:
[0021] An image acquisition module is used to acquire two images to be stitched with overlapping areas, one of which is used as a reference image I1 and the other as an image to be registered I2;
[0022] The image registration module is used to perform image registration on the reference image I1 and the image to be registered I2 based on the sequence template matching method; specifically, it includes:
[0023] Select an area in the prior overlapping area of the reference image I1 as the template image T;
[0024] The template image T is divided into L template sub-images {T i} 0<i≤L ;
[0025] According to the grayscale histogram variance of the template sub-image, L template sub-images {T i} 0<i≤L Sorting, generating sequence template {T(k)} 0<k≤L ;
[0026] Based on the image to be registered I2, the search area S is obtained, and the set of points to be matched P = {p(i, j)|(i, j)∈S} is obtained;
[0027] Map the sequence templates to the search area S in turn to find the best matching point:
[0028] Traverse the points to be matched and calculate the similarity between the sequence template and the corresponding area of the points to be matched;
[0029] Eliminate the matching points whose similarity is lower than the similarity threshold to shrink the matching area;
[0030] Calculate the similarity between the next sequence template and the corresponding area of the remaining points to be matched, and continue to eliminate the points to be matched until the number of points to be matched is 1, and obtain the best matching point;
[0031] An image stitching module is used to stitch the reference image I1 and the image to be registered I2 using a selective stitching method based on the best matching point;
[0032] The image fusion module is used to perform image fusion using a weighted average method to obtain a spliced image of two images to be spliced.
[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0034] 1) The intelligent image stitching method and system in IC reverse engineering analyzes the characteristics of IC microscopic images and uses a regional grayscale-based image registration method to perform IC microscopic image registration: In view of the problem that the traditional template matching method has a large amount of unnecessary calculations in the matching process, a sequence template matching method is improved and proposed on this basis to achieve IC microscopic image registration; by dividing the template image into blocks and setting priorities for the divided sub-templates according to the amount of information in the region to generate a sequence template, the sub-templates are matched in sequence during template search, which greatly reduces the amount of calculation, thereby improving the efficiency of image registration.
[0035] 2) The intelligent image stitching method and system in IC reverse engineering evaluates the information content of the template image and detects the peak value of the image registration result, and adopts a selective stitching method to select a reasonable stitching path to avoid misregistration caused by unclear features of the overlapping area or the existence of periodic repetitive structures. It can solve the problem of poor panoramic image stitching effect caused by local registration errors, thereby improving the accuracy of image stitching.
[0036] 3) The intelligent image stitching method and system in IC reverse engineering adopts weighted average method to fuse images, which can eliminate the obvious seam marks caused by brightness difference between images at the stitching point, so that a smooth transition can be achieved between multiple stitched images. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a flow chart of the image stitching method of the present invention;
[0038] Figure 2 This is a schematic diagram of image registration based on sequence templates of the present invention;
[0039] Figure 3 Schematic diagram of template segmentation in the sequence template matching method of the present invention;
[0040] Figure 4 A schematic diagram of a template and a search area for the template matching method of the present invention;
[0041] Figure 5 This is a flow chart of the sequence template matching method of the image registration algorithm of the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0043] The modern chip industry is in a highly competitive market, with rapid product iterations and updates, and we must keep up with the times. If we want to speed up development, we need to improve our chip design capabilities. Nowadays, many companies use chip reverse analysis to learn advanced design technologies and shorten production cycles. The key to chip reverse analysis engineering is to seamlessly stitch a large number of chip microscopic images, and then process and analyze them. Although there have been many research results on chip microscopic image stitching algorithms, there are still shortcomings and deficiencies. Many methods can only be applied to specific scenarios, and the stitching accuracy and speed need to be improved.
[0044] The present invention proposes an intelligent splicing method for IC microscopic images by deeply analyzing the characteristics of chip images. The method uses a sequential template matching method to improve the efficiency of image registration, combines a selective splicing strategy to improve the splicing accuracy, and finally uses a weighted average method to achieve seamless fusion of images. The present invention can improve the splicing accuracy and the splicing efficiency of microscopic images in IC reverse engineering, thereby shortening the overall IC reverse engineering cycle and accelerating the speed of chip design.
[0045] The present invention provides an image splicing method in IC reverse engineering, such as Figure 1 and Figure 5 As shown, the method comprises the following steps:
[0046] S1, obtaining two images to be stitched with overlapping areas, one as the reference image I1, and the other as the image to be registered I2;
[0047] S2, based on the sequence template matching method, performing image registration on the reference image I1 and the image to be registered I2; specifically including:
[0048] S21, selecting an area in the prior overlapping area of the reference image I1 as a template image T;
[0049] S22, divide the template image T into L template sub-images {T i} 0<i≤L ;
[0050] S23, according to the grayscale histogram variance of the template sub-image, L template sub-images {T i} 0<i≤L Sorting, generating sequence template {T(k)} 0<k≤L ;
[0051] S24, based on the image to be registered I2, the area to be searched S is obtained, and a set of points to be matched P = {p(i,j)|(i,j)∈S} is obtained;
[0052] S25, mapping the sequence templates to the search area S in turn to find the best matching point:
[0053] Traverse the points to be matched and calculate the similarity between the sequence template and the corresponding area of the points to be matched;
[0054] Eliminate the matching points whose similarity is lower than the similarity threshold to shrink the matching area;
[0055] Calculate the similarity between the next sequence template and the corresponding area of the remaining points to be matched, and continue to eliminate the points to be matched until the number of points to be matched is 1, and obtain the best matching point;
[0056] S3, based on the best matching point, using the selective stitching method to stitch the reference image I1 and the image to be registered I2;
[0057] S4. Use the weighted average method to perform image fusion to obtain a stitching image of the two images to be stitched.
[0058] The intelligent image stitching method in IC reverse engineering according to an embodiment of the present invention comprises the following steps:
[0059] For two images to be stitched that have overlapping areas, one is used as the reference image and the other is used as the image to be registered.
[0060] S1: IC microscopic image registration based on sequence template, specifically including the following steps:
[0061] S11, select a template, and select a region as large as possible in the prior overlapping area of the reference image as the template. Taking the adjacent images in the horizontal direction as an example, let the width and height of the reference image I1 and the image to be registered I2 be W and H respectively, as follows: Figure 2 As shown in the figure, the width and height of the a priori overlap area are w and h respectively. Since adjacent images in the horizontal direction ideally only have translational motion, h = H. Image I1 is used as the reference image, and a template image is constructed in the a priori overlap area of I1. Considering that the actual overlap area and the a priori overlap area have a certain deviation due to the stepping accuracy of the motor equipped with the acquisition lens, the perturbation amount Δr is set in the x and y directions. x and Δr y , assuming the size of the template image T is M×N, then M and N can be calculated by the following expressions:
[0062]
[0063] S12. Perform similarity measurement, using the Normalized Cross Correlation (NCC) as the similarity measurement criterion. The larger the calculated NC3(i, j), the higher the similarity between the template and the corresponding window. Suppose the movement of the template in the image to be registered is i, j, then the calculation expression of NCC(i, j) is as follows:
[0064]
[0065] S13, generate a sequence template, and perform grid segmentation on the template image T. Assume that the template image T is equally divided into a×b L template sub-images {T i} 0<i≤L , T i The size is m×n, and the template block diagram is as follows Figure 3 As shown. m, n represent the template subgraph T i The size of k represents the kth grayscale, c k Indicates the number of pixels in the kth grayscale in the area, Pk Indicates the frequency of the k-level grayscale appearing in the region. Template subgraph T i The histogram can be expressed as follows:
[0066]
[0067] So we can get the template subgraph T i The mean μ and variance δ of the histogram 2 They are as follows:
[0068]
[0069]
[0070] Therefore, according to the grayscale histogram variance of the template sub-image from large to small, {T i} 0<i≤L Sorting can generate a sequence template {T(k)} 0<k≤L .
[0071] S14, perform sequence template matching, map the template image to the search area S of image I2 to find matching points, that is, perform template search and template matching, such as Figure 4 As shown. The subgraph covered by template T each time is denoted as S i,j , (i, j) is the coordinate of the upper left corner vertex of the subgraph in the search area S. Sequence template matching is a multiple search process, the specific flow chart is as follows Figure 5 As shown, each search only allows one template subgraph T(k) to participate in the calculation, and the set of points to be matched is P = {p(i,j)|(i,j)∈S};
[0072] S15, move the template T to point P(i, j), and calculate the similarity value D(i, j) between the template subgraph T(k) and the corresponding window;
[0073] S16. After one round of searching, if D(i, j) is less than the threshold β, remove point p(i, j) from the set of points to be matched P. Let the maximum value in the matching result be D max , the minimum value is D min , then the threshold β is defined as follows:
[0074]
[0075] S17. If there is only one point left in the set of points to be matched P or k = L, then the template matching is terminated. Otherwise, let k = k + 1 and continue the registration. Table 1 is Figure 5 Pseudocode written as a flowchart of the sequence template matching method.
[0076] Table 1 Sequence template matching method
[0077]
[0078] S2. Select a suitable splicing path based on the selective splicing method to splice the images. The specific steps include the following:
[0079] S21. First, select templates for all images and evaluate the information content of the template images using the variance method of the grayscale histogram to determine whether the information content is rich.
[0080] S22. Then, use the breadth-first search algorithm BFS to traverse the microscopic images to be spliced. Each image attempts to be spliced with its adjacent images in the four directions of up, down, left, and right. If the information content is insufficient or there are multiple peaks in the matching result in a certain direction, it means that the path is blocked.
[0081] S23. During the splicing process, use the idea of the union-find set to group the spliced microscopic images into a set. However, during the splicing process, there may be a situation where the sets cannot be merged normally because the information content is insufficient or there are multiple peaks in the matching result, resulting in the two sets not being able to be merged normally. Attempt to splice the boundary images, and the following two situations may occur: The first situation is that there are multiple peaks in the matching result, and the peak points are far apart. The second situation is that there are multiple peaks in the matching result, and the peak points are close to each other. For the first situation, based on the already spliced images, determine whether there is a reasonable peak point among the peak points. If there is a reasonable peak point, use this point as the registration point. In other cases, complete the splicing through manual intervention.
[0082] S3. Perform image fusion based on the weighted average method. Let I1(x, y) and I2(x, y) be the two microscopic images to be fused, then the fused image g(x, y) is:
[0083]
[0084] where w1 and w2 are the fusion weights, and they satisfy w1 + w2 = 1, 0 < w1, w2 < 1. After image fusion, the seams caused by the brightness difference between images can be eliminated, and a smooth transition between images can be achieved.至此, the entire process of image splicing is completed.
[0085] The present invention provides a method and system that can improve the splicing speed of IC microscopic images in IC reverse engineering while ensuring the splicing accuracy, making the human-computer interaction in the process of IC microscopic image splicing more convenient, shortening the overall cycle of IC reverse engineering, and improving the efficiency of IC reverse engineering.
[0086] It will be easily understood by those skilled in the art that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An image stitching method in IC reverse engineering, characterized in that: The method comprises the following steps: S1, obtaining two images to be stitched with overlapping areas, one as the reference image I1, and the other as the image to be registered I2; S2, based on the sequence template matching method, performing image registration on the reference image I1 and the image to be registered I2; specifically including: S21, selecting an area in the prior overlapping area of the reference image I1 as a template image T; S22, divide the template image T into L template sub-images {T i } 0<i≤L ; S23, according to the grayscale histogram variance of the template sub-image, L template sub-images {T i } 0<i≤L Sorting, generating sequence template {T(k)} 0<k≤L ; S24, based on the image to be registered I2, the area to be searched S is obtained, and a set of points to be matched P = {p(i,j)|(i,j)∈S} is obtained; S25, mapping the sequence templates to the search area S in turn to find the best matching point: Traverse the points to be matched and calculate the similarity between the sequence template and the corresponding area of the points to be matched; Eliminate the matching points whose similarity is lower than the similarity threshold to shrink the matching area; Calculate the similarity between the next sequence template and the corresponding area of the remaining points to be matched, and continue to eliminate the points to be matched until the number of points to be matched is 1, and obtain the best matching point; S3, based on the best matching point, using the selective stitching method to stitch the reference image I1 and the image to be registered I2; S4. Use the weighted average method to perform image fusion to obtain a stitching image of the two images to be stitched.
2. The image stitching method in IC reverse engineering according to claim 1, characterized in that: The specific method of selecting an area in the prior overlapping area of the reference image I1 as the template image T is as follows: Assuming that the width and height of the prior overlapping area of the reference image I1 are w and h respectively, and the image acquisition direction is the image width direction, the template image T is obtained by the following formula: Where M and N are the width and height of the template image T, respectively, and Δr x and Δr y are the disturbances in the width and height directions respectively.
3. The image stitching method in IC reverse engineering according to claim 1, characterized in that: Perform grid segmentation on the template image T and divide the template image T into L equal template sub-images.
4. The image stitching method in IC reverse engineering according to claim 1, characterized in that: Template subgraph T i The grayscale histogram variance δ 2 The calculation formula is as follows: Where m and n represent the template subgraph T i The size of k represents the kth grayscale, k′ represents the total number of grayscale levels, C k Represents the template subgraph T i The number of pixels at the kth gray level, P k Represents the template subgraph T i The frequency of occurrence of the kth grayscale level, μ represents the mean.
5. The image stitching method in IC reverse engineering according to claim 1, characterized in that: The a priori overlapping area of the image to be registered I2 is used as the area S to be searched.
6. The image stitching method in IC reverse engineering according to claim 1, characterized in that: The normalized product correlation criterion is used to calculate the similarity D(i, j) between the sequence template and the area corresponding to the point p(i, j) to be matched.
7. The image stitching method in IC reverse engineering according to claim 6, characterized in that: The calculation formula of the similarity threshold β is as follows: Where D max Indicates the maximum value of similarity, D min Indicates the minimum value in similarity.
8. The image stitching method in IC reverse engineering according to claim 1, characterized in that: Use the breadth-first search algorithm BFS and the union-find idea to select the splicing path.
9. The image stitching method in IC reverse engineering according to claim 1, characterized in that: I1(x,y) and I2(x,y) are two microscopic images to be fused, and the fused image g(x,y) is: Where w1 and w2 are fusion weights, which satisfy w1+w2=1,0 <w1,w2<1。 10. An image stitching system in IC reverse engineering, characterized in that: The system includes: An image acquisition module is used to acquire two images to be stitched with overlapping areas, one of which is used as a reference image I1 and the other as an image to be registered I2; The image registration module is used to perform image registration on the reference image I1 and the image to be registered I2 based on the sequence template matching method; specifically, it includes: Select an area in the prior overlapping area of the reference image I1 as the template image T; The template image T is divided into L template sub-images {T i } 0<i≤L ; According to the grayscale histogram variance of the template sub-image, L template sub-images {T i } 0<i≤L Sorting, generating sequence template {T(k)} 0<k≤L ; Based on the image to be registered I2, the search area S is obtained, and the set of points to be matched P = {p(i, j)|(i, j)∈S} is obtained; Map the sequence templates to the search area S in turn to find the best matching point: Traverse the points to be matched and calculate the similarity between the sequence template and the corresponding area of the points to be matched; Eliminate the matching points whose similarity is lower than the similarity threshold to shrink the matching area; Calculate the similarity between the next sequence template and the corresponding area of the remaining points to be matched, and continue to eliminate the points to be matched until the number of points to be matched is 1, and obtain the best matching point; An image stitching module is used to stitch the reference image I1 and the image to be registered I2 using a selective stitching method based on the best matching point; The image fusion module is used to perform image fusion using a weighted average method to obtain a spliced image of two images to be spliced.
