Chip navigation using virtual maps, including sample navigation based on hand-painted or digitally generated images

Abstract

A computer-implemented method, the method comprising: accessing an image of a sample; detecting one or more shapes in the image; constructing a virtual map based on the detected one or more shapes in the image and a set of virtual map construction rules; and automatically navigating to a target area of the sample based on the virtual map to process the sample using a charged particle beam microscope. Another method includes accessing dynamically obtained shape data associated with an image of a sample; detecting one or more shapes in the image and converting the detected one or more shapes into found shape data; accessing artificial reference shape data; comparing the found shape data with the artificial reference shape data by object matching optimization to determine an alignment between the found shape data and the artificial reference shape data; and navigating to a target area of the sample based on the alignment.

Description

Technical Field

[0001] This field relates to the processing and automation of microscopic samples using charged particle beam microscopy. Background Technology

[0002] Automated charged particle beam microscopy (e.g., automated transmission electron microscopy) is a field of technology involving the continuous development of automated technologies and related systems for handling microscopic samples. These samples often involve cutting-edge technologies, such as the development or fabrication of novel, more complex microchips, or the exploration and analysis of the characteristics of cryopreserved biological samples. However, a common challenge in automating technologies at such a microscopic scale lies in the accurate identification of the target and the efficiency with which the identified target can be processed. Unfortunately, current methods fall short in both aspects and may require significant user intervention, which is both costly and time-consuming. Improved devices and technologies capable of addressing at least some of these shortcomings remain in demand. Summary of the Invention

[0003] The disclosed examples include techniques and charged particle beam microscopy employing these techniques, in which a virtual map can be automatically constructed for a portion of a sample being imaged. This construction can be performed by detecting basic parametric shapes and relationships within the imaging portion according to user-defined rules. The virtual map can then be used to automatically navigate the charged particle beam to selected targets within the imaging portion of the sample, allowing various processing steps to be performed on those selected targets.

[0004] According to one aspect of the disclosed technology, a computer-implemented method includes: accessing an image of a sample; detecting one or more shapes in the image; constructing a virtual map based on the detected one or more shapes in the image and a set of virtual map construction rules; and automatically navigating to a target region of the sample based on the virtual map to process the sample using charged particle beam microscopy. In some examples, accessing an image of the sample includes viewing a region of the sample using charged particle beam microscopy. Some examples also include automatically navigating to a subsequent target region of the sample based on the virtual map to process the sample using charged particle beam microscopy at the subsequent target region. In some examples, constructing the virtual map includes accessing an embedded virtual map definition. In some examples, constructing the virtual map includes accessing user-defined text-based virtual map generation rules. In some examples, constructing the virtual map includes interacting with a graphical user interface to draw shapes depicting one or more target structures. In some examples, constructing the virtual map includes interacting with a graphical user interface to select and arrange shapes depicting one or more target structures. Some examples also include accessing a processing scheme for processing a sample containing a target region, wherein the scheme is combined by selecting map targets from a predefined list of targets. Some examples also include access to a processing scheme for processing samples containing target areas, wherein the scheme is combined by selecting map targets from a virtual map representation. Some examples also include access to a processing scheme for processing samples containing target areas, wherein the scheme is combined by selecting map targets based on a reference image. In some examples, selecting map targets based on a reference image includes selecting a default target location on the reference image. In some examples, selecting map targets based on a reference image includes selecting a user-selected precise target location on the reference image. Some examples also include access to a processing scheme for processing samples containing target areas, wherein the scheme is combined by: selecting target locations in a virtual map arrangement; determining the shape closest to the selected target location based on Euclidean distance; and storing an offset vector between the selected target location and the determined nearest shape in the scheme, wherein automatic navigation to the target area of ​​the sample includes locating the determined nearest shape in the constructed virtual map and navigating to the target area using the stored offset. In some examples, constructing a virtual map based on one or more shapes detected in an image and a set of virtual map construction rules includes: performing edge detection on the image; detecting shapes in the edge detection results; comparing the arrangement of shapes defined by the virtual map construction rules with the arrangement of detected shapes; and selecting a registration scheme for the virtual map relative to the image accessed with the highest likelihood.In some examples, constructing a virtual map based on one or more shapes detected in an image and a set of virtual map-building rules includes: accessing edge or shape detection associated with the accessed image and the virtual map-building rules; constructing a shape point map connected by orientation and adjacency for shape detection and a virtual map defined by the virtual map-building rules; determining candidate anchors based on the maximum likelihood estimation scores of the detected shapes, where candidate solutions are defined as pairings of anchor candidates with corresponding graphics; and for one or more candidate solutions, comparing their candidate graphics with corresponding virtual map graphics to determine the closest match. Some examples include a charged particle beam microscope configured to process samples according to any of the examples described herein. Some examples include a computer-readable medium configured with stored processor-executable instructions for performing any of the examples described herein.

[0005] According to another aspect of the disclosed technology, a method includes: accessing dynamically acquired shape data associated with an image of a sample; detecting one or more shapes in the image and converting the detected one or more shapes into found shape data; accessing artificial reference shape data; comparing the found shape data with the artificial reference shape data through object matching optimization to determine an alignment between the found shape data and the artificial reference shape data; and based on this alignment, navigating to a target region of the sample to process the sample using charged particle beam microscopy. In some examples, the artificial reference shape data is based on source data including any of the following:

[0006] Images extracted using hand-drawn, digitally drawn, or algorithmically extracted shape segmentation; CAD and / or GDS files extracted using shape analysis; and / or text-based lists. Some examples include charged particle beam microscopes configured to process samples according to any of the examples described herein.

[0007] The foregoing and other objects, features and advantages of the disclosed technology will become more apparent from the following detailed description with reference to the accompanying drawings. Brief description of the attached figures

[0008] Figure 1 It is a flowchart of automated processing methods based on various publicly available examples.

[0009] Figure 2 This is a flowchart illustrating an example of defining a virtual map that can be used in charged particle beam processing.

[0010] Figure 3 This is a flowchart illustrating examples of processing schemes that can be used in charged particle beam processing.

[0011] Figure 4This is a flowchart illustrating an exemplary construction of a virtual map.

[0012] Figure 5 This is a flowchart of a selected example of building a virtual map.

[0013] Figure 6 This is a flowchart of an exemplary method for navigating to and processing a sample using a charged particle beam microscope.

[0014] Figure 7 This is a schematic diagram illustrating an example of mapping an image onto a virtual map.

[0015] Figure 8 This is a flowchart of an exemplary method for sample navigation using input reference image data.

[0016] Figure 9 This is a flowchart of various examples used to obtain artificial reference image data.

[0017] Figure 10 This is a flowchart of an exemplary shape-finding method.

[0018] Figure 11 This is a flowchart of an exemplary NM matching method.

[0019] Figure 12 This is a flowchart of another exemplary NM matching method.

[0020] Figure 13 This is a schematic diagram of a general example of a suitable computing environment in which the innovation can be implemented.

[0021] Figure 14 This is an exemplary system including charged particle beam microscopes and / or optical microscopes. Detailed Implementation

[0022] Automation of microscopy techniques can involve navigating a target beam to various locations on a microscopic sample, such as a semiconductor chip. For example, this could allow users to automatically prepare sheet samples from specific locations on a larger sample for further analysis or processing, such as using electron or ion beams in transmission electron microscopy (TEM), focused ion beam (FIB), deposition tools, or other instruments. Processing may require imaging, milling, extraction, bonding, and combinations thereof. For many applications, highly precise navigation on chips or other samples is required to process specific target areas, such as producing sample sheets via beam milling. This typically involves active imaging of the sample so that the acquired images can be used as part of the navigation process. For example, many current solutions use reference images of the sample and compare dynamically acquired images to these reference images, often referred to as “template matching.” The algorithm can then find the location on the sample that best matches the reference template image so that processing can begin at the found location. This matching can be accomplished using a variety of computer vision methods, ranging from simple cross-correlation to neural network solutions.

[0023] However, these current methods have many drawbacks. A serious problem stems from the need to store a large number of automated schemes for processing various samples. Specifically, acquiring reference images can be very time-consuming and typically requires significant storage capacity. Furthermore, the algorithmic processing requirements for comparison can consume substantial processor resources. Another complicating issue is the high failure rate of reference image comparisons on periodic samples, as the target appearance may not be specific enough to form a match. To reduce the number of schemes, template images are often shared between chip locations, manufacturing steps, and designs. However, this can further reduce the robustness of template matching methods due to the increased appearance mismatches.

[0024] As will be discussed further below, the disclosed examples eliminate the burden of creating and maintaining a large and storage-intensive library of template images for various chip samples by constructing virtual maps using basic shapes and / or rules rather than relying on specific appearances. Such methods can reduce the failure rate associated with template matching techniques and can reduce errors that are periodically associated with chip samples. Based on the understanding that existing template matching methods that rely on direct or training image comparisons are severely lacking in awareness of the scene being imaged during navigation.

[0025] Therefore, in many examples, the disclosed methods can replicate certain characteristics of a human operator's manual workflow in some respects. For example, although manual workflows are much slower, operators tend to focus on certain features, such as key structures and shapes visible through a microscope's field of view. This focus allows human operators, whose overall operating speed may be much slower than automated processes, to navigate relatively quickly even under human-based limitations, and with significantly higher accuracy in locating targets. Human operators can see the scene and quickly distinguish various macroscopic features, and can relatively easily distinguish certain other features (such as contamination) without being misled by them. This ability allows such template recognition to be extended to different samples, layouts, and processes with fundamental commonalities (e.g., in terms of physical proportions), even if the appearance and other characteristics of different samples may vary significantly. In contrast, existing template matching algorithms tend to be confused by variations in appearance, thus reducing the success rate of navigation.

[0026] The periodicity of sample features is particularly problematic in automated microscopy processing because many locations may visually resemble the desired target, but a more precise understanding of the scene may be required to select the correct location. For example, with current methods, the appearance differences between samples may necessitate that users prepare and acquire reference images for each type of sample, including at every step in several manufacturing steps, to achieve reasonable navigation reliability. Simultaneously, users may prefer to reuse processing templates as much as possible. However, with existing methods, reusing templates across different manufacturing steps leads to performance degradation, as navigation reliability decreases accordingly.

[0027] Furthermore, existing methods lack the ability to identify sample characteristics, whether on the same sample or different samples. These samples may have different physical dimensions but share one or more common properties, such as scale. One example is a rectangular array that scales up or down proportionally. More generally, different chip samples may differ significantly in shape, size, etc., but may share similarities in scale, array structure, spacing, and / or scale. Existing methods fail to identify commonalities where certain invariant properties exist between different samples, thus often requiring new templates even when commonalities exist. Similarly, preparing and storing additional templates can be cumbersome for users. The ability to identify such invariants could be beneficial, as it would allow the use of a single template across many different samples sharing some fundamental characteristics.

[0028] As discussed above, the disclosed techniques and apparatus can, in some respects, algorithmically simulate decisions made by a human operator, where the operator has a higher level of understanding of the scene being viewed—for example, knowing which features, such as shapes and / or patterns, to look for in the field of view, and what is unimportant, such as negligible contamination or unique structures. For instance, an operator can understand certain important features of a chip layout in order to correctly locate the desired target. However, this common-sense approach can be extended to various types of microscope samples and manufacturing processes involving automated microscopy techniques, provided that the relevant features can be described by geometries and their corresponding relationships, which constitute a pattern of sample-specific layout characteristics.

[0029] Figure 1 An exemplary method 100 for automating the processing of microscopic samples includes navigating on the sample to target features for processing. Prior to automation, a scheme can be created that includes various steps, rules, and parameters that can guide the automated processing. For example, scheme creation may involve selecting beam processing characteristics, including but not limited to electron or ion beam intensity, focus variation, scan path, pulse timing and beam duration, spot size, beam sequence, stage movement, beam repetition, etc. Navigation commands associated with selecting and locating various target features on the sample to be processed can also be included as part of the scheme or along with scheme selection for automated processing. For example, at 102, a scheme navigation target for automated processing can be established. As part of the automated processing of the sample, at 104, virtual map building rules can be defined by the user and / or accessed from memory, for example, as a reference to or part of the selected scheme. During sample processing, at 106, the virtual map building rules can be dynamically used to build a virtual map using the virtual map definition defined / accessed at 104 and the region of the sample being processed being viewed at 108. Virtual map building rules can have more general characteristics than existing processor- and memory-intensive template matching methods, and can therefore complement or better form the basis for replacing these navigation tools.

[0030] For example, a user can arrange a sample in an electron or ion beam chamber, such as by coupling the sample to a moving stage. The sample can be positioned within the field of view of the charged particle beam and can be manually or automatically adjusted by the user to view selected areas of the sample where target features may be found. In response to viewing an area of ​​the sample at 108, a virtual map can be constructed at 106. Construction may include analyzing an image of the viewed area to find target structures corresponding to construction rules. These target structures may include simple geometries that can be combined to form patterns specific to the sample type, such as a particular chip layout or series of chip layouts. Such patterns can be described by precise definitions, such as a set of specially positioned circles, rectangles, or other shapes, or by generation rules, such as an array of rectangles with predetermined spacing. Analysis can use various techniques to identify shapes and / or patterns, such as using various edge detection techniques to locate the edges of basic shapes and / or patterns, and using probability functions (such as maximum likelihood estimation (MLE)) to determine matches for specific shapes. The virtual map can then be aligned with the viewed area by comparing the arrangement of virtual maps defined by the construction rules with the arrangement of the found shapes.

[0031] The virtual map can then be used as an overlay within the field of view of the viewed area to navigate to the programmatic navigation target at 102. In some examples, the virtual map can be overlaid on the image of the field of view like a mask, or more typically, the virtual map can be a stored map used to guide automated processing to the desired navigation target. For example, at 110, using the constructed virtual map, the view and / or charged particle beam can be navigated to one of the defined programmatic targets, and the sample can be processed (imaging, milling, etc.) with respect to the defined programmatic target (which could be located at a target position, offset relative to that position, or along a path about that target position). Multiple targets can then be navigated to the constructed virtual map and processed based on it. For example, at 112, if another target on the virtual map needs to be processed, the process of navigating to the next position and processing the beam at the new position can be repeated at 110. If there are no more targets in the current view, at 114, the automated process can determine whether to view another area of ​​the sample for processing. The process of processing other areas and targets on the sample can be done by looking at another area at 108 (e.g., by manual or automated adjustment).

[0032] Navigation accuracy can be improved by using a virtual map constructed at 106 points in automated microscopy processes like AutoTEM. Virtual maps can also reduce protocol time for automated semiconductor chip analysis. For example, in automated chip sample analysis in a TEM, a wafer needs to be extracted from a precise location on the chip (selected by the user). This can be performed using small dual-beam and large dual-beam (SDB / LDB) systems to navigate, mill, extract, and solder the wafer onto individual microneedle stages inserted into the TEM. In a typical manual workflow, a human operator navigates the sample by imaging it with a beam of charged particles (typically an electron beam, but in some examples, an ion beam). The operator then efficiently navigates to specific target locations on the chip using visible salient structures, ignoring less important details and potential contamination. These target structures can correspond to various prominent features of the chip layout, such as semiconductor cell matrices, NAND cells, etc., and can be described by simple geometries. The disclosed examples (such as exemplary method 100) can leverage these basic geometries for navigation by defining and using virtual maps, such as a virtual map constructed at 106 points.

[0033] Figure 2Various examples 200 are shown that can be used alone or in combination to define a virtual map 202, such as for defining or accessing virtual map building rules at 104 of example 100, and in conjunction with other examples disclosed herein. In many examples, a user can enjoy a simplified process at 204 by embedding (e.g., hard-coding) the virtual definition into an application or scheme used during the automation process and accessing the virtual map definition during the automation process. In some examples of such examples, a set of scheme objectives can be provided to the user for selection so that the navigation portion of the automated microscopy process can be customized to some extent. In some examples, at 206, text-based or other prompts can be provided to allow the user to create user-defined text-based virtual map generation rules that summarize the basic shape and construction of the virtual map, such as an M x N array with selected shapes and spacing. In an additional example, at 208, interface code can be provided so that the user can interact with a graphical user interface (GUI) to select basic shapes and arrange them into a pattern. Then, at 202, a virtual map can be defined using a user-arranged set of basic shapes (such as the spacing relationships between different shapes, lines, etc.), which can be scalable. In other examples, at 210, interface code can be provided so that the user can interact with the GUI to draw basic shapes and arrange them into a pattern that can then be used as the basis for the virtual map. Example 200 is not exhaustive, and it should be understood that virtual maps can be defined in other ways. At 212, the virtual map can then be used in charged particle beam processing, for example, to navigate to a selected target in the sample.

[0034] Figure 3 Various examples 300 are shown at 302 that can be used to select a navigation target in a combination of processing schemes. Example 300 may correspond to the method used at 102 of example 100 and other examples disclosed herein for establishing a scheme navigation target. Once established, the processing scheme can be virtually configured at 304. Figure 1 This is used to navigate to a target on the sample during charged particle beam treatment. In some examples, at 306, a scheme target can be selected from a set of predefined targets, such as from a drop-down menu in the GUI. In some examples, at 308, a map target can be selected from a virtual map representation displayed to the user, for example, using a pointing device or touch-sensitive screen for easy selection. In some cases, a virtual map representation can be selected from one of several stored virtual maps. This selection can be for a specific predefined target, such as a specific shape and a specific location relative to the shape on the virtual map representation.

[0035] In some examples, at point 310, map targets can be selected based on a reference image of the sample. This can be achieved in various ways, such as by selecting a default or predetermined target location on the reference image at point 312. Such default locations can be defined using various disclosed techniques for constructing virtual maps of the viewed area of ​​the sample. For example, the reference image can be analyzed for target structures corresponding to rules defined in the virtual map. For instance, for virtual maps with a selected width-to-spacing ratio in the width direction and a selected height-to-spacing ratio in the height direction... Figure 9 A 9x9 rectangular array. The reference image can be processed using an edge detection procedure to identify candidates for identifying the 9x9 array, and an MLE procedure can be used to determine the most likely match. Once found, a default position can be assigned based on the mapping, corresponding to a navigation target for the combined scheme. In other examples, at 314, the user can use a GUI (e.g., using a pointing device or touch-sensitive screen) to select a specific target position on the reference image. The selected position can be assigned based on the mapping, corresponding to a separate or additional navigation target for the combined scheme. In some examples, the reference image can be an artificial reference image, such as a hand-drawn, digitally constructed, CAD file, etc.

[0036] Figure 4 An exemplary method 400 is shown for constructing a virtual map based on an image, such as an image viewed at 108 of exemplary method 100 or a reference image used at 310 in example 300 in a combination of processing schemes. Such construction may also be used in other examples disclosed herein, or in combination with other examples disclosed herein. At 402, an image can be accessed. This may correspond to an image stored in memory, which may be a reference image that is not an image of the sample to be processed, but rather an image of a different but similarly designed sample or object. In other examples, the accessed image may correspond to an image of a sample actively viewed for processing, for example, using a charged particle beam microscope.

[0037] At position 404, an edge detection procedure is applied to the image, and at position 406, edges and shapes in the image are detected. Various edge detection techniques well understood by those skilled in the art can be applied, including the Canny algorithm, the Sobel operator, the Hough transform, neural networks, etc. In a specific example involving DRAM samples, a cell matrix that can be composed of rectangles is detected by first estimating the image intensity gradient using the Sobel operator. Horizontal edges are detected by analyzing the horizontal proximity of each pixel. If the gradients of a sufficient number of neighboring pixels are approximately aligned with the Y-axis, the center pixel is considered a horizontal edge. A similar procedure is performed for vertical edges. This gradient can be called orientation consistency, which can increase the detection recall of low-contrast images (which inevitably occurs in samples from certain manufacturing steps). If the image has high contrast, a standard procedure of gradient magnitude thresholding can be performed. Automatic physical size inference can be accomplished by selecting peaks in the run-length encoded histogram of the edge mask, using row-priority for rectangle width and column-priority for rectangle height. This peak selection can be skipped if the rectangle dimensions are provided beforehand. Rectangle detection can be performed on an edge mask by searching the position parameter space (X and Y) using a Hough transform defined for a rectangular shape with known dimensions. Box filters can be used for efficient implementation.

[0038] A virtual map can be constructed at 410 using the shapes detected in the edge mask at 406 and the virtual map construction rules accessible at 408. To construct the virtual map, the arrangement of the virtual map defined by the construction rules can be compared to the arrangement of the detected shapes. Alignment between the two with the highest likelihood, performed using maximum likelihood estimation (MLE), can correspond to the virtual map selected as associated with the image or selected as an image overlay. This comparison can be performed using various pattern matching techniques, such as MLE pattern matching, fuzzy regular expression matching, etc. In some examples, the detected shapes can correspond to anchor points used to compare the basic arrangement of the virtual map with the basic arrangement of the detected shapes. The arrangement of the detected shapes can be compared to the virtual map at each anchor point to determine the most likely match. The most likely match can then correspond to the location of the virtual map relative to the image accessed at 402.

[0039] Figure 5This is a specific example of method 500 for constructing virtual maps that can be used for various charged particle beam microscopy samples, such as DRAM samples. The virtual map construction of method 500 can be used in various examples disclosed herein, such as exemplary methods 100, 200, 300, 400, and subsequent examples. In some examples, the virtual map can be defined as a rectangular Cartesian grid, such as a uniformly spaced two-dimensional array. In other examples, the virtual map can be multiple such grids. At 502, edge / shape detection and virtual map construction rules stored in memory or dynamically stored can be accessed. For DRAM samples, detected shapes can correspond to a set of rectangle detections. Robust comparisons may be beneficial because samples may be damaged, contain contamination, errors may occur during their fabrication (e.g., shapes are missing where expected), image contrast may be poor, detections may include false positives, or misaligned virtual maps may mislocate target areas. In some examples, at 504, a shape point map connected by orientation and adjacency can be constructed for the detected shapes and the shape arrangements corresponding to the virtual map. For example, rectangles aligned in the vertical or horizontal direction can be connected. At 506, majority voting can be used to select the physical dimensions of the vertical and horizontal spaces between rectangles, for example, if at 502 or otherwise the physical dimensions are not provided. Majority voting includes a set of decision-making techniques, such as, in the current case, clustering similar physical spaces together, and selecting the largest cluster as the correct cluster. The physical space can then be selected as the average of that cluster. At 508, connections between shapes (e.g., rectangles) that are not directly adjacent (e.g., because there is another rectangle between them) can be removed.

[0040] At 510, candidate anchor points can be determined and selected based on the previously or currently determined MLE scores of the detected shapes. For example, the detected rectangles most likely to correspond to the expected rectangular structure, such as those obtained from the Hough transform at 406 in exemplary method 400, can be selected as candidate anchor points. Each anchor point now defines a candidate solution (grid) along with the mentioned graphical structure. At 512, by iterating over all anchor points, the anchor point with the largest number of other rectangles consistent with the candidate solution can be selected as the correct location for the virtual map. For example, rectangles that conform to or contradict the solution can be removed, and the procedure can be repeated until no more candidate solutions are available. Solutions that do not have enough rectangles (e.g., at least a 2x2 grid in a particular example) can be discarded. In general, anchor-based methods can be used to define virtual maps, but these methods can be modified in various ways based on the complexity of the arrangement, shape, and / or pattern being mapped. Each shape detection can define a set of possible virtual maps. For example, for a set of ten circles in a hexagonal grid, this allows for matching between the circles, from which the best match can be selected.

[0041] As discussed above, in many applications, the constructed virtual map can be used to navigate to a target on a sample to be processed with a charged particle beam. In many cases, the target location can be predefined relative to the virtual map, such as by locating the top-left corner of a selected rectangle within a positioning grid. This can be achieved through symbol encoding or pattern matching associated with the virtual map, similar to matching regular expressions in text. For example, depending on the definition of the virtual map, this can include basic sequential matching or graphical matching. In the specific example involving DRAM samples, a Cartesian grid can be used, which allows matching to be found through two separate sequential matching methods (row and column matching).

[0042] Figure 6A specific navigation example 600 is shown, where at 602 the user selects a target location for navigation based on a virtual map arrangement. In many examples, this can be done offline during scheme preparation. Example 600 may correspond to a specific method where the map target is selected from the virtual representation at 308 of example 300, or at the navigation step at 110 of example method 100, and this specific method may be used as part of examples 200, 400, 500, or in conjunction with other steps of examples 100, 300. At 604, the processor may determine the shape closest to the selected target location based, for example, the shortest Euclidean distance to the shape centroid or default grid location associated with the shape. At 606, the offset vector between the selected target locations on the determined shapes may be stored in memory, for example, as part of the navigation section of the scheme file. At 608, during sample processing, a virtual map of the viewed area is typically constructed based on the sample. The nearest shape determined at 604 can be found in the constructed virtual map, and the stored offset can then be used to determine the target location to which the charged particle beam can navigate.

[0043] At 610, the target location can then be processed using a charged particle beam. In many examples, the constructed virtual map can be scaled during shape matching and construction, such as when the target structure has a scale different from the baseline construction rules (e.g., due to variations between chip samples, image scaling, etc.). For invariant adjustments based on physical scale changes, the offset can also be scaled, for example, based on physical scale adjustments inferred in the shape detection step. In some examples, where the virtual map is a finite object with a well-defined geometric center location, this center location can also be used as a reference for the offset vector.

[0044] Figure 7 Example 700 illustrates a mapping where a virtual map 702 is mapped to an acquired image 704. The virtual map 702 can be a schematic representation of an arrangement of shapes such as four rectangles 706a to 706d. The acquired image 704 can include an arrangement of shapes 708a to 708d that have some aberrations, occlusions, contamination, or other differences relative to the virtual map 702. As shown, contamination 710 occludes the rectangular structure 708d. During comparison matching, the virtual map 702 and its shapes 706a to 706d can be compared to the acquired image 704. For example, the graphic characteristics of the virtual map 702 can be compared to different locations in the acquired (real) image 704 (such as corresponding to positions 712a to 712d). Even though contamination 710 occludes rectangle 708d, the best match of the compared graphics can still correspond to the correct position 712d.

[0045] Additional examples using human reference data

[0046] In many cases, a robust set of input reference image data types might be desirable, which could be used to create virtual maps and / or provide a basis for finding similar data in the target image (which might be referred to as the "real" image). As discussed above, providing automated navigation for sample processing can be challenging because current methods rely on image-to-image template matching. These methods require the creation and storage of large numbers of reference images, which can significantly increase protocol time. Based on various examples in this paper, this problem can be addressed using artificial data that describes the sample and is readily available. Examples could include using available CAD data. Furthermore, artificial data can come from many unexpected sources and does not need to be as precise as formal CAD data. For example, it has been found that images of roughly drawn, hand-drawn, or digitally plotted rectangles can be used. That is, even with the inaccuracies of rough or somewhat imprecise reference image data, it is still possible to use such data as a basis for navigation due to the fundamental characteristics of the shapes (including spatial relationships and dimensions). Thus, in a non-traditional example, a user could draw several rectangles on a napkin or whiteboard, take a photograph of the drawing, send it to a microscope, and the microscope could navigate to and find the matching locations on the sample based on the hand-drawn artificial reference data. In a more general example, the user can provide the system with various forms of artificial image reference data. The system uses a microscope (e.g., scanning electron microscope, focused ion beam microscope, optical microscope) to acquire and analyze the images, performs a matching of the structure from the artificial reference data with the analyzed image, and can locate and navigate to the target area based on this matching. In some examples, to obtain more accurate artificial data, the matched positions can be compared to determine aberrations in the acquired images of the sample, whether optical aberrations (such as distortion, magnification, etc.), or errors in the sample's manufacturing process or deviations from design specifications.

[0047] Figure 8An exemplary method 800 for sample navigation using artificial reference image data is illustrated. For example, exemplary method 800 forms part of or can be used in other examples disclosed herein, including but not limited to method examples 100, 200, 300, 400, 500, and 600. In a processing workflow, navigation can be guided by a schematic image of the sample being viewed and processed. Schematic images typically have varying degrees of accuracy and only provide the operator with basic guidance on what structures to look for in microscopic objects such as storage circuits. As discussed above, existing automated methods typically rely on template matching methods that compare the field of view with a reference image. Often, a large number of specific reference images are required, increasing the time and workflow burden of scheme preparation while also hindering the scalability of the scheme. Furthermore, reference images often have minor differences or other mismatched characteristics regarding the sample being viewed, such as contamination, scale, or new, different, or missing structures, which reduce the matching likelihood and weaken navigation capabilities. By performing feature matching on artificial reference image data and the image being viewed, and then performing an object matching procedure between the two, sample navigation can be simplified for a variety of applications.

[0048] An exemplary method 800 includes taking a real image 802 and artificial reference image data 804 as input. Image 802 may correspond to an image of a sample being viewed or processed, for example, using a scanning electron microscope (SEM), focused ion beam (FIB), optical microscope, or other imaging equipment. In some examples, image 802 may also be filtered or artificial in some way. Artificial reference image data 804 may come from a variety of sources, such as hand-drawn sketches, digital drawings, or extracted from GDS / CAD data using algorithms. For the input image 802, image content may be filtered at 806, segmented at 808, and shapes extracted at 810 to produce data describing the arrangement of the found shapes 812. For example, the found shape data may include a list of shapes, spatial relationships between shapes (e.g., grid-based information), etc. In some examples, the data may be a graphical representation of the point-edge relationships between shapes in a suitable coordinate system (e.g., Cartesian coordinates, hexagonal coordinates, polar coordinates, etc.). In many examples, these steps may be performed dynamically while processing the sample, for example, during the search and navigation of the target area to be processed.

[0049] Before or during processing, the artificial reference image data 804 can be analyzed at 814 or transformed (if not already in a suitable form) into shape data describing one or more shapes to be found. The shape data can take a similar format to the shape data found at 814, such as including a list of shapes, spatial relationships between shapes (e.g., grid-based information), etc. In some examples, the data can be a graphical form describing the point-edge relationships between shapes in a suitable coordinate system (e.g., Cartesian, hexagonal, polar, etc.). At 816, for example, by using NM object matching optimization, the two sets of shape data can be compared to determine the matching likelihood. This can produce an estimate of the alignment between the two sets of shape data. NM object matching involves finding a set of matching objects N in a set of objects M, where M is typically greater than N. This alignment can then be used at 818 for navigational purposes, such as guiding a beam of charged particles to a selected target region in the input image 802, which can be identified relative to the shape and features of the artificial reference image data. As with other publicly available examples, affine adjustments such as scaling can be made during the alignment and shape matching process to robustly utilize the human-made reference image data in a scale-invariant manner. Therefore, a less processor-intensive object matching optimization can be performed instead of using processor-intensive image processing. Furthermore, this can be done for various types of human-made input image data to provide users with a variety of robust options to aid in sample navigation.

[0050] Figure 9 Various examples 900 of procedures for obtaining artificial reference image data 902 are shown, which can be used, for example, exemplary method 800 and other examples described herein (such as examples 100, 200, 300, 400, 500, 600, 700, etc.). According to some examples, a hand-drawn image 904, a digitally drawn image 906, or an algorithmically extracted image 908 can be used as the source image 910. At 912, the source image can be segmented, and at 914, shapes can be extracted from the segmented image, for example, using any of the techniques described herein. In other examples, a CAD or GDS file 916 can be used as source data. At 918, the file 916 can be analyzed and shapes extracted. In additional examples, a text-based list 920, such as having an XML 922, JSON 924, or other file format, can be used. The list 920 can be provided in a suitable format so that the artificial reference image data can be compared with the shape data of the found real image.

[0051] Figure 10This is an example of a shape-finding method 1000 that can be used to find shapes in images (such as artificial reference images and real images). At 1002, an input image can be provided, such as a segmented image with detected edges, or a raw or filtered image. At 1004, a sliding window of the target region can be used to perform an analytical scan of the image; this sliding window of the region of interest can be referred to as a convolution kernel in some examples. The standard deviation of the intensity of pixels in the target region of the sliding window can be calculated. The intensity can be classified into a series of intensity peaks using a histogram. At 1006, the spatial regions of the image can be defined as separate regions using the peaks of the histogram. For example, the first peak can include all black pixels, the second peak can include gray pixels, and the third peak can include white pixels, which can define different corresponding intensity regions in the image.

[0052] At position 1008, Connected Component Analysis (CCA) can be used on separated intensity regions to find shapes. For example, two separate, spaced rectangles in an image may have similar intensities, but they can be separated individually by the edges found. Other regions that do not correspond to similar intensities can be converted to black, while the two spaced rectangles can be white (or vice versa). CCA can search for and find connected components in the image from separated regions. Each of these connected components can be analyzed to determine if it meets some additional parameter requirements that satisfy the input conditions (such as shape size, width, height, angle, shape, or other conditions). Another parameter can include a pixel ratio threshold associated with the histogram peak. For example, artifacts (contamination, spelling errors, etc.) may appear in connected components, but these artifacts should not otherwise cause the component to be excluded from the found shape candidates. Those connected components that satisfy the input conditions can correspond to the found shapes.

[0053] Generally speaking, shape matching methods involve comparing combinations of shapes in a reference image with combinations in a real image. Figure 11 This is an exemplary method 1100 for finding N×M matching shapes between a reference image and a real image. At 1102, a reference image with N found shapes and a real image with M shapes are input for processing. At 1104, each reference shape is compared with each shape in the real image to obtain N×M homography matrices. At 1106, the homography matrices are grouped together based on the similarity of the homography matrices, for example, using k-means clustering. At 1108, the largest group or cluster can be selected as the best match. In some examples, the homography matrix selected for mapping and navigation may correspond to the average value of that group. Figure 12Here is another example 1200 of NM matching that can be used to find matching shapes between a reference image and a real image. At 1202, the input is a reference image with N found shapes and a real image with M shapes for processing. At 1204, each reference shape is compared with each shape in the real image to obtain N×M homography matrices. At 1206, for each matrix, the deviation caused by the mapping is found. For a selected matrix, the main shape is typically mapped without deviation, but each other shape typically includes deviations, such as Euclidean distances, between one or more reference image shape location points (such as corner points) and the associated mapped real image shape location points. At 1208, the homography matrix with the minimum overall deviation can be selected.

[0054] Navigation can be based on the virtual map described herein. In some examples, reference images can be filled onto real-world images so that they share a common size. After expanding the reference images to the viewport of the real-world image, image segmentation, shape recognition, and homography matching can be performed. Once the homography matrix is ​​obtained, it can be used to map the coordinates of the original target region (including specific shapes or locations selected by the user) to the coordinates of the real-world image. For example, this can overlay the reference image onto the real-world image as a virtual map. The homography matrix can then be used to convert selected reference image locations into locations within the real-world image.

[0055] General considerations

[0056] Figure 13 A general example of a suitable computing environment 1300 in which the described innovations can be implemented is depicted. The computing environment 1300 is not intended to impose any limitations on its scope of use or functionality, as these innovations can be implemented in a variety of general-purpose or special-purpose computing systems. For example, the computing environment 1300 can be any of a variety of computing devices (e.g., desktop computer, laptop computer, server computer, tablet computer, etc.).

[0057] refer to Figure 13 The computing environment 1300 includes one or more processing units 1310, 1315 and memories 1320, 1325. Figure 13 In this configuration, 1330 is included within the dashed lines. Processing units 1310 and 1315 execute computer-executable instructions. Processing units can be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. In a multiprocessor system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 13A central processing unit 1310 and a graphics processing unit or coprocessor 1315 are shown. Physical memories 1320 and 1325 may be volatile memories (e.g., registers, caches, RAM), non-volatile memories (e.g., ROM, EEPROM, flash memory, etc.), or some combination thereof, accessible by the processing unit. Memories 1320 and 1325 store software 1380 in the form of computer-executable instructions suitable for execution by the processing unit, which implement one or more methods, portions of methods, combinations of methods, or combinations of portions of methods described herein. For example, software 1380 may be configured to combine charged particle beam processing schemes, analyze and compare images and image features, define and / or construct virtual maps, align virtual maps with acquired images, navigate samples based on virtual maps and / or image comparisons, and / or process samples using charged particle beam microscopy.

[0058] The computing system may have additional features. For example, computing environment 1300 includes storage device 1340, one or more input devices 1350, one or more output devices 1360, and one or more communication connectors 1370. Interconnect mechanisms (not shown), such as buses, controllers, or networks, interconnect the components of computing environment 1300. Typically, operating system software (not shown) provides an operating environment for other software executing in computing environment 1300 and coordinates the activities of the components of computing environment 1300.

[0059] The physical storage device 1340 may be removable or non-removable and includes a magnetic disk, magnetic tape or cassette tape, CD-ROM, DVD, or any other medium that can be used to store information in a non-transitory manner and is accessible within the computing environment 1300. The storage device 1340 stores instructions for implementing one or more innovative software 1380 described herein.

[0060] Input device 1350 may be a touch input device (such as a keyboard, mouse, pen, or trackball), a voice input device, a scanning device, or another device that provides input to computing environment 1300. Output device 1360 may be a monitor, printer, speaker, CD burner, or another device that provides output from computing environment 1300.

[0061] Communication connector 1370 enables communication with another computing entity via a communication medium. The communication medium transmits information, such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal having one or more of its characteristics set or altered in a manner that encodes information in the signal. By way of example and not limitation, the communication medium may be electrical, optical, RF, or other carriers.

[0062] Although some of the methods disclosed in this application are described in a specific order for ease of description, it should be understood that, unless explicitly defined below, this descriptive method encompasses the rearrangement of the order of steps. For example, the sequentially described operations may be rearranged or performed simultaneously in some cases. Furthermore, for the sake of brevity, the accompanying drawings may not show various ways in which the disclosed methods can be combined with other methods.

[0063] Any of the disclosed methods can be implemented as computer-executable instructions stored on one or more computer-readable storage media (e.g., one or more optical media, optical discs, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as flash memory or hard disk drives) and executed on a computer (e.g., any commercially available computer, including smartphones or other mobile devices containing computing hardware). The term "computer-readable storage medium" does not include communication connections such as signals and carrier waves. Any computer-executable instructions used to implement any of the disclosed techniques, as well as any data created and used during implementation of the disclosed embodiments, can be stored on one or more computer-readable storage media. The computer-executable instructions can be, for example, part of a dedicated software application or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed using one or more networked computers, for example on a single local computer (e.g., any suitable commercially available computer) or in a networked environment (e.g., via the Internet, wide area network, local area network, client-server network (such as a cloud computing network), or other such networks).

[0064] For clarity, only selected aspects of software-based implementations are described. Other details well-known in the art are omitted. For example, it should be understood that the disclosed techniques are not limited to any particular computer language or program. For instance, aspects of the disclosed techniques can be implemented by software written in C++, Java, Perl, or any other suitable programming language. Similarly, the disclosed techniques are not limited to any particular type of computer or hardware. Certain details of suitable computers and hardware are well-known and do not need to be elaborated in this disclosure.

[0065] It should also be fully understood that any functionality described herein may be performed, at least in part, by one or more hardware logic components rather than software. For example, and not as a limitation, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0066] Furthermore, any of the software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, corporate intranets, software applications, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0067] Figure 14 An exemplary system 1400 is shown, comprising a charged particle beam microscope and / or an optical microscope configured to process samples. System 1400 may be coupled to or include a navigation tool for navigating samples, such as one or more processors configured with instructions to perform any of the methods or portions thereof described herein.

[0068] As used in this application and claims, the singular forms “a,” “an,” and “the” include the plural forms unless the context clearly indicates otherwise. Additionally, the term “comprising” means “including.” Furthermore, the term “coupled” does not exclude the existence of intermediate elements between coupled items.

[0069] The systems, apparatuses, and methods described herein should not be construed as limiting in any way. Rather, this disclosure relates to all novel and non-obvious features and aspects (whether alone or in various combinations and sub-combinations) of the various disclosed embodiments. The disclosed systems, methods, and apparatuses are not limited to any particular aspect or feature or combination thereof, nor are they required to possess any one or more particular advantages or problems solved. Any operational theory should be readily applicable, but the disclosed systems, methods, and apparatuses are not limited to such operational theories.

[0070] Although some of the methods disclosed in this application are described in a specific order for ease of description, it should be understood that this descriptive style encompasses the rearrangement of the order of steps unless explicitly defined below. For example, operations described sequentially may be rearranged or performed simultaneously in some cases. Furthermore, for the sake of brevity, the accompanying drawings may not show the various ways in which the systems, methods, and apparatuses disclosed herein can be used in conjunction with other systems, methods, and apparatuses. Moreover, terms such as “produce” and “provide” are sometimes used in the specification to describe the disclosed methods. These terms are high-level abstractions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation and are readily apparent to those skilled in the art.

[0071] In some examples, values, programs, or devices are referred to as “lowest,” “best,” “smallest,” etc. It should be understood that such descriptions are only used to indicate that there are many available functional options to choose from, and that such a choice is not necessarily better, smaller, or superior in any other way than other choices.

[0072] Given the many possible embodiments to which the disclosed technical principles can be applied, it should be recognized that the illustrated embodiments are merely examples of the disclosed technology and should not be considered as limiting the scope. Therefore, we claim the full benefits within the scope of all such claims.

Claims

1. A computer-implemented method, the method comprising: accessing an image of a specimen; detecting one or more shapes in the image; constructing a virtual map based on the detected one or more shapes in the image and a set of virtual map construction rules; and based on the virtual map, automatically navigating to a target region of the specimen for processing the specimen using a charged particle beam microscope.

2. The method of claim 1, wherein accessing the image of the specimen comprises viewing a region of the specimen using the charged particle beam microscope.

3. The method of claim 1 or claim 2, further comprising based on the virtual map, automatically navigating to a subsequent target region of the specimen for processing the specimen at the subsequent target region using the charged particle beam.

4. The method of any one of claims 1 to 3, wherein constructing the virtual map comprises accessing an embedded virtual map definition.

5. The method of any one of claims 1 to 3, wherein constructing the virtual map comprises accessing user-defined text-based virtual map generation rules.

6. The method of any one of claims 1 to 3, wherein constructing the virtual map comprises interacting with a graphical user interface to draw shapes depicting one or more target structures.

7. The method of any one of claims 1 to 3, wherein constructing the virtual map comprises interacting with a graphical user interface to select and arrange shapes depicting one or more target structures.

8. The method of any one of claims 1 to 3, further comprising accessing a processing recipe for processing the specimen containing the target region, wherein the recipe is assembled by selecting map targets from a predefined list of targets.

9. The method of any one of claims 1 to 3, further comprising accessing a processing recipe for processing the specimen containing the target region, wherein the recipe is assembled by selecting map targets from a virtual map representation.

10. The method of any one of claims 1 to 3, further comprising accessing a processing recipe for processing the specimen containing the target region, wherein the recipe is assembled by selecting map targets based on a reference image.

11. The method of claim 10, wherein selecting map targets based on a reference image comprises selecting a default target position on the reference image.

12. The method of claim 10, wherein selecting map targets based on a reference image comprises selecting a user-selected precise target position on the reference image.

13. The method of any one of claims 1 to 3, further comprising accessing a processing recipe for processing the specimen containing the target region, wherein the recipe is assembled by: selecting target positions with respect to a virtual map arrangement; and storing in the recipe offset vectors between the selected target positions and the determined nearest shapes. determining a shape closest to the selected target position based on a Euclidean distance; ​ ​ wherein automatically navigating to the target region of the sample includes locating the determined nearest shape in the constructed virtual map and using the stored offset to navigate to the target region.

14. The method of any one of claims 1 to 3, wherein constructing the virtual map based on the detected one or more shapes in the image and a set of virtual map construction rules includes: performing edge detection on the image; detecting shapes in the edge detection result; comparing a shape arrangement defined by the virtual map construction rules to an arrangement of the detected shapes; and selecting a registration scheme of the virtual map relative to the accessed image with the highest likelihood.

15. The method of any one of claims 1 to 3, wherein constructing the virtual map based on the detected one or more shapes in the image and a set of virtual map construction rules includes: accessing edge or shape detection associated with the accessed image and the virtual map construction rules; constructing a shape point graph for shape detection connected by orientation and adjacency and the virtual map defined by the virtual map construction rules; determining candidate anchor points based on a maximum likelihood estimate score of detected shapes, where a candidate solution is defined as a pairing of an anchor candidate to a corresponding graph; and for one or more candidate solutions, comparing its candidate graph to a corresponding virtual map graph to determine a closest match.

16. A charged particle beam microscope configured to process the sample according to the method of any one of claims 1 to 3.

17. A computer readable medium configured with stored processor executable instructions for performing the method of any one of claims 1 to 3.

18. A computer implemented method comprising: accessing dynamically obtained shape data associated with an image of a sample; detecting one or more shapes in the image and converting the detected one or more shapes to found shape data; accessing artificial reference shape data; comparing the found shape data to the artificial reference shape data by object match optimization to determine an alignment between the found shape data and the artificial reference shape data; and based on the alignment, navigating to a target region of the sample for processing the sample using a charged particle beam microscope.

19. The method of claim 18, wherein the artificial reference shape data is based on source data comprising any of: hand drawn, digitally rendered or algorithmically extracted images using extracted shapes segmentation; CAD and / or GDS files using extracted shapes analysis; and / or text based lists.

20. A charged particle beam microscope configured to process the sample according to claim 18.

Images