Systems and methods for electron microscope imaging

Abstract

The present invention provides a method for electron microscopy imaging, including a method for imaging a sample using a charged particle microscope. [Solution] The method includes acquiring an initial image of the surface of the sample using the charged particle microscope, receiving one or more initial regions of interest indications for one or more automated functions of the charged particle microscope, and acquiring a series of subsequent images of the subsequent surface of the sample using the charged particle microscope. For each subsequent image, acquiring the subsequent image includes exposing each of the subsequent surfaces of the sample, identifying one or more regions of interest corresponding to one or more initial regions of interest, and imaging each of the subsequent surfaces, wherein imaging each of the subsequent surfaces includes applying one or more automated functions based on at least one of the identified regions of interest corresponding to the one or more regions of interest.

Description

Technical Field

[0001] The present invention relates to identifying a region of interest for imaging a sample using a charged particle microscope. In particular, a method and system for identifying a region of interest through a slice of a sample for use with an automated function are presented.

Background Art

[0002] Charged particle microscopy is a well-known technique for imaging microscopic objects. Charged particle microscopy refers to irradiating and imaging a sample using a beam of charged particles. Usually, the charged particles used are electrons, which provide a significantly higher image resolution compared to visible light microscopes. Examples of charged particle microscopes include scanning electron microscopes (SEM), focused ion beam scanning electron microscopes (FIB-SEM), transmission electron microscopes (TEM), scanning transmission electron microscopes (STEM) and such devices.

[0003] To obtain a correctly resolved image of a given sample, various parameters of the charged particle microscope need to be set (or adjusted). Such parameters include lens adjustment, eucentric height, focus, spherical aberration, focus centering, beam centering, contrast, brightness, etc. In order to be able to automatically adjust such parameters, an automated function (referred to as autofunction) has been developed. Such a function is usually applied to a part of the sample surface (or a part of the image of the sample) to determine one or more appropriate parameter values. These automated functions (AF) include lens adjustment function, spherical aberration correction function, focus function, contrast / brightness function, focus Centering functions and / or astigmatism-corrected centering functions are included, but are not limited to, these. One of the many automatic functions is autofocus, which is particularly important in charged particle microscopy and unsupervised imaging. One method of autofocus is to use a laser, which is used to measure the distance to the sample surface. To set the focus, an offset from the sample surface to the actual focal point is used. Another method of autofocus is image-based autofocus. Image-based autofocus relies on measuring and maximizing the sharpness of the image at various focal positions within a predicted range. When the image is properly focused, the intensity difference between adjacent pixels naturally increases, so the optics can be adjusted until maximum sharpness is detected.

[0004] It will also be understood that various sample types, such as biological samples (frozen cells on a grid, high-pressure frozen samples, resin-embedded samples, etc.), semiconductor samples, material samples, or other samples that may be studied under a charged particle microscope, may be sensitive to radiation damage. Such damage can degrade the sample during imaging. In some cases, a sample may become charged when imaged with an electron microscope, leading to a decrease in image quality over time through data acquisition. Not all parts of a sample are susceptible to radiation damage or charging. However, even if only a portion of the sample is sensitive, image quality will degrade over time through data acquisition. To avoid this, users typically manually adjust the charged particle microscope and run automated functions on the relevant parts of the sample outside the desired imaging area to avoid radiation damage. However, manually adjusting the area on which automated functions run can be time-consuming when acquiring data from highly sensitive samples.

[0005] Furthermore, if the goal is to image one or more regions of interest (e.g., specific cells, selected areas of tissue, organelles, cell walls, etc.), the sample may be sliced ​​and examined. This allows, for example, to image the 3D structure of the region of interest. For samples with clearly defined planes, the position of the automated function will be determined based on the shape of the stage, beam, and sample surface. However, since the shape of the feature of interest may be curved, that feature may not be present in the same region in all slices of the sample. In such cases, conventional imaging methods using the automated function may fail. Again, the user can manually adjust the charged particle microscope for each slice, but manual adjustment can be time-consuming. [Overview of the Initiative]

[0006] The present invention aims to solve the above-mentioned problems when applying automated functions during imaging. In particular, the present invention uses machine learning techniques and / or image processing techniques to select regions of interest suitable for applying automated functions. Thus, human intervention to define the automated function region can be eliminated while maintaining image quality. The present invention is particularly advantageous when imaging multiple slices (or surfaces) of a sample whose internal shape is unknown. Here, automated functions can be performed throughout the entire data acquisition process without unnecessary user intervention. Note that throughout, the terms slice and surface are used synonymously.

[0007] Although the above description pertains to biological samples, the same applies to all types of samples that can be studied with a charged particle microscope. Therefore, it will be understood that the present invention is not limited to data collection of biological samples, but can also be applied to data collection of semiconductor samples, materials science samples, and the like.

[0008] This specification describes methods and systems that can be used to easily track and locate similar regions of interest that require automated functions for imaging using an electron microscope.

[0009] In a first embodiment, a method is provided for imaging a sample using a charged particle microscope. The method includes: acquiring an initial image of the surface of the sample using the charged particle microscope; receiving instructions for one or more initial regions of interest for one or more automated functions of the charged particle microscope; and acquiring a series of subsequent images of subsequent surfaces of the sample using the charged particle microscope. For each subsequent image, acquiring the subsequent image includes exposing each of the subsequent surfaces of the sample, identifying one or more regions of interest corresponding to one or more initial regions of interest; and imaging each of the subsequent surfaces, wherein imaging each of the subsequent surfaces includes applying one or more automated functions based on at least one of the identified regions of interest corresponding to the one or more regions of interest. Receiving instructions for initial regions may include receiving annotations (or equivalent) indicating the initial regions of interest by the user. The sample for imaging is a 2D or 3D sample. For example, the sample may be a 2D or 3D biological sample prepared by cryogenic freezing, a 2D or 3D biological sample embedded in resin, a 2D or 3D materials science sample, or a 2D or 3D semiconductor sample. As used herein, "2D" means a sample having two-dimensional properties, i.e., a sample that is not explicitly two-dimensional but has some two-dimensional features, such as a wafer with a thin cross-section embedded in resin.

[0010] Exposing each of the subsequent surfaces of the sample may be done by using focused ion or electron beam milling, a laser or a diamond blade.

[0011] Appropriate examples of the aforementioned automatic functions include one or more of the following: lens adjustment function, astigmatism correction function, focus function, contrast / brightness function, focus centering function, astigmatism correction centering function, etc.

[0012] In some embodiments, identifying one or more surface regions of interest corresponding to one or more initial regions of interest includes identifying one or more regions of interest corresponding to the one or more initial regions of interest in a previous image (or image of a previous surface) and imaging each of the subsequent surfaces, wherein imaging each of the subsequent surfaces includes applying one or more automated functions based on at least one of the identified regions of interest in the previous image.

[0013] Alternatively, identifying one or more surface regions of interest corresponding to one or more initial regions of interest includes imaging the surface, identifying one or more regions of interest corresponding to one or more initial regions of interest, and imaging (or re-imaging) the surface. Here, imaging (or re-imaging) each of the subsequent surfaces may include applying one or more of the automated functions to at least one of the identified regions of interest.

[0014] In some embodiments, the method includes calculating similarity scores between one or more initial regions of interest in the initial image and a plurality of subregions of the images of each of the subsequent surfaces of the sample, and selecting one or more subregions as candidate regions of interest within each of the subsequent surfaces of the sample based on the respective similarity scores (such as cross-correlation or cosine similarity). The method further includes calculating reference encodings for the one or more initial regions of interest in the initial image, and calculating encodings for a plurality of subregions of the images of each of the subsequent surfaces of the sample, where the similarity scores for each pair of initial regions of interest and subregions of interest are the similarity scores between the respective encodings.

[0015] The reference encoding may be a deep embedding computed by a trained machine learning algorithm (or model), and the encoding of the plurality of sub-regions may be a deep embedding of the plurality of sub-regions computed by the trained machine learning model.

[0016] In some embodiments, the reference encoding and the sub-region encoding are respective image histograms, and the similarity score may be the Kullback-Leibler divergence between the respective histograms. In some embodiments, the reference encoding and the sub-region encoding are generated by a Gabor filter.

[0017] In some embodiments, a region of interest is selected from candidate regions of interest based on one or more selection criteria. The selection criteria may include one or more distance metrics to the first region of interest, exclusion areas (zones) on the surface, and distance metrics to the center of the field of view.

[0018] In some embodiments, the step of identifying one or more regions of interest involves using a trained machine learning model (such as a video object segmentation model) to define one or more initial regions of interest, where the identification includes using the trained machine learning model to track the one or more features of interest through a series of previous images. The trained machine learning model may be configured to segment each of the subsequent images using the initial regions of interest as reference segmentation of the features of interest.

[0019] The present invention also provides an apparatus for one or more appropriately configured computing devices, including elements, modules, or components configured to carry out the above-described method, such as those described below.

[0020] In particular, the present invention provides a system for imaging a sample using a charged particle microscope. The system comprises one or more processors and a memory storing instructions that, when executed by the system, cause the system to perform the following actions: The instructions cause the system to perform (or receive from) an initial image of the sample surface using the charged particle microscope; receive instructions for one or more initial regions of interest for one or more automated functions of the charged particle microscope; and perform (or receive from) a series of subsequent images of the subsequent surface of the sample using the charged particle microscope. For each subsequent image, the acquisition of the subsequent image includes exposing each of the subsequent surfaces of the sample, identifying one or more regions of interest corresponding to one or more initial regions of interest, and imaging each of the subsequent surfaces, wherein imaging each of the subsequent surfaces includes applying one or more automated functions based on at least one of the identified regions of interest corresponding to the one or more regions of interest. Thus, the system may include the charged particle microscope, or it may be provided separately from the charged particle microscope and communicated with.

[0021] The present invention also provides one or more computer programs suitable for execution by one or more processors, the computer programs being configured to carry out methods outlined above and described herein. The present invention also provides one or more computer-readable media comprising (or storing thereon) such one or more computer programs, and / or data signals to be carried over a network. [Brief explanation of the drawing]

[0022] Herein, embodiments of the present invention will be described for illustrative purposes only with reference to the accompanying drawings. [Figure 1] This diagram schematically illustrates an exemplary scenario in which a sample is analyzed using a particle microscope setup. [Figure 2]A diagram schematically showing an example of a computer system that can be used in the present invention. [Figure 3A] A diagram schematically showing the state where the automatic function fails on different surfaces when attempting to identify an area similar to the initial area of interest. [Figure 3B] A diagram schematically showing the state where the automatic function fails on different surfaces when attempting to identify / track the area of interest through different surfaces. [Figure 4A] A diagram schematically showing an example of a system for imaging a sample surface. [Figure 4B] A diagram schematically showing an exemplary embodiment of a system of the region of interest module 4300. [Figure 4C] A diagram schematically showing an exemplary embodiment of a system of the region of interest module 4300. [Figure 5] A flowchart showing a method 500 for imaging a subsequent sample surface. [Figure 6A] A flowchart showing a method 610 for acquiring an image of a subsequent surface. [Figure 6B] A flowchart showing a method 620 for acquiring an image of a subsequent surface. [Figure 7A] A diagram showing the state of identifying a similar region of interest in a subsequent sample when an initial region of interest is given. [Figure 7B] A diagram showing the state of identifying (or tracking) the region of interest within a subsequent sample when an initial region of interest is given.

Modes for Carrying Out the Invention

[0023] Specific embodiments of the present invention are described in the following description and figures. However, it will be understood that the present invention is not limited to the embodiments described, and some embodiments may not include all of the features described below. Nevertheless, it will be apparent that various modifications and changes can be made herein without departing from the broader spirit and scope of the invention as set out in the appended claims.

[0024] Figure 1 is a schematic diagram illustrating an example of a system 100 for analyzing a sample 120 using a charged particle microscope 110 (or charged particle microscope). The charged particle microscope 110 may be a SEM, FIB-SEM, STEM, TEM, etc. The sample 120 may be, for example, a biological sample having features 130 of interest that need to be imaged.

[0025] Sample 120 may be, for example, a biological sample. Sample 120 has a feature 130 of interest. The sample may have multiple features 130 of interest. The feature 130 of interest in sample 120 is typically one that the user wants to image across subsequent slices of the sample. To do so, the user selects a region of interest that contains the feature of interest. An automated function region (i.e., an area to which automated functions can be applied) can be defined in relation to the region of interest. Typically, the automated function region is defined outside the region of interest so that the region of interest is not damaged by radiation when the automated function is performed. In particular, the automated function region may be defined based on a certain offset from the region of interest (e.g., a pixel offset). The region of interest is typically tracked or identified across subsequent slices of sample 120 using machine learning techniques. The region of interest is based on the feature of interest; that is, the region of interest is selected to contain all or at least some of the feature 130 of interest. The charged particle microscope 110 applies one or more automated functions to an automated function region defined in relation to the region of interest, and configures the charged particle microscope 110 based on the output of the functions, i.e., the imaging parameters generated by these automated functions. The charged particle microscope then images the region of interest on the sample slice using the parameters generated by the application of the automated functions. Typically, the entire sample slice is imaged by the charged particle microscope, including the region of interest. However, it should be understood that only a portion containing the region of interest may be imaged.

[0026] The feature of interest 130 may have an unknown shape. The sample 120 may be sliced ​​by an external system of the charged particle microscope 110 or by an internal system of the charged particle microscope 110. The sample may be sliced ​​into many slices 151, 152, 153...156, and each slice (or surface) may be imaged separately (or individually) by the charged particle microscope 110.

[0027] The charged particle microscope apparatus 110 is configured to receive a sample 120. The charged particle microscope apparatus 110 may also be configured to slice the sample using, for example, a diamond knife, an ion beam, or a laser.

[0028] Sample 120 can be prepared using various techniques in the field of charged particle microscopy, which are well known to those skilled in the art.

[0029] The imaging sample 120 may be one or more of the following: a 2D or 3D biological sample prepared by cryogenic freezing, a 2D or 3D biological sample embedded in resin, a 2D or 3D materials science sample, or a 2D or 3D semiconductor sample, or a sample prepared by one or more different techniques already known in the field of charged particle microscopy. Preparation techniques will not be discussed further here.

[0030] In operation, the charged particle microscope 110 is used to image the first surface (or initial surface) of the sample 120. As part of imaging the surface of the sample 120, the charged particle microscope 110 performs (or implements) one or more functions to set various imaging parameters of the charged particle microscope 110. Thus, the obtained image of the surface of the sample 120 is an image acquired using the values ​​of various imaging parameters set by one or more functions (or automatic functions). It will be understood that, in addition to using imaging parameters set by one or more automatic functions, other imaging parameters set by the user (or predefined based on standard values) can also be used in image acquisition. These automatic functions may include, but are not limited to, one or more of the following: lens adjustment function, astigmatism correction function, focus function, contrast / brightness function, focus centering function, astigmatism correction centering function.

[0031] With its automatic functions, the charged particle microscope can automatically adjust parameters such as lens alignment, eucentric height, focus, astigmatism, centering, contrast, and brightness.

[0032] To ensure that the feature of interest 130 on the surface of sample 120 is imaged correctly (or reliably or accurately), one or more functions are applied to an automated function region defined in relation to the region of interest on the sample surface. This region of interest is selected to include at least a portion of the feature of interest 130. The relationship between the automated function region and the region of interest is defined so that the automated function region reliably represents the region of interest. In this way, the feature of interest in a particular image is reliably and appropriately resolved.

[0033] As described above, slicing sample 120 exposes subsequent surfaces of sample 120 (or surfaces of subsequent slices of sample 120). As each subsequent surface of the sample is exposed, it is imaged using the charged particle microscope 110. Such imaging is performed in the same manner as the initial surface imaging described above. In particular, the charged particle microscope 110 performs (or implements) one or more functions to set various imaging parameters of the charged particle microscope 110 on an automated functional area defined in relation to the region of interest of the subsequent surface. Such a region of interest is based on the features of interest and is ideally selected to include all or at least some of the features of interest 130 on the surface.

[0034] In this way, it will be understood that a particular feature of interest within the sample, which may appear in multiple slices of the sample, will be correctly imaged across all slices in which that feature appears.

[0035] The above discussion is presented from the perspective that each surface of a sliced ​​3D sample is a continuous surface, but it will be understood that this also applies to a series of surfaces of different samples that contain features of a common interest, which may be 2D or 3D (e.g., the type of specific organelle imaged in different cell samples).

[0036] The results when the subsequent region of interest on the surface is not based on feature 130 of the object of interest are described below in relation to Figures 3A and 3B.

[0037] Figure 2 is a schematic diagram showing an example of a computer system 1000 that may be used in embodiments of the present invention. System 1000 comprises a computer 1020. Computer 1020 comprises a storage medium 1040, memory 1060, a processor 1080, an interface 1100, a user output interface 1120, a user input interface 1140, and a network interface 1160, all of which are linked to each other via one or more communication buses 1180. The computer system 1000 may be built into the charged particle microscope apparatus 110, or the computer system 1000 may be external. If the computer system is external to the charged particle microscope apparatus 110, they may be connected via a network interface or other suitable data connection means.

[0038] The storage medium 1040 can be one or more non-volatile data storage devices of any form, such as a hard disk drive, magnetic disk, optical disk, or ROM. The storage medium 1040 can store the operating system that the processor 1080 runs on in order to make the computer 1020 function. The storage medium 1040 may also store one or more computer programs (or software or instructions or code).

[0039] Memory 1060 may be any random access memory (storage unit or volatile storage medium) suitable for storing data and / or computer programs (or software or instructions or code).

[0040] The processor 1080 can be any data processing unit suitable for executing one or more computer programs (such as those stored in the storage medium 1040 and / or memory 1060), some of which, when executed by the processor 1080, cause the processor 1080 to perform the method according to embodiments of the present invention, thereby configuring the system 1000 as a system according to embodiments of the present invention. The processor 1080 may comprise a single data processing unit or a plurality of data processing units operating in parallel or in cooperation with each other. When performing data processing operations according to embodiments of the present invention, the processor 1080 can store data in the storage medium 1040 and / or memory 1060, and / or read data from the storage medium and / or memory. The processor 1080 may comprise one or more graphics processing units (GPUs) operating in cooperation with other data processing units of the processor 1080.

[0041] Interface 1100 can be any unit for providing an interface to a device 1220 that is external to or detachable from the computer 1020. Device 1220 can be one or more data storage devices, such as optical disks, magnetic disks, or solid-state storage devices. Device 1220 may have processing capabilities and may be, for example, a smart card. Thus, interface 1100 can access data from device 1220, provide data to the device, or interface with the device in accordance with one or more commands received from processor 1080.

[0042] The user input interface 1140 is configured to receive input from a user or operator of system 1000. The user may provide this input via one or more input devices of system 1000, such as a mouse (or other pointing device) 1260 and / or keyboard 1240 that is connected to or communicating with the user input interface 1140. However, it will be understood that the user may provide input to computer 1020 via one or more additional or alternative input devices (such as a touchscreen). Computer 1020 may store the input received from the input devices via the user input interface 1140 in memory 1060 for later access and processing by processor 1080, or it may pass the input directly to processor 1080 so that processor 1080 can respond to user input in response to the input.

[0043] The user output interface 1120 is configured to provide visual and / or audio output to the user or operator of the system 1000. Therefore, the processor 1080 can be configured to instruct the user output interface 1120 to form an image / video signal representing a desired visual output and to provide this signal to a monitor (or display unit 1200) of the system 1000 connected to the user output interface 1120. Alternatively, the processor 1080 can be configured to instruct the user output interface 1120 to form an audio signal representing a desired audio output and to provide this signal to one or more speakers 1210 of the system 1000 connected to the user output interface 1120.

[0044] Finally, the network interface 1160 provides the computer 1020 with the functionality to download data from and / or upload data to one or more data communication networks.

[0045] The architecture of system 1000 shown in Figure 2 and described above is merely illustrative, and it will be understood that other computer systems 1000 having different architectures (for example, having fewer components than shown in Figure 2, or having additional and / or alternative components than shown in Figure 2) can be used in embodiments of the present invention. For example, computer system 1000 may include one or more of the following: personal computers, server computers, mobile phones, tablets, laptops, other mobile devices or consumer electronics, distributed (or cloud) computing systems, etc.

[0046] Figure 3A schematically illustrates how the automated function fails on different surfaces when attempting to identify a region similar to the initial region of interest. For example, the user annotates an initial region of interest on the initial surface 310 to which one or more automated functions are to be applied. The user chooses to examine a cell wall and annotates the region of interest 302 on the initial surface 310. Thus, the cell wall is the feature of interest 130 in this example. On the subsequent surface 320, the user's initial annotation for the region of interest 302 is still near the feature of interest, but due to the heterogeneous and curved 3D structure of the cell, it is no longer directly on the feature of interest. On the subsequent surface 330, the user's initial annotation is no longer near the feature of interest because the surface of each slice changes due to the heterogeneous and curved 3D structure of the cell.

[0047] In the case of surface 320, the feature of interest 130 is still near the initial region of interest 302. This initial region of interest can be used to define an auto-function region to which the auto-functions are applied when imaging surface 320. The resulting image of surface 320 should still be correctly resolved with respect to the feature of interest, and therefore the resulting image should correctly (or accurately) depict the feature of interest. However, in the case of surface 330, the initial region of interest is not near the feature of interest 304. Therefore, if this initial region of interest is used to define an auto-function region for which the auto-functions are applied when imaging surface 330, the resulting image will not correctly depict the feature of interest. For example, the feature of interest may be out of focus, or there may not be enough contrast to distinguish the feature of interest. This is because the auto-function (or group of auto-functions) that sets the parameters determining such characteristics does not consider the feature of interest 130 because it is not near the region to which the auto-function (or group of auto-functions) is applied.

[0048] Similarly, Figure 3B schematically illustrates how the automated function fails on different surfaces when attempting to identify / track a feature of interest across different slices (regions of interest across different surfaces). For example, subject 350 needs to be imaged on sample 340. The user annotates a region of interest 365 on the image of the first sample surface 360, which is tracked across subsequent surfaces, and allows one or more automated functions to be applied to automated function regions defined in relation to the tracked region of interest. However, because the 3D structure of the feature of interest 350 is unknown and unevenly distributed within the cell, the placement of the automated function based on the initial user annotation fails on subsequent surfaces 370, 380, and 390.

[0049] Figure 4A is a schematic diagram illustrating an example of a system for imaging the surface of sample 120. System 4000 consists of various modules configured to send and receive inputs and outputs to and from each other. System 4000 includes a receiving module 4100, an indicator module 4200, and a region of interest module 4300. The system may optionally include an automated function region module 4400. System 4000 may be part of the charged particle microscope 110 or it may be a separate external system connected to the charged particle microscope 110.

[0050] The receiving module 4100 is configured to receive a sample surface image 4025. The sample surface image 4025 received by the receiving module 4100 may be an image of the sample surface being re-imaged, or it may be an image of the sample surface prior to the sample surface being imaged.

[0051] The receiving module 4100 may be configured to directly receive the sample surface image 4025 from the charged particle microscope 110, for example, when the system 4000 is part of the charged particle microscope 110 and the receiving module 4100 is involved in acquiring the sample surface image. The receiving module 4100 may also be configured to receive the input of the sample surface image 4025 indirectly from the charged particle microscope, for example from the memory of the charged particle microscope 110, via a communication network between the system 4000 and the charged particle microscope 110. It will be understood that the receiving module 4100 can be configured to receive the input of the sample surface image 4025 from an appropriate external system.

[0052] The instruction module 4200 is configured to receive one or more initial regions of interest 4050 on the surface of the sample 120. One or more initial regions of interest may be on the initial surface (or slice) of the sample. However, it will be understood that one or more initial regions of interest do not necessarily have to be on the initial surface, as regions of interest may not yet be visible and may only become visible on subsequent surfaces. The instruction module 4200 may also be configured to receive one or more initial regions of interest 4050 from the user. The user may annotate (or indicate) one or more regions of interest on an image of the initial surface of sample 120. The regions of interest are typically selected by the user to include all or part of the feature of interest 130, as described above. The feature of interest is a feature that the user wants to track (or identify) through subsequent sample surfaces. Alternatively, the instruction module 4200 may be configured to receive one or more initial regions of interest 4050 from an external / another system configured to generate one or more initial regions of interest.

[0053] The region of interest module 4300 is configured to identify one or more regions of interest corresponding to one or more initial regions of interest. The identified one or more regions of interest relate to a predetermined surface of the sample 130. The initial regions of interest may be regions of interest received by the instruction module 4200. Alternatively, the initial regions of interest may be regions of interest previously identified by the region of interest module 4300 for a previous surface of the sample 130.

[0054] It will be understood that one or more identified regions of interest correspond to one or more initial regions of interest, and that the identified regions of interest contain features (or parts thereof) of interest common to the initial regions of interest with respect to each surface of sample 130. Here, the same features of interest may mean the same physical entity. For example, if each slice consists of slices of the same organelle, then one or more identified regions of interest correspond to one or more initial regions of interest, both containing parts of the organelle. Alternatively, in some embodiments, the same features of interest may mean different physical entities of the same type. For example, if each slice consists of slices of different organelles of the same type, then one or more identified regions of interest correspond to one or more initial regions of interest containing parts of each organelle.

[0055] The region of interest module 4300 may be configured to output one or more identified regions of interest for later acquisition of an image of a given surface. Alternatively, the region of interest module 4300 may be configured to define an automated function region using one or more regions of interest and to trigger (or instruct, or otherwise cause) imaging of a sample by applying the automated functions used for imaging. Imaging includes applying one or more automated functions to an automated function region defined in relation to at least one of the identified regions of interest and acquiring an image using the charged particle microscope 110 based on the output of the functions. It will be understood that the system 4000 may be configured to apply functions to one or more identified regions of interest and to set up the charged particle microscope 110 based on parameters generated by these automated functions.

[0056] As described above, automatic functions may be executed within the domain of interest itself. However, more generally, automatic functions are executed within an automatic function domain defined in relation to the domain of interest. Therefore, system 4000 may include an automatic function domain module 4400. The automatic function domain model is configured to define automatic function domains based on the domain of interest output by the domain of interest module 4300. The automatic function domain module 4400 may store one or more predefined relative automatic function domain definitions that specify the relative location and / or range of the automatic function domain with respect to the domain of interest. An example of such a relative definition is a vector offset from the domain of interest.

[0057] Further description of the region of interest module 4300 and its alternative embodiments is provided below.

[0058] Figure 4B is a schematic diagram illustrating an exemplary embodiment of the system of the region of interest module 4300. In the exemplary embodiment of Figure 4B, the region of interest module 4300 comprises a machine learning (ML) model 4310 and a processing module 4320. The processing module 4320 comprises a cosine similarity module 4322 and a filtering module 4324.

[0059] The ML model 4310 includes an encoder 4315. The encoder 4315 of the ML model 4310 is configured to compute a deep embedding of the input received by the ML model 4310. The ML model 4310 receives a sample surface image 4025 and one or more initial regions of interest 4050 as inputs. A deep embedding is a mathematical representation of data, in this case the data is the sample surface image 4025 and one or more initial regions of interest 4050 as inputs. Thus, it will be understood that a deep embedding is a specific type of data encoding (or encoding) of the input data.

[0060] ML model 4310 may also be a multilayer neural network that encodes an arbitrary image into a deep embedding space of vectors.

[0061] The ML Model 4310 is a convolutional neural network or vision transformer neural network. The ML model 4310 may be trained on publicly available datasets. The ML model may be trained on common, non-domain-specific images, such as photographs from a consumer camera. The encoder 4315 of the ML model 4310 typically computes its own deep embedding for each sub-region of the received input image. Typically, a sub-region is a square array of pixels (a common size is 16x16 pixels). However, this is merely an example of a sub-region, and it should be understood that the ML model is not limited to 16x16 pixel blocks.

[0062] The encoder 4315 of the ML model 4310 receives one or more initial regions of interest 4050 and computes and stores the reference deep embeddings for one or more of the initial regions of interest 4050. As previously mentioned, one or more initial regions of interest 4050 may contain all or part of the features of interest 130 on the initial surface of sample 120. The reference deep embeddings for one or more initial regions of interest 4050 may be stored within the ML model 4310 or in external memory connected to the ML model 4310 via an appropriate communication network (not shown in Figure 4B).

[0063] Next, the encoder 4315 of the ML model 4310 receives a sample surface image 4025 that identifies similar regions of interest. The encoder 4315 of the ML model 4310 calculates and stores the deep embedding of the sample surface image 4025. The deep embedding of the sample surface image 4025 is stored either within the ML model 4310 or in external memory connected to the ML model 4310 via an appropriate communication network (not shown in Figure 4B).

[0064] The processing module 4320 comprises a similarity module 4322 and a filtering module 4324. The calculated reference deep embeddings and deep embeddings are input directly from the ML model 4310 to the similarity module 4322. Alternatively, the reference deep embeddings and deep embeddings are input to the similarity module 4322 from an external memory connected to the ML model 4310 via an appropriate communication network (not shown in Figure 4B).

[0065] The similarity module 4322 is configured to determine a set of candidate regions of interest from the sample surface image 4025. Typically, such candidate regions of interest are determined based on deep embedding of similarity (or similarity score) of proposed regions of interest and embedding of one or more initial regions of interest that satisfy predefined criteria. Predefined criteria may include any of the following: a predefined threshold (e.g., any candidate region with similarity above the threshold), or ranking criteria (e.g., selecting the candidate region with the highest similarity score).

[0066] Typically, the similarity module uses cosine similarity. Thus, the similarity module 4322 may be configured to calculate the cosine similarity between the reference deep embeddings of one or more initial regions of interest 4050 and the deep embeddings of candidate regions of interest. It will be understood that other similarity scores may be used.

[0067] Next, the candidate regions of interest are typically input to the filtering module 4324, where one or more filtering criteria are applied to reduce the number of candidate regions of interest. The filtering module 4324 outputs the remaining regions of interest to determine the regions where an automatic function can be applied. The automatic function to be applied may be one or more automatic functions listed in Figure 1.

[0068] The filtering module 4324 filters the generated candidate regions and outputs regions of interest. The filtering module 4324 can filter candidate regions of interest and generate regions of interest by applying one or more of the following criteria: distance metric to the first region of interest, exclusion areas (zones) on the surface, distance metric to the center of the field of view, etc.

[0069] It will be understood that the filtering module can be omitted entirely. For example, the similarity module may be configured to output a single region of interest, such as the region of interest with the highest similarity score.

[0070] The ML model 4310 described above may consist of one or more algorithms from a number of well-known ML algorithms. A specific example is the Vision Transformer ML model. Details of the Vision Transformer model are described in “An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale”, Alexey Dosovitskiy et al, arXiv:2010.11929, https: / / doi.org / 10.48550 / arXiv.2010.11929. In this example, the trained encoder of the Vision Transformer model is used as the encoder. The trained encoder computes deep embeddings of the input image, generating one embedding for every 16x16 pixel region in the image. Reference centers for one or more initial regions of interest 4050 are computed and stored as reference embeddings. Deep embeddings of the sample surface 4025 are computed. The deep embeddings are compared to the reference embeddings using a cosine similarity metric. The most similar deep embedding is selected, mapped to coordinates within the sample surface image 4025, and the similarity is calculated.

[0071] In this way, it will be understood that the region of interest module 4300 in Figure 4B identifies subsequent regions of interest within the surface that are similar to the initial region of interest. In particular, the identified regions of interest may not track specific physical entities (such as specific organelles) throughout the surface sequence. However, the identified regions of interest will contain the same or similar types of features of interest as the initial region of interest.

[0072] Although the above description uses a machine learning model, it will be understood that the approach may be implemented using conventional image processing techniques instead. In particular, the machine learning model 4310 and the similarity module 4322 may be replaced with a single image processing similarity module. The image processing similarity module may receive a sample surface image 4025 and one or more initial regions of interest 4050 as input. The image processing similarity module then determines candidate regions of interest using known image comparison methods.

[0073] Such image comparison methods include, The cross-correlation defines the similarity score between the initial region of interest 4050 and the candidate region of interest, and This may include either a histogram comparison that calculates image histograms for the initial region of interest 4050 and the candidate region of interest. The Kullback-Leibler divergence between the respective histograms may define a similarity score between the initial region of interest 4050 and the candidate region of interest.

[0074] Alternatively, Gabor filter similarity may be used, in which case encodings are calculated for the initial region of interest 4050 and the candidate region of interest. These encodings can, in practice, serve as a substitute for the deep embeddings described above. Thus, the similarity score between the initial region of interest 4050 and the candidate region of interest may be defined as the cosine similarity between the respective Gabor filter encodings.

[0075] Figure 4C schematically shows an exemplary embodiment of the system of the region of interest module 4300. In the exemplary embodiment of Figure 4C, the region of interest module 4300 comprises an ML segmentation model 4330.

[0076] The ML model 4330 is configured to receive input of one or more initial regions of interest 4050 based on one or more features of interest 4055. In this case, the initial regions of interest 4050 typically identify the corresponding features of interest 4055. Thus, the regions of interest 4050 in this embodiment may typically be thought of as segmentation masks generated by the user. In the example image shown in Figure 4C, the regions of interest 4050 and the features of interest 4055 are shown in yellow. The ML model 4330 is also configured to receive a sample surface image 4025 that tracks the regions of interest 4050.

[0077] The ML model is configured to track an initial region of interest 4050 through a sample surface image 4025. Such tracking can be understood as analogous to tracking an object through subsequent frames of a video. In particular, the initial region of interest may be thought of as segmenting (or identifying) the object to be tracked (in this case, the feature of interest). The initial region of interest may also be thought of as a reference segmentation, and the image corresponding to the initial region of interest may be thought of as a reference frame. The ML model 4330 is configured to generate a region of interest on the sample surface image 4025 that corresponds to (or defines, or contains) the feature of interest 4055 of the initial region of interest. Typically, the output region of interest takes the form of a segmentation map (or mask) of the feature of interest.

[0078] Similar to the example described above, the output region of interest may be used to execute an automated function for imaging. In particular, the output region of interest may be used to define the automated function region on which the aforementioned automated function is performed.

[0079] It will be understood that any suitable video or image segmentation model (e.g., any video object segmentation model) may be used as ML model 4330. An example of such a model is discussed in Gao, M., Zheng, F., Yu, JJQ et al. Deep learning for video object segmentation: a review. Artif Intell Rev 56, 457-531 (2023). https: / / doi.org / 10.1007 / s10462-022-10176-7. The inventors found that particularly advantageous ML models were the Adaptive Feature Bank and Uncertain-Region Refinement (AFB-URR) models. Details of this model can be found in Yongqing Liang, Xin Li, Navid Jafari, and Qin Chen. 2020. Video object segmentation with adaptive feature bank and uncertain-region refinement. In Proceedings of the 34 th This is described in the International Conference on Neural Information Processing Systems (NIPS'20), Curran Associates Inc., Red Hook, NY, USA, Article 289, 3430-3441. When using such a model, the initial region of interest replaces the reference segmentation, and the image corresponding to the initial region of interest becomes the reference frame. Sample surface image 4025 replaces the target frame.

[0080] Figure 5 is a flowchart illustrating a method 500 for imaging a subsequent sample surface (surface N), which can be performed by system 100 in Figure 1 and system 4000 in Figure 4A, according to several embodiments of the present invention.

[0081] In step 502, an initial image of the surface is acquired using the charged particle microscope 110 according to system 100.

[0082] In step 504, one or more initial regions of interest are indicated based on the characteristics of one or more objects of interest (by the indication module of system 4000 in Figure 4A).

[0083] In step 506, the sample surface to be imaged is exposed to the charged particle microscope (the sample surface may be received by the receiving module of system 4000 in Figure 4A).

[0084] In step 508, one or more regions of interest to be imaged are identified by the system described in Figures 4B and 4C.

[0085] In step 510, one or more automated functions described with respect to Figure 1 are applied based on one or more regions of interest identified on the sample surface.

[0086] In step 512, the sample surface is imaged by applying the automatic function.

[0087] Various embodiments of Figure 5 will be described in more detail below.

[0088] Figure 6A is a flowchart of a method 610 for acquiring an image of a subsequent surface N, which can be performed by system 100 in Figure 1 and system 4000 in Figure 4A, according to several embodiments of the present invention. In Figure 6A, it can be seen that the region of interest is actually identified within an image of a previously captured surface (surface N-1). This identified region of interest is then used to apply an automated function on surface N. This is because the drift of the region of interest from the previously captured surface (surface N-1) to the surface being captured (surface N) is minimized. This method is useful when the sample surface is highly sensitive. This will be discussed in more detail below. As mentioned above, the automated function may be applied to the region of interest itself, or more generally, to an automated function region defined in relation to the region of interest.

[0089] Figure 6B is a flowchart of a method 620 for acquiring a subsequent image of a surface N, which can be performed by system 100 in Figure 1 and system 4000 in Figure 4A, according to several embodiments of the present invention. In Figure 6B, it will be understood that the surface N is imaged, and then a region of interest is identified on the imaged surface N. After the automated function is performed, the region of interest on the surface N is re-imaged to obtain a better image. This method is useful when the sample is less likely to degrade due to imaging and re-imagement. This will be discussed in more detail below.

[0090] In some cases, users may want to identify similar regions of interest in subsequent samples based on one or more initial regions of interest. Exemplary embodiments that can be performed by the systems in Figures 4A and 4B are described below, following the steps shown in the flowcharts of Figures 5, 6A, and 6B.

[0091] In one embodiment, to image surface N, the receiving module 4100 receives an image 4025 of surface N-1, and the indicating module 4200 receives an indication of one or more initial regions of interest as input 4050. As previously stated, one or more initial regions of interest 4050 may include all or part of the features 130 of interest on the initial surface of sample 120. The image 4025 of surface N-1 is an image of the surface of surface N that has already been imaged by one embodiment of the present invention. The image 4025 of surface N-1 is acquired by the receiving module 4100 as described in relation to Figure 4A. One or more initial regions of interest 4050 are input to the region of interest module 4300, and the encoder 4315 of the ML model 4310 calculates the deep embedding of one or more initial regions of interest 4050 and outputs the reference deep embedding of one or more initial regions of interest 4050. The ML model 4310 may be the ML model shown in Figure 4B. The encoder 4315 of the ML model 4310 may compute one or more reference deep embeddings for each 16x16 pixel region in the input of one or more initial regions of interest 4050. Next, the image 4025 of surface N-1 is used in the ML model within the region of interest module 4300. The data is input to encoder 4315 of ML model 4310. Encoder 4315 of ML model 4310 calculates deep embeddings for surface N-1 image 4025. Encoder 4315 of ML model 4310 may calculate one or more deep embeddings for every 16x16 pixel region in surface N-1 image 4025. The calculated base deep embeddings and deep embeddings are input to cosine similarity module 4322 of processing module 4320, where the cosine similarity between the base deep embeddings and deep embeddings is calculated and candidate regions of interest are identified. The candidate regions of interest are input to filtering module 4324. Filtering Module 4324 outputs a region of interest used to image surface N using an automated function. Filtering is performed in the manner described in relation to Figure 4B. The aforementioned imaging includes applying one or more automated functions based on at least one of the identified regions of interest and acquiring an image of surface N using the charged particle microscope 110 based on the output of the automated functions. It will be understood that the system 4000 may be configured to apply the automated functions and set up the charged particle microscope 110 based on the parameters generated by these automated functions and the image of surface N. Similarly, after surface N has been imaged, surface N+1 can be imaged using the image of surface N. This embodiment is particularly useful when the sample surface to be imaged is sensitive and may be damaged by the charged particle microscope.

[0092] In one embodiment, to image surface N, the instruction module 4200 receives one or more initial regions of interest as input 4050. As previously stated, one or more initial regions of interest 4050 may include all or part of the features 130 of interest on the initial surface of sample 120. One or more initial regions of interest 4050 are input to the region of interest module 4300, and the encoder 4315 of the ML model 4310 calculates deep embeddings for one or more initial regions of interest 4050 and outputs reference deep embeddings for one or more regions of interest 4050. The encoder 4315 of the ML model 4310 can calculate one or more reference deep embeddings for each 16 × 16 pixel region in the input of one or more initial regions of interest 4050. The ML model may be the ML model 4310 shown in Figure 4B. Surface N is imaged by the charged particle microscope 110 and input 4025 to the receiving module 4100. Next, the image 4025 of surface N is input to the encoder 4315 of the ML model 4310 in the region of interest module 4300. The encoder 4315 of the ML model 4310 calculates the deep embeddings of the image 4025 of surface N. The encoder 4315 of the ML model 4310 can calculate one or more deep embeddings for every 16 × 16 pixel region in the image of surface N 4025. The reference deep embedding and the deep embeddings are input to the cosine similarity module 4322 of the processing module 4320, where the cosine similarity between the reference deep embedding and the deep embeddings is calculated. The calculated cosine similarity is input to the filtering module 4324. The filtering module 4324 outputs the region of interest based on the automated functions available for imaging surface N. Filtering is performed in the manner described in relation to Figure 4B. The aforementioned imaging includes applying one or more automated functions and acquiring an image of surface N using the charged particle microscope 110 based on the output of the automated functions. It will be understood that the system 4000 may be configured to apply functions to one or more identified regions of interest and to set up the charged particle microscope 110 based on parameters generated by these automated functions.This embodiment is particularly useful when the sample surface to be imaged is not very sensitive to a charged particle microscope and / or is not easily damaged.

[0093] Figure 7A shows an example of the above embodiment, in which a similar region of interest is identified on a subsequent sample surface. An indication of region of interest 7110 on sample surface 7100 is given. Following the steps described in any of the above embodiments, which can be performed by the system in Figures 4A and 4B, in accordance with the procedure shown in the flowcharts of Figures 5, 6A and 6B, similar regions of interest 7130, 7140 and 7150 are found on a subsequent sample surface 7120.

[0094] In some cases, users may want to track one or more regions of interest across the entire sample surface, based on one or more initial regions of interest. Exemplary embodiments that can be performed by the system in Figures 4A and 4C, following the steps shown in the flowcharts of Figures 5, 6A, and 6B, are described below.

[0095] In one embodiment, to image surface N, the receiving module 4100 receives an image 4025 of surface N-1, and the indicating module 4200 receives an indication of an initial region of interest as input 4050. The initial region of interest specifies (or identifies) a feature of interest. The image 4025 of surface N-1 is an image of a previous surface of surface N already imaged by one embodiment of the present invention. The image 4025 of surface N-1 is acquired by the receiving module 4100 as described in relation to Figure 4A. The initial region of interest 4050 is input to the region of interest module 4300, typically as a reference segmentation.

[0096] The ML model is configured to track an initial region of interest 4050 through sample surface images N-1, N, N+1, and so on. Such tracking will be understood as analogous to tracking an object through subsequent frames of a video. The ML model 4330 is configured to generate a region of interest on sample surface image N-1 that corresponds to (or defines, or contains) the features of interest 4055 of the initial region of interest. Typically, the output region of interest takes the form of a segmentation map (or mask) of the features of interest.

[0097] A region of interest (or segmentation mask) is output to the charged particle microscope 110, which estimates the location of one or more regions of interest on surface N and uses automated functions to image those regions based on the estimated one or more regions of interest. The imaging includes applying one or more automated functions based on at least one of the identified regions of interest and acquiring images using the charged particle microscope 110 based on the output of these functions. It will be understood that the system 4000 may be configured to apply functions to one or more identified regions of interest and to set up the charged particle microscope 110 based on parameters generated by these automated functions. Similarly, after surface N has been imaged, a segmentation mask can be generated for imaging surface N+1 using the image of surface N. This embodiment is particularly useful when the sample surface to be imaged is sensitive and may be damaged by the charged particle microscope.

[0098] In another embodiment, to image the surface N, the surface N is first imaged by a charged particle microscope 110 and input 4025 to a receiving module 4100. Next, the image 4025 of the surface N is input to an ML model, and an initial region of interest 4050 is tracked through the image of the surface N. The ML model 4330 is configured to generate a region of interest on the sample surface image N that corresponds to (or defines or contains) the features of interest 4055 of the initial region of interest. Typically, the output region of interest takes the form of a segmentation map (or mask) of the features of interest.

[0099] The region of interest (or segmentation mask) is output to the charged particle microscope 110, which uses the charged particle microscope 110 to re-image the surface N. This re-imagement includes applying one or more automated functions based on at least one of the identified regions of interest, and acquiring images using the charged particle microscope 110 based on the output of these automated functions. It will be understood that the system 4000 can be configured to apply automated functions to one or more identified regions of interest and to configure the charged particle microscope 110 based on parameters generated by these automated functions. This method can be used when the sample or sample surface is not highly sensitive and will not be damaged by initial imaging.

[0100] Figure 7B shows an example of the above embodiment, in which an initial region of interest is identified on a subsequent sample surface. An indication of the region of interest 7210 on the sample surface 7200 is given. Following the steps described in any of the above embodiments, which can be performed by the system in Figures 4A and 4C, the region of interest 7210 is identified on a subsequent sample surface 7220, following the steps described in the flowcharts shown in Figures 5, 6A and 6B.

[0101] It will be understood that the described method is presented as individual steps to be performed in a specific order. However, those skilled in the art will understand that these steps can be performed in combination or in a different order while achieving the desired results.

[0102] It will be understood that embodiments of the present invention can be carried out using a variety of different information processing systems. In particular, the figures and their descriptions illustrate exemplary computing systems and methods, but these are presented solely to provide a useful reference when describing various aspects of the present invention. Embodiments of the present invention can be run on any suitable data processing device, such as a personal computer, laptop, personal digital assistant, mobile phone, television, or server computer. Naturally, the descriptions of systems and methods have been simplified for the purposes of discussion, and these are merely examples of the many different types of systems and methods that may be used in embodiments of the present invention. The boundaries between logical blocks are merely illustrative, and it will be understood that in alternative embodiments, logical blocks or elements may be merged, or different functional divisions may be applied to various logical blocks or elements.

[0103] It will be understood that the above functions may be implemented as hardware and / or software, as one or more corresponding modules. For example, the above functions may be implemented as one or more software components executed by the system's processor. Alternatively, the above functions may be implemented as hardware, such as one or more field-programmable gate arrays (FPGAs), and / or one or more application-specific integrated circuits (ASICs), and / or one or more digital signal processors (DSPs), and / or other hardware configurations. Each method step implemented in the flowcharts included herein, or in the flowcharts described above, may be implemented by a corresponding module. Furthermore, multiple method steps implemented in the flowcharts included herein, or in the flowcharts described above, may be implemented together by a single module.

[0104] To the extent that embodiments of the present invention are implemented by computer programs, it will be understood that a storage medium for recording such computer programs and a transmission medium for carrying such computer programs constitute embodiments of the present invention. A computer program may have one or more program instructions or program code, and when these are executed by a computer, embodiments of the present invention are executed. As used herein, the term “program” means a set of instructions designed to run on a computer system and can include subroutines, functions, procedures, modules, object methods, object implementations, executable applications, applets, servlets, source code, object code, shared libraries, dynamic link libraries, and / or other sets of instructions designed to run on a computer system. A storage medium may be a magnetic disk (such as a hard drive or floppy disk), an optical disk (such as a CD-ROM, DVD-ROM, or Blu-ray disk), or memory (such as ROM, RAM, EEPROM, EPROM, flash memory, or portable / removable memory device). A transmission medium may be a communication signal, data broadcast, or a communication link between two or more computers.

Claims

1. A method for imaging a sample using a charged particle microscope, Using the charged particle microscope, an initial image of the sample surface is obtained, Receiving indications for one or more initial regions of interest for one or more automated functions of the charged particle microscope, This includes obtaining a series of subsequent images of the subsequent surface of the sample using the charged particle microscope, For each subsequent image, acquiring the subsequent image means exposing each of the subsequent surfaces of the sample, Identifying one or more regions of interest corresponding to the one or more initial regions of interest mentioned above, A method comprising imaging each of the aforementioned subsequent surfaces, wherein imaging each of the aforementioned subsequent surfaces includes applying one or more automated functions based on at least one of the identified regions of interest corresponding to one or more regions of interest.

2. A method according to claim 1, wherein identifying one or more surface regions of interest corresponding to one or more initial regions of interest is Identifying one or more regions of interest corresponding to the one or more initial regions of interest in the previous image, A method for imaging each of the aforementioned subsequent surfaces, wherein imaging each of the aforementioned subsequent surfaces includes applying one or more automated functions based on at least one identified region of interest of the previous image.

3. A method according to claim 1, wherein identifying one or more surface regions of interest corresponding to one or more initial regions of interest is To image the aforementioned surface, Identifying one or more regions of interest corresponding to the one or more initial regions of interest mentioned above, A method comprising imaging the aforementioned surface, wherein imaging each of the subsequent surfaces comprises applying one or more of the automated functions to at least one of the identified regions of interest.

4. The method according to claim 1, The process involves calculating similarity scores between one or more initial regions of interest in the initial image and multiple subregions of the images of each subsequent surface of the sample. A method comprising selecting one or more subregions as candidate regions of interest within each of the subsequent surfaces of the sample based on the respective similarity scores.

5. The method according to claim 4, The process involves calculating the reference encoding of one or more initial regions of interest in the initial image, This includes calculating the encoding of multiple sub-regions of the image of each subsequent surface of the sample, A method wherein the similarity score of each pair of the initial region of interest and sub-regions of interest is the similarity score between the respective encodings.

6. A method according to claim 5, wherein the reference encoding is a deep embedding computed by a trained machine learning algorithm, and the encodings of the plurality of sub-regions are deep embeddings of the plurality of sub-regions computed by the trained machine learning model.

7. A method according to claim 6, wherein the trained machine learning model is a vision transformer neural network.

8. A method according to claim 4, wherein the similarity scores are interrelated.

9. A method according to claim 5, wherein the reference encoding and the sub-region encoding are respective image histograms, and the similarity score is the Kullback-Leibler divergence between the respective histograms.

10. A method according to claim 5, wherein the reference encoding and the sub-region encoding are generated by a Gabor filter.

11. A method according to claim 4, wherein the similarity score is cosine similarity.

12. The method according to claim 4, A method further comprising selecting a region of interest from candidate regions of interest based on one or more selection criteria.

13. A method according to claim 12, A method wherein the selection criteria include one or more of the following: a distance metric to the initial region of interest, an exclusion area on the surface, and a distance metric to the center of the field of view.

14. The method according to claim 1, The step of identifying one or more regions of interest uses a trained machine learning model, and the one or more initial regions of interest define the features of one or more objects of interest. The identification method includes using the trained machine learning model to track the features of one or more objects of interest through a series of previous images.

15. A method according to claim 14, wherein the trained machine learning model is a video object segmentation model.

16. A method according to claim 14, wherein the trained machine learning model is configured to segment each of the subsequent images using the initial region of interest as a reference segmentation of the features of interest.

17. The method according to claim 1, wherein receiving an instruction for an initial region means A method comprising receiving an annotation indicating the initial region of interest by the user.

18. A method according to claim 1, wherein the sample for imaging is a 3D biological sample prepared by cryogenic freezing, a 3D biological sample embedded in resin, a materials science sample, or a semiconductor sample.

19. A method according to claim 1, wherein exposing each of the subsequent surfaces of the sample comprises exposing each of the subsequent surfaces using focused ion or electron beam milling, a laser or a diamond blade.

20. A method according to claim 1, wherein the automatic function includes one or more of the following: a lens alignment function, an astigmatism correction function, a focus function, a contrast / brightness function, a focus centering function, an astigmatism correction centering function, or a combination of the basic automatic functions described above.

21. An apparatus configured to perform the method described in any one of claims 1 to 20.

22. One or more computer-readable media that, when executed by one or more processors, stores instructions causing the processors to perform the method according to any one of claims 1 to 20.

Images