Acquisition strategies for neural network based image restoration.

Abstract

To provide a method of and a system for restoring an image on a neural network basis.SOLUTION: A method according to the present invention has the steps of acquiring a plurality of training image pairs of a material by using the same acquired parameters by each image of the plurality of training image pairs where at least each training image is an image of the same part of the material, updating an artificial neural network based on the plurality of training image pairs, and removing a noise of a plurality of material images by using the updated artificial neural network. The plurality of material images is acquired by using the same acquired parameters as those used for acquiring the plurality of training image pairs.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates generally to artificial intelligence (AI)-enabled image restoration, and more particularly to AI-enabled noise reduction and sparse reconstruction. [Background technology]

[0002] In many types of microscopes, noise is difficult to remove or reduce, resulting in less than desired results. Moreover, the type of image desired by the user may make noise even more difficult to manage. For example, in charged particle microscopy, the parameters of the charged particle beam and other image acquisition parameters may affect the noise of the image, but may also have an adverse effect on the desired field of view and sample interaction. For further examples, there is a trade-off between the dwell time of the charged particle beam and the noise. In particular, a short dwell time may be preferred to ensure that there is no or minimal change during image acquisition, such as drift. A short dwell time may also result in rapid image acquisition. However, a disadvantage is that images from short dwell times tend to be noisy and / or sparse, e.g., have pixels lacking information, which is typically undesirable for end users. A long dwell time may reduce noise, e.g., increase the signal-to-noise and provide a less sparsity image, but a longer dwell time tends to increase the image acquisition time, which may further damage the sample, especially in biological samples. Long dwell times also typically result in drift during image acquisition. These tradeoffs and issues are just a few of the ones that can affect noisy images.

[0003] End users typically desire large fields of view and / or three-dimensional volumes, which require long acquisition times (on the scale of a month in some cases), which only increase if long dwell times are used. Therefore, fast acquisition times with reduced noise are desirable. Acquisition times can be further reduced by sparse image acquisition. Various attempts at image restoration have been made over the years to solve this problem, but the solutions offered have their own shortcomings, such as overly smoothed or virtually unimproved images. Thus, a need remains to solve the problem of noisy images. Summary of the Invention

[0004] Apparatus and methods for neural network based image restoration are disclosed herein. Image restoration includes denoising an image, sparse reconstruction of an image, and combinations thereof. An exemplary method for neural network based denoising includes at least acquiring a plurality of training image pairs of a sample, where each training image of each of the plurality of training image pairs is an image of a same location of the sample, and where each image of the plurality of training image pairs is acquired using the same acquisition parameters; updating an artificial neural network based on the plurality of training image pairs; and denoising the plurality of sample images using the updated artificial neural network, where the plurality of sample images are acquired using the same acquisition parameters used to acquire the plurality of training image pairs.

[0005] An exemplary apparatus for implementing neural network based denoising includes at least a charged particle microscope imaging platform and a controller at least coupled to control the charged particle microscope imaging platform, the controller coupled to a non-transitory computer readable medium including code that, when executed by the controller, causes the system to: acquire a plurality of training image pairs of a sample, where each training image of each of the plurality of training image pairs is an image of a same location of the sample, and where each image of the plurality of training image pairs is acquired using same acquisition parameters; update an artificial neural network based on the plurality of training image pairs, where the artificial neural network is included with or coupled to the system; and denoise the plurality of sample images using the updated artificial neural network, where the plurality of sample images are acquired using the same acquisition parameters used to acquire the plurality of training image pairs.

[0006] Another embodiment of neural-based image restoration is directed towards sparse reconstruction. An exemplary method for sparse reconstruction includes at least acquiring a plurality of training image pairs of a sample, where each training image of each of the plurality of training image pairs is an image of a same location of the sample but includes different image pixels, and where each image of the plurality of training image pairs is acquired using the same acquisition parameters, updating an artificial neural network based on the plurality of training image pairs, and reconstructing each of the plurality of sample images using the updated artificial neural network, where the plurality of sample images are acquired using the same acquisition parameters as used to acquire the plurality of training image pairs, and where each image of the plurality of sample images is a sparse image. [Brief description of the drawings]

[0007] [Figure 1]1 is an example of a charged particle microscope system according to one embodiment of the present disclosure. [Diagram 2] 1 is an exemplary method for removing noise from an image, according to one embodiment of the present disclosure. [Diagram 3] 1 is an exemplary method for reconstructing a sparse image, according to one embodiment of the present disclosure. [Figure 4] 1 is an example image sequence illustrating denoising an image according to one embodiment of the present disclosure. [Diagram 5] 1 is an example image sequence illustrating sparse reconstruction, according to one embodiment of the present disclosure. [Figure 6] 6 is an exemplary functional block diagram of a computing system 600 according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Like reference numerals refer to corresponding parts throughout the several views of the drawings.

[0009] An embodiment of the present invention relates to neural network-based image restoration. In one example, an artificial neural network may be trained on a small number of noisy images to be a denoiser, and then the denoiser is used to denoise a larger dataset of sample images. In such an example, the acquisition parameters used to acquire the small number of noisy images are also used to acquire the sample images, thereby ensuring that the noise in both sets of images is similar. In another example, instead of denoising, an artificial neural network may be similarly trained to provide a sparse reconstruction. In an example of sparse reconstruction, the training images include sparse images that are at least slightly overlapping, so that the network learns how to fill in the sparse images of a large dataset of sparse sample images. However, it should be understood that the methods described herein are generally applicable to a wide range of different AI-enhanced or enabled image restoration techniques and should not be considered limiting.

[0010] As used in this application and claims, the singular forms "a", "an" and "the" include the plural, unless the context clearly dictates otherwise. In addition, the term "comprises" means "comprising". Furthermore, the term "coupled" does not exclude the presence of intermediate elements between the coupled items. In addition, in the following description and claims, the terms "including" and "comprising" are used freely and should be interpreted to mean "including, but not limited to". The term "training image pair" or "training pair" refers to a pair of images acquired at the same location of the sample and using the same acquisition parameters. As used herein, the term "acquisition parameters" refers to the settings of the charged particle microscope used to acquire one or more images, including at least the dwell time of the beam per pixel, the spot size of the beam, the landing energy of the beam, but not the number of images acquired, or the grid size used in acquiring the images. In general, the "acquisition parameters" of the description herein are parameters that primarily control the magnitude, including the time, of the interaction of the charged particle beam with the sample. Additionally, the term "sparse training image pair" or "sparse training pair" refers to a pair of images of the same location on a sample, but containing different patterns of scanned pixels, acquired using the same acquisition parameters.

[0011] The systems, devices, and methods described herein should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, both alone and in various combinations and subcombinations with one another. The disclosed systems, methods, and devices are not limited to any particular aspect or feature or combination thereof, nor do the disclosed systems, methods, and devices require that any one or more particular advantages exist or problems be solved. While any theory of operation is for ease of explanation, the disclosed systems, methods, and devices are not limited to such theory of operation.

[0012] Although some operations of the disclosed methods are described in a particular order for convenient presentation, it is to be understood that this method of description encompasses reordering, unless a particular order is required by specific terms described below. For example, operations described in a sequence may, in some cases, be reordered or performed simultaneously. Moreover, for simplicity, the accompanying drawings may not show the various ways in which the disclosed systems, methods, and apparatus may be used in conjunction with other systems, methods, and apparatus. In addition, the description may use terms such as "produce" and "provide" to describe the disclosed methods. These terms are high-level abstractions of actual operations that are performed. The actual operations that correspond to these terms may vary, depending on the particular implementation, and are readily discernible to those of ordinary skill in the art.

[0013] Generally, charged particle images, for example, contain various amounts of noise. The various amounts of noise may be based primarily on the dwell time of the probe charged particle beam, although other factors may also affect the noise. With respect to dwell time, the longer the dwell time, the less noise is contained, e.g., the higher the signal-to-noise ratio (SNR), whereas the shorter the dwell time, the more noise is contained, e.g., the lower the SNR. This compromise seems to be easy to obtain an image with less noise, but the type of sample and the volume of the image desired further influence the calculation of the dwell time. For example, soft and delicate samples, e.g., biological samples, may be damaged by long dwell times. It should also be noted that biological samples may typically suffer from a decrease in SNR due to a lack of image contrast, and a longer dwell time may be recommended. Furthermore, the number of images required to image a larger area or volume of the sample may be significantly increased by a long dwell time. In some cases, multiple images of a soft sample may be desired, which may result in a longer dwell time to accommodate the additional noise of the soft sample. Such an imaging workflow can lead to a month's worth of image acquisition. Clearly, this is undesirable and faster, and therefore less traumatic, image acquisition is desirable.

[0014] Previous solutions to the noise problem are available, but the results are typically less than desirable. Such solutions can include classical denoising algorithms and even more recent deep learning-based solutions. Classical approaches are typically based on oversimplified assumptions about the actual noise in the sample image, resulting in over-smoothing of the image. As such, information is lost. For deep learning approaches, the training data is typically not specific enough and may be artificially distorted clean images, mainly for training with different noises. In addition, these deep learning-based denoising networks are trained on noisy images and associated clean images, but the associated noisy clean images are obtained under different settings that affect the denoising ability of the network. Clearly, newer and better solutions are desired.

[0015] One solution to the problem disclosed above is to update the training of a pre-trained blind denoiser, e.g., a pre-trained blind artificial neural network, using only noisy images (training image pairs) that contain similar noise to the sample image that needs to be denoised. For example, the pre-trained blind denoiser can be lightly trained using noisy clean images, but the update can use only noisy image pairs with system-specific noise, which helps the denoiser provide clean images after being updated. In addition, a large dataset of sample images can be acquired under the same acquisition parameters, e.g., settings of a charged particle microscope system, as those used to acquire the training image pairs. After the sample images are acquired, the updated denoiser is used to denoise the sample images.

[0016] Some advantages of the present solution over previous solutions are the use of a small set of training images, which improves the denoising ability of the denoiser and increases the training time. For example, the denoiser update training takes less than an hour. Furthermore, the overall image-to-data time can be significantly increased by using short dwell times for both the training image pairs and the large dataset of sample images. Furthermore, the denoising can be much better than traditional deep learning denoisers due to training on images with system-specific noise that the denoiser would encounter in the large dataset of sample images. Therefore, the denoised images are of higher quality than other methods.

[0017] As mentioned above, this technique can also be used to reconstruct sparse images with only minor changes to the training image pairs. While the denoiser training image pairs are of the same location on the sample, the sparse training image pairs can have some overlapping areas and some non-overlapping areas, which provides training that allows a sparse reconstruction network, e.g., an artificial neural network, to fill in the sparse (empty) areas of the images.

[0018] FIG. 1 is an example of a charged particle microscope system 100 according to one embodiment of the present disclosure. The charged particle microscope (CPM) system 100, or simply the system 100, includes at least a CPM environment 102, a network 104, one or more servers 106, and an artificial neural network 114. The CPM system 100 can be used to investigate and image samples of various sizes and structures. As an example, the CPM system 100 can be implemented, at least in part, at a commercial or research site and used to image various samples in the field. In some embodiments, the CPM system 100 can be distributed across various locations. For example, the CPM environment 102 can be located at a development site, the network 104 can be distributed locally, regionally, or nationally, or the server 106 can be located at a server farm and coupled to the CPM environment 100 via the network 104. Regardless of the organization of the CPM system 100, the system 100 may be used to implement one or more artificial neural networks (ANNs) 114 in conjunction with performing various image restoration tasks, such as at least noise removal and / or sparse reconstruction

[0024] . The CPM environment 102 includes any type of charged particle microscope, although the application of the neural network and image restoration techniques disclosed herein is not limited to charged particle microscopes, which are used for illustrative purposes only. Exemplary CPMs include scanning electron microscopes (SEMs), transmission electron microscopes (TEMs), scanning transmission electron microscopes (STEMs), focused ion beams (FIBs), and dual beam (DB) systems that include both electron and ion beam capabilities, to name just a few. The CPM environment 102 can be used to obtain electron or ion images of a sample, some of which may be images of successive slices of the sample, such as for imaging a volume of the sample. Volumetric imaging, sometimes referred to as slices and views, involves imaging a surface of the sample, removing that surface, and imaging the newly exposed surface, repeated until the desired volume of the sample is imaged. The CPM environment 102 can include various aspects that can be included in a single tool or located in separate tools. For example, the CPM environment 102 may include an imaging platform 108, e.g., a SEM, TEM, or STEM, a sample preparation platform 110, and one or more controllers 112. Of course, each platform 108 and 110 may include more than one microscope / sample preparation tool as well.

[0019] The imaging platform 108 is used to acquire images of the samples, some of which may, but need not, have been prepared by the sample preparation platform 110. The images are acquired using an electron and / or ion source that irradiates the sample with a respective beam of charged particles. For example, an ion and / or electron beam column provides a respective beam of ions and / or electrons. In some embodiments, the charged particle beam imaging is acquired by a scanned beam moved across the sample, such as by rastering the beam, while in other embodiments, the charged particle beam is not scanned. Backscattered, secondary, or transmitted electrons are then detected, for example, and a grayscale image is formed based thereon. The image includes a grayscale contrast depending on the material of the sample, with changes in the grayscale indicating changes in material type or crystal orientation. The imaging platform 108 may be controlled by an internal control (not shown), a controller 112, or a combination thereof.

[0020] The sample preparation platform 110 forms some of the samples that are imaged by the imaging platform 108. Of course, the imaged samples may be formed by other tools (not shown). The sample preparation 110 may be, for example, a DB system that uses a FIB to assist in the preparation and removal of thin samples from larger samples, such as by ion milling, ion induced etching, or a combination thereof, and other processes to prepare the sample for imaging. Other processes may include, but are not limited to, planarization mill / etch, fiducial generation, cross section formation, top-down lamella preparation, and the like. The sample preparation platform 110 may also include an electron beam imaging component that allows the sample preparation process to be monitored, although an electron beam imaging component is not required. In some embodiments, the sample preparation platform 110 may include other physical preparation aspects used to prepare the sample for the imaging platform 108 - lasers, cutting tools, resin encapsulation tools, cryotools, and the like. The sample preparation platform 110 may be controlled by an internal control (not shown), a controller 112, or a combination thereof.

[0021] The network 104 may be any type of network for transmitting signals between the CPM environment 102 and the server 106. For example, the network 104 may be a local area network, a large area network, or a distributed network such as the Internet, a telephone backbone, or a combination thereof.

[0022] The server 106 may include one or more computing platforms, virtual and / or physical, capable of executing code for various algorithms, neural networks, and analysis suites. Although not shown, a user of the CPM environment 102 may access the server 106 to retrieve data, update software code, perform analytical tasks on the data, etc., from the user's local computing environment (not shown) through the network 104. In some embodiments, the user accesses image data stored on the server 106 and implements image restoration, e.g., denoising and / or sparse reconstruction, using the ANN 114 (which may run on the server 106 or the CPM environment 102) from the user's local computing environment.

[0023] In operation of the noise removal embodiment, the imaging platform 108 is set to acquire an image using desired acquisition parameters. The acquisition parameters may determine at least the energy of the charged particle beam, the spot size of the beam, and the dwell time. An exemplary dwell time may be 0.3 μs to 3.0 μs. The dwell time determines the time that the charged particle beam strikes the sample at each pixel of the acquired image, and a short dwell time reduces system drift, beam damage, and sample charge state, among other advantages. As used herein, a "pixel" may refer to a location within the visible area of ​​the acquired image, and the beam may be moved between pixels to acquire images within the visible area. The visible area may be adjustable based on the desired field of view. Once the acquisition parameters are set, a small number of training image pairs of the sample are acquired. Each training image pair includes at least two images of the same location on the sample, which may be referred to as spot images or spot training pairs. Acquiring two or more spot training images of the same location provides information about charged particle specific system noise. In some embodiments, the number of training image pairs may be between 100 and 1000 total images, e.g., between 200 and 2000 images. The number of training image pairs acquired may be adjusted based on how close the noise in pre-training the ANN 114 is to the system-specific noise of the training image pairs. If the noise is close, fewer training image pairs may be required, and vice versa.

[0024] After a small number of training image pairs are acquired, the pre-trained ANN 114 may be updated to fine-tune the ANN 114 to the system-specific noise in the training image pairs. The update of the ANN 114 may use only training image pairs that result in noisy images due to a short dwell time. Furthermore, the training image pairs are not labeled as is typically done in training artificial neural networks. Additionally, the associated clean images are not used during the training update of the ANN 114. Thus, the ANN 114 may use backpropagation of each training image pair to learn the noise of the system. This training technique may be referred to as noise-to-noise training, which takes advantage of the statistical nature of noise. The statistical nature of noise is based on the stance that the average of many noise sources is zero. Therefore, averaging multiple images in the same location can suppress the noise and reveal a clear image. Furthermore, training with a small number of training image pairs allows for fast training due to fast convergence that leads to a good match with the system-specific noise.

[0025] Before, after, or in parallel with updating the training of the ANN 114, a large dataset of sample images can be acquired using the same acquisition parameters. This large dataset can result in terabytes of data (e.g., 10,000 or more images) that need to be denoised. Because the same fast dwell time is used, the time to acquire all the sample images can be much shorter than acquiring a traditional large-scale dataset. After acquiring the sample images, updating the ANN 114 can denoise the sample images. This denoising step can be performed by the system 100 or the server 106 and can be initiated by the user at any time.

[0026] Although the above exemplary operations are discussed in a denoising context, the sparse reconstruction techniques can be similarly implemented with modifications to sparse training image pairs as described above. Except for the differences between the training images, the processes of denoising and sparse image reconstruction (which may be collectively referred to as image restoration) are similar, if not the same. With respect to sparse reconstruction, sparse images, e.g., images containing missing data, can be completed, e.g., with the missing data filled in by a trained artificial neural network.

[0027] Although the images provided to the ANN 114 are described as being acquired by the imaging platform 108, in other embodiments the images may be provided by a different imaging platform and provided to the ANN 114 over the network 104. For example, both the training image pairs and the sample image may be provided to the server 106 over the network, and then an update of the ANN 114 may be performed using the training image pairs before denoising the sample image. The charged particle system used for imaging does not need to be housed with the ANN 114, as long as the acquisition parameters used to acquire the training images and the sample images.

[0028] In one or more embodiments, the ANN 114, which may also be referred to as a deep learning system, is a machine learning computing system. The ANN 114 includes a collection of connected units or nodes called artificial neurons. Each connection sends a signal from one artificial neuron to another. The artificial neurons may be aggregated into layers. Different layers may perform different types of transformations on the input.

[0029] One type of ANN 114 is a convolutional neural network (CNN). CNNs are traditionally designed to process data that comes in the form of multiple arrays, such as a color image that is composed of three two-dimensional arrays containing pixel intensities in three color channels. An exemplary architecture of a CNN is configured as a series of stages. The first few stages may consist of two types of layers: convolutional layers and pooling layers. A convolutional layer applies a convolution operation to the input and passes the result to the next layer. Convolution emulates the response of an individual neuron to a visual stimulus. A pooling layer combines the output of a neuron cluster in one layer into a single neuron in the next layer. For example, max pooling uses the maximum value from each of the neuron clusters in the previous layer. In one or more embodiments, the ANN 114 is a CNN configured to reconstruct images, such as by denoising images and / or sparse reconstruction of images.

[0030] 2 is an exemplary method 200 for denoising an image, according to an embodiment of the present disclosure. Method 200 may be implemented in part by a charged particle microscope system and in part by one or more processing cores that may receive data from a charged particle microscope system. An exemplary charged particle microscope system is system 100. Of course, other charged particle microscope systems may be used. In general, method 200 retrains or updates an ANN, such as ANN 114, based on a small number of training image pairs, and then the retrained / updated ANN may be used to denoise a large dataset of sample images.

[0031] Method 200 can begin at process block 201, which includes setting image acquisition parameters, such as, for example, electron beam dwell time, electron beam landing energy, beam spot size, etc. In some embodiments, the dwell time may be the largest factor affecting image quality and should be the same for all images acquired for use in method 200. For example, a short (e.g., 0.3 μsec) dwell time helps limit or reduce drift that occurs during image acquisition, further speeding up the image acquisition process. Limiting drift helps keep the underlying image signal, e.g., noise-free images, the same for images of the same location, and increasing the speed of the image acquisition process allows for faster data acquisition and, in the case of delicate samples, may further limit sample damage.

[0032] Processing block 201 may be followed by processing block 205, which includes acquiring training image pairs. The training image pairs may be pairs of images of the same spot of the sample, where each image in each image pair will have different noise. The number of training image pairs acquired may be relatively small, especially compared to the number of sample images that will be acquired later, but the number acquired may also be affected by the similarity of the image noise used to pre-train the denoising model, e.g., ANN.

[0033] Method 200 further includes processing block 203, which includes providing a pre-trained denoiser. The performance of this block may be performed in parallel with processing blocks 201 and 205, before those blocks, or after those blocks. The pre-trained denoiser model is a pre-trained ANN, such as ANN 114, trained to denoise images. In general, the pre-training can be done using any noisy images, and need not be of the same sample type of images of processing block 205, or even have the same type of noise. However, the more similar the training images are to those acquired in processing block 205 in terms of noise, the more rapid the denoiser may be retrained or updated.

[0034] Processing block 203 may be followed by processing block 207, which includes updating the denoiser using training image pairs. The denoiser, i.e., ANN 114, is updated using images with noise similar to that which will be present in the sample image. By updating the denoiser with images with noise similar to that of the sample image, the denoiser is more likely to provide a clean image, e.g., an image with reduced noise, in return. Furthermore, because each training image pair is of the same spot and differs only due to noise, the denoiser is able to learn the noise and remove more of it without affecting the rest of the image.

[0035] Method 200 further includes processing block 209, which includes acquiring a large dataset of sample images. The sample images should be acquired using the same acquisition parameters as those used to acquire the training image pairs to ensure that the sample images contain similar noise. Processing block 209 may be performed any time after processing block 201 is performed, such as before, after, or in parallel with processing blocks 203-207. The large dataset of sample images will be of the same sample as used to acquire the training image pairs, but will cover a much larger area or volume of the sample. Generally, the number of sample images will be substantially greater than the number of training image pairs. For example, the number of sample images can easily exceed 10,000 images, resulting in terabytes of data.

[0036] Processing blocks 209 and 207 may be followed by processing block 211, which includes denoising the sample image using the updated noise detector. Denoising the sample image using a denoiser, e.g., an ANN, may occur when the sample image is obtained by an ANN coupled to the charged particle microscope, or denoising the image may be performed on a user's desktop computer coupled to access the sample image and the ANN.

[0037] 3 is an exemplary method 300 for sparse image reconstruction, according to one embodiment of the present disclosure. Method 300 may be implemented by a charged particle microscope system, such as system 100, or by one or more computing environments coupled to receive images from a charged particle microscope system. Generally, method 300 can reconstruct a sparse image to provide a complete image. For example, an image that includes 10-20% of pixels with information can be reconstructed to provide a complete 100% of the pixels with information of the sample.

[0038] In addition, method 300 may be implemented by an ANN, such as ANN 114, using similar techniques used to denoise images as discussed in method 200. The sparse reconstruction uses a trained ANN similar to that of method 200, and there are only minor differences between the trained image pairs used in methods 200 and 300.

[0039] Method 300 may begin at process block 301 with obtaining sparse reconstruction training image pairs of a sample. A sparse reconstruction training image pair is a pair of images of overlapping spots on a sample, but not of the exact same spot. Each pair should have overlapping and non-overlapping areas, so that an ANN is trained to fill in sparse or non-existent areas from the images. For example, one image of the training pair may include data for pixels 1, 3, and 5, while the other image of the training pair may include data for pixels 2, 4, and 5. Alternatively or additionally, the training image pairs are of the same spot on the sample, but each picture of the training pair uses / includes a different mix of pixels from the image of that spot, such that there is no pixel overlap. As in method 200, the number of sparse training image pairs may be small, especially compared to the number of sample images.

[0040] Processing block 301 may be followed by processing block 303, which includes updating an artificial neural network using the sparse training image pairs. The training pairs may be provided to a pre-trained ANN to be updated for sparse reconstruction. The pre-trained ANN may be partially trained to reconstruct an image, for example, trained using the sparse image and the associated complete image.

[0041] Processing block 303 may be followed by processing block 305, which includes acquiring a large dataset of sparse sample images of the sample. The sparse images may be acquired using the same parameters used to acquire the training image pairs. Although processing block 303 is shown to be performed after processing block 303, in other embodiments processing block 305 may be performed before or in parallel with processing block 303.

[0042] Processing block 305 may be followed by processing block 307, which includes reconstructing an image of the sample using the updated artificial neural network. The updated ANN reconstructs the sparse image by determining the missing data and including it with the associated output image.

[0043] 4 is an example image sequence 400 illustrating image denoising according to one embodiment of the disclosure. Image sequence 400 includes a training image pair and associated denoised images provided by an updated denoiser. Images 402 and 404 are example noisy images, e.g., a training image pair, of the same location of a sample that are used in updating the denoiser, such as in processing block 207 of method 200. Image 406 is an example of a denoised image of the same location that has been denoised by the updated denoiser. Although the example image 406 is blurred in some areas, the overall image contains much less noise than the noise shown in images 402 and 404.

[0044] 5 is an example image sequence 500 illustrating sparse reconstruction, according to one embodiment of the present disclosure. Image 502 is a sparse image of a sample and may be part of a larger dataset of sample images and / or one image of a training image pair. In contrast, image 504 is a reconstructed image provided by a trained ANN, such as by processing block 307 of method 300.

[0045] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing devices may be hardwired or may include digital electronic devices such as one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) to perform the techniques, and may include one or more general-purpose hardware processors or graphic processing units (GPUs) that are permanently programmed or programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination to execute the techniques. Such dedicated computing devices may also combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to realize the techniques. The dedicated computing devices may be desktop computer systems, portable computer systems, handheld devices, network devices, or other devices that incorporate hardwired and / or program logic to implement the techniques.

[0046] For example, Figure 6 is a block diagram illustrating a computer system 600 on which an embodiment of the present invention may be implemented. The computing system 600 may be an example of computing hardware included in the CPM environment 102, such as the controller 112, the imaging platform 108, the sample preparation platform 110, and / or the server 106. In addition, the computer system 600 may be used to implement one or more neural networks disclosed herein, such as the ANN 114. The computer system 600 includes at least a bus 640 or other communication mechanism for communicating information and a hardware processor 642 coupled with the bus 640 for processing information. The hardware processor 642 may be, for example, a general-purpose microprocessor. The computing system 600 may be used to implement the methods and techniques disclosed herein, such as the methods 200 and 300.

[0047] Computer system 600 also includes a main memory 644, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 640 for storing information and instructions executed by processor 642. Main memory 644 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 642. Such instructions, when stored in a non-transitory storage medium accessible to processor 642, render computer system 600 a special-purpose machine customized to perform operations specified in the instructions.

[0048] Computer system 600 further includes a read only memory (ROM) 646 or other static storage device coupled to bus 640 for storing static information and instructions for processor 642. A storage device 648, such as a magnetic disk or optical disk, is provided and coupled to bus 640 for storing information and instructions.

[0049] Computer system 600 may be coupled via bus 640 to a display 650, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 652, including alphanumeric and other keys, is coupled to bus 640 for communicating information and command selections to processor 642. Another type of user input device is a cursor control 654, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 642 and for controlling cursor movement on display 650. This input device typically has bidirectional degrees of freedom in two axes, a first axis (e.g., "x") and a second axis (e.g., "y"), that allow the device to specify a position in a plane.

[0050] Computer system 600 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that in combination with a computer system renders or programs computer system 600 into a special-purpose machine. According to one embodiment, the techniques described herein are performed by computer system 600 in response to processor 642 executing one or more sequences of one or more instructions contained in main memory 644. Such instructions may be read into main memory 644 from another storage medium, such as storage device 648. Execution of the sequences of instructions contained in main memory 644 causes processor 642 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0051] The term "storage media" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 648. Volatile media include dynamic memory, such as main memory 644. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape or other magnetic data storage media, CD-ROMs, other optical data storage media, media with patterns of physical holes, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, other memory chips or cartridges, Content Addressable Memory (CAM), and Ternary Content Addressable Memory (TCAM).

[0052] Storage media is distinct from, but may be used in combination with, transmission media. Transmission media involves transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 640. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0053] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 642 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 600 may receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 640. The bus 640 carries the data to main memory 644, from which the processor 642 retrieves and executes the instructions. The instructions received by main memory 644 may optionally be stored on storage device 648 either before or after execution by processor 642.

[0054] Computer system 600 also includes a communication interface 656 coupled to bus 640. The communication interface 656 provides a two-way data communication coupling to a network link 658 that is connected to a local network 660. For example, the communication interface 656 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 656 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Or it may be a wireless link implementation. In such an implementation, the communication interface 656 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0055] Network link 658 typically provides data communication through one or more networks to other data devices. For example, network link 658 may provide a connection through a local network 660 to a host computer 662 or to data equipment operated by an Internet Service Provider (ISP) 664. ISP 664 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 666. Local network 660 and Internet 666 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 658 and through communication interface 656, which carry the digital data to and from computer system 600, are exemplary forms of transmission media.

[0056] Computer system 600 can send messages and receive data, including program code, through the network(s), network link 658 and communication interface 656. In the Internet example, a server 668 might transmit a requested code for an application program through the Internet 666, ISP 664, local network 660 and communication interface 656.

[0057] The received code may be executed by processor 642 as it is received, and / or stored in storage device 648, or other non-volatile storage device for later execution.

[0058] In some instances, values, procedures, or devices are referred to as "lowest," "best," "minimum," etc. It will be understood that such descriptions are intended to indicate that a selection from among many functional options may be made, and that such a selection is not necessarily better, smaller, or otherwise preferable than other options. Additionally, the selected values ​​may be obtained by numerical or other approximation means, or may merely be approximations to theoretically correct values.

[0059] [Appendix 1] 1. A method for denoising a charged particle image, the method comprising: acquiring a plurality of training image pairs of a sample, each training image in each of the plurality of training image pairs being an image of a same location on the sample, and each image in the plurality of training image pairs being acquired using the same acquisition parameters; updating an artificial neural network based on the plurality of training image pairs; and and denoising the plurality of sample images using the updated artificial neural network; The method of claim 1, wherein the plurality of sample images are acquired using the same acquisition parameters used to acquire the plurality of training image pairs. [Appendix 2] 2. The method of claim 1, wherein the noise of each image in the plurality of training image pairs is noise characteristic of a charged particle microscope used to acquire the training image pairs. [Appendix 3] 3. The method of claim 2, wherein the acquisition parameters affect the noise specific to the charged particle microscope used to acquire the training image pairs. [Appendix 4] 4. The method of claim 3, wherein the acquisition parameters include at least a dwell time of the charged particle beam. [Appendix 5] 2. The method of claim 1, wherein the noise of each image in the plurality of training image pairs is similar to the noise of each image in the plurality of sample images. [Appendix 6] 2. The method of claim 1, wherein updating the artificial neural network based on the plurality of training image pairs comprises updating the artificial neural network based on noise characteristic of the plurality of training image pairs and a charged particle microscope used to acquire the plurality of sample images. [Appendix 7] 2. The method of claim 1, further comprising acquiring a plurality of images of the sample, the sample images covering a larger area of ​​the sample than covered by the plurality of training image pairs. [Appendix 8] 2. The method of claim 1, further comprising providing a pre-trained artificial neural network, wherein the pre-trained artificial neural network is not trained on images having noise similar to that of any of the plurality of training pairs or the plurality of sample images. [Appendix 9] 2. The method of claim 1, wherein a number of images in the plurality of training image pairs is less than a number of images in the plurality of sample images. [Appendix 10] 2. The method of claim 1, wherein the multiple training image pairs include spot images of different locations on the sample. [Appendix 11] 1. A system comprising: a charged particle microscope imaging platform; a controller coupled to at least control the charged particle microscope imaging platform; The controller is coupled to a non-transitory computer readable medium containing code, the code, when executed by the controller, providing the system with: acquiring a plurality of training image pairs of a sample, each training image of each of the plurality of training image pairs being an image of a same location on the sample, and each image of the plurality of training image pairs being acquired using the same acquisition parameters; updating an artificial neural network based on the plurality of training image pairs, the artificial neural network being included with or coupled to the system; and denoising the plurality of sample images using the updated artificial neural network; The system wherein the plurality of sample images are acquired using the same acquisition parameters used to acquire the plurality of training image pairs. [Appendix 12] 12. The system of claim 11, wherein the noise of each image in the plurality of training image pairs is noise characteristic of a charged particle microscope used to acquire the training image pairs. [Appendix 13] 13. The system of claim 12, wherein the acquisition parameters affect the noise characteristic of the charged particle microscope used to acquire the training image pairs. [Appendix 14] 14. The system of claim 13, wherein the acquisition parameters include at least a dwell time of the charged particle beam. [Appendix 15] 12. The system of claim 11, wherein the noise of each image in the plurality of training image pairs is similar to the noise of each image in the plurality of sample images. [Appendix 16] 12. The system of claim 11, wherein updating the artificial neural network based on the plurality of training image pairs includes updating the artificial neural network based on noise characteristic of the plurality of training image pairs and a charged particle microscope used to acquire the plurality of sample images. [Appendix 17] 12. The system of claim 11, wherein the computer-readable medium further comprises code that, when executed by the controller, causes the system to acquire the plurality of sample images, the sample images covering a larger area of ​​the sample than covered by the plurality of training image pairs. [Appendix 18] 12. The system of claim 11, wherein the computer readable medium further comprises code that, when executed by the controller, causes the system to provide a pre-trained artificial neural network, wherein the pre-trained artificial neural network is not trained on images having noise similar to that of any of the plurality of training pairs or the plurality of sample images. [Appendix 19] 12. The system of claim 11, wherein a number of images in the plurality of training image pairs is less than a number of images in the plurality of sample images. [Appendix 20] 12. The system of claim 11, wherein the plurality of training image pairs include spot images of different locations on the sample.

Claims

1. 1. A method for denoising a charged particle image, the method comprising: acquiring a plurality of training image pairs of a sample, each training image in each of the plurality of training image pairs being an image of a same location on the sample, and each image in the plurality of training image pairs being acquired using the same acquisition parameters; updating an artificial neural network based on the plurality of training image pairs; and and denoising the plurality of sample images using the updated artificial neural network; the plurality of sample images are acquired using the same acquisition parameters used to acquire the plurality of training image pairs; updating the artificial neural network based on the plurality of training image pairs includes updating the artificial neural network based on the plurality of training image pairs and noise characteristic of a charged particle microscope used to acquire the plurality of sample images; The method of claim 1, wherein the artificial neural network is a pre-trained artificial neural network generated by training it to denoise images.

2. The method described in claim 1, wherein the noise of each image of the plurality of training image pairs is noise characteristic of the charged particle microscope used to acquire the training image pairs.

3. The method described in claim 2, wherein the acquisition parameters affect the noise specific to the charged particle microscope used to acquire the training image pair.

4. The method of claim 3, wherein the acquisition parameters include at least a dwell time of the charged particle beam.

5. The method described in claim 1, wherein the noise of each image in the plurality of training image pairs is similar to the noise of each image in the plurality of sample images.

6. The method of claim 1, further comprising acquiring a plurality of sample images, the sample images covering an area of ​​the sample that is larger than that covered by the plurality of training image pairs.

7. The method of claim 1, wherein the pre-trained artificial neural network is not trained on images having noise similar to that of any of the plurality of training image pairs or the plurality of sample images.

8. The method described in claim 1, wherein the number of images in the plurality of training image pairs is less than the number of images in the plurality of sample images.

9. The method of claim 1, wherein the multiple training image pairs include spot images of different locations on the sample.

Images