Wide field of view and improved resolution electron tomography imaging method

Abstract

Method for training an automatic artifact correction model in an electron tomography image of a sample, comprising the steps of: Performing (201) an acquisition of a first set of projection data of a sample using a transmission electron microscope capable of emitting an electron beam towards the sample, for a plurality of inclination angles of the sample relative to the electron beam belonging to a first angular range, Determining (202) a first set of tomographic reconstructions of the sample from the first set of projection data, Determining (203) a second set of degraded tomographic reconstructions of the sample from a subset of the first set of projection data associated with a subset of inclination angles of the first angular range,Train (204) the artifact correction model to correct the second set of degraded tomographic reconstructions in order to obtain the first set of tomographic reconstructions. Figure 2,

Description

Title of the invention: Wide field of view and improved resolution electron tomography imaging method

[0001] The invention relates to the field of electron tomography, which is an imaging technique that makes it possible to obtain a three-dimensional (3D) image of a nano-object with a resolution on the order of the nanometer.

[0002] The invention relates more specifically to a 3D imaging method by electron tomography allowing both a wide field of view and improved resolution.

[0003] The invention is based on the use of a trained artificial intelligence model to correct certain artifacts in a 3D reconstruction obtained via an electron tomography imaging technique.

[0004] Electron tomography is a 3D imaging technique widely used in the life sciences and materials science to image a sample. This technique involves performing a series of projections by rotating the sample to be imaged within a transmission electron microscope (TEM). This step is followed by an alignment phase and then 3D reconstruction using a dedicated reconstruction algorithm. The sample is placed on a specimen holder which is inserted into the microscope; this holder is tilted to obtain different angles of inclination of the object relative to the electron beam.

[0005] To obtain an accurate 3D reconstruction of the sample, it is desirable to carry out a series of acquisitions by varying the angle of inclination over as complete an angular range as possible, i.e. [-90° ;+90°].

[0006] Furthermore, it is also necessary that the field of view be sufficiently wide so that the reconstructed volume is representative of the sample.

[0007] However, the accessible 3D field of view is closely linked to the geometry of the sample. To perform an acquisition, a sample is generally placed on a dedicated specimen holder which is positioned inside the microscope.

[0008] There are, in particular, two types of sample holders. The first type of sample holder allows for the 3D analysis of samples prepared in the form of a tip. In this configuration, the sample diameter is limited to around one hundred nanometers in materials science. This value can reach 500 nm at 1 pm when the sample is composed mainly of light elements (biological samples, for example).

[0009] A second type of specimen holder allows for 3D analysis of samples in slide form. In this second configuration, the field of view is as wide as the length of the slide (typically on the order of a few microns), and the thickness is about one hundred nanometers.

[0010] The preparation in the form of a slide therefore allows a wider field of view than the preparation in the form of a point.

[0011] However, slide preparation has a disadvantage in terms of resolution. Indeed, in this configuration, a dedicated specimen holder allows the sample to be tilted relative to the electron beam axis within a limited angular range, on the order of [-70°; 70°] or even less if the area of ​​interest is shaded at high angles of inclination.

[0012] Thus, it is not possible to acquire projections over the full angular range when the sample is prepared in the form of a slide. This results in a missing area of ​​information in the Fourier space of the object, inducing anisotropic resolution in the reconstruction.

[0013] Conversely, in the case of a sample prepared in the form of a point, the complete angular range [-90°; 90°] is available. In this case, the Fourier space of the object is sampled without any missing areas and the reconstruction resolution is isotropic.

[0014] Fig. 1a shows an example of reconstruction 101 of a 2D cross-section of a sample obtained over a complete angular range [-90°; 90°], for a sample prepared in the form of a point. Scheme 100 illustrates the sampling of the Fourier space of the object over a complete angular range.

[0015] Figure 11b shows an example of reconstruction 103 obtained for a limited angular range [-60°; 60°] for a sample prepared as a slide. Diagram 102 illustrates the sampling of the Fourier space of the object for this limited angular range. In the example in Figure 11b, since the missing area is in the horizontal direction, the pores and the overall shape of the tip are horizontally distorted, and the particles (resolved in the complete reconstruction 101 of Figure 1aa) are not visible in the degraded reconstruction.

[0016] Thus, preparing a sample in the form of a slide has the disadvantage of degraded resolution although offering a wider field of view than preparing it in the form of a point.

[0017] Several approaches have been proposed to improve the resolution of angle-limited electron tomography. A technique called 'dual-axis tomography' (DN Mastronarde, J Struct Biol 1997, 120(3), 343) reduces the missing angular area by performing a 90° planar rotation of the sample and acquiring a second series of projections. The missing area then has a conical shape, which slightly reduces reconstruction artifacts without achieving isotropic resolution. This technique requires doubling the electron dose and acquisition time, which can damage the sample being analyzed.

[0018] Advanced reconstruction methods have been proposed to improve the quality of results obtained by electron tomography. Examples include "discrete algebraic reconstruction technique" (DART - X. Zhuge et al., Ultramicroscopy 2017, 175, 87) and "compressed sensing" (CS - Z. Saghi et al., Nano Lett. 2011, 11(11), 4666). These approaches have shown promising results, but only on very simple structures and with strong prior knowledge of the gray levels (DART) and the object's sparsity domain (CS). For complex structures, it has been shown that these approaches do not correct artifacts related to the missing angular range (e.g., Y. Jiang et al., Ultramicroscopy 2018, 186, 94). Methods for restoring the sinogram by 'inpainting' have also been proposed to fill the missing area and thus achieve isotropic resolution (R. Tovey et al., Inverse Prob 2019, 35, 024004).The results obtained, illustrated with a very basic example, are inconclusive and offer very little improvement compared to CS methods.

[0019] Recently, deep learning methods have been applied in the field of X-ray tomography to improve the quality of reconstructions from a limited number of projections (sparse acquisitions) or from very noisy acquisitions ('low-dose'). Promising results have been reported in both cases (e.g., Z. Liu et al., J Opt Soc Am A 2020, 37, 422), but very few articles have addressed the specific problem of limited angular range (e.g., Z. Li et al., Sensors 2019, 19, 3941).

[0020] Thus, there is a need for a new electron tomography imaging method that allows both a wide field of view and reconstruction with isotropic resolution.

[0021] The invention proposes the use of a machine learning model to learn the function of correcting artifacts specific to a limited angular range during the acquisition of projections.

[0022] The invention thus makes it possible to carry out acquisitions with samples prepared in the form of a slide, in order to benefit from a wide field of view, while correcting the artifacts visible in the reconstructions due to the limited angular range.

[0023] The invention relates to a method for training an automatic artifact correction model in an image of a sample obtained by electron tomography, the method comprising the steps of: - Acquire a first set of projection data from a sample using a transmission electron microscope capable of emitting an electron beam towards the sample, for a plurality of inclination angles of the sample relative to the electron beam belonging to a first angular range, - Determine a first set of tomographic reconstructions of the sample from the first set of projection data, - Determine a second set of degraded tomographic reconstructions of the sample from a subset of the first projection data set associated with a subset of inclination angles from the first angular range, - Train the artifact correction model to correct the second set of degraded tomographic reconstructions in order to obtain the first set of tomographic reconstructions.

[0024] According to a particular aspect of the invention, the model is a convolutional artificial neural network or a generative adversarial artificial neural network.

[0025] According to a particular aspect of the invention, each tomographic reconstruction corresponds to a two-dimensional cross-section of a three-dimensional volume.

[0026] According to a particular aspect of the invention, the first angular range is complete and the subset of tilt angles corresponds to a reduced angular range or to an angular range undersampled from the first angular range.

[0027] According to a particular aspect of the invention, the sample is prepared in the form of a point.

[0028] The invention also relates to a method for imaging a sample by electron tomography, the method comprising the steps of: - To acquire a set of projection data from a sample using a transmission electron microscope capable of emitting an electron beam, for a plurality of inclination angles of the sample relative to the electron beam, belonging to a reduced angular range, - To determine a first set of tomographic reconstructions of the sample from said projection data set, - To receive an automatic artifact correction model trained using the training method according to the invention to correct artifacts related to the reduced angular range, - Execute said model for the first set of tomographic reconstructions so as to generate a second corrected set of tomographic reconstructions corresponding to an increased angular range

[0029] According to a particular aspect of the invention, the sample is prepared in the form of a slide.

[0030] According to a particular aspect of the invention, the sample to be imaged is identical to the sample used to train the automatic artifact correction model.

[0031] The invention also relates to an electron tomography imaging device comprising a transmission electron microscope and a processing unit adapted to perform the steps of one of the processes according to the invention, as well as a computer program comprising instructions which lead the imaging device according to the invention to perform the steps of one of the processes according to the invention.

[0032] Other features and advantages of the present invention will become more apparent from the following description in relation to the following accompanying drawings.

[0033] [Fig. la] represents an example of 2D reconstruction of a sample for an acquisition with full angular range,

[0034] [Fig. 1b] represents an example of 2D reconstruction of a sample for an acquisition with limited angular range,

[0035] [Fig.2] represents a flowchart of a method for training a model of automatic correction of artifacts according to an embodiment of the invention,

[0036] [Fig.3] represents an illustrative diagram of the training described in [Fig.2],

[0037] [Fig.4] represents a flowchart of a tomography imaging process electronics using the model trained by means of the [Fig.2] method

[0038] [Fig.5] illustrates an example of results obtained through the invention,

[0039] [Fig. 6] represents a schematic of a configured tomography imaging device to implement the invention.

[0040] The present invention comprises two phases: a first phase in which an artificial intelligence model is trained to learn to correct artifacts related to a reduced angular acquisition range and then a second phase in which the trained model is used to correct these artifacts.

[0041] Figure 2 describes the steps for implementing the method of training an automatic artifact correction model according to an embodiment of the invention.

[0042] The method begins in step 201 with the acquisition of a series of projections using a transmission electron microscope. The series of projections is associated with a series of inclination angles that vary over a complete angular range [-90°; 90°] with a predefined increment.

[0043] To perform an acquisition over a complete angular range, one solution is to prepare the sample to be imaged in the form of a tip and insert it into a dedicated specimen holder. In this configuration, the specimen holder is inclined, at each new acquisition, at a different angle relative to the electron beam emitted by the microscope. The specimen holder rotates relative to the electron beam so as to cover the complete angular range [-90°; 90°].

[0044] At the end of this first step 201, we obtain a set of projections whose number corresponds to the number of angles of inclination, over the angular range complete ([-90° :+90°]). Each projection has a predefined resolution that depends on the characteristics of the microscope.

[0045] In step 202, a tomographic reconstruction algorithm is then applied to determine, from all the projections measured in step 201, a set of 2D cross-sectional images of a volume containing the sample to be analyzed. The number of images (equal to the number of 2D cross-sections) is defined by the resolution of the projections. Typically, it is on the order of several thousand. An example of a tomographic reconstruction algorithm is the SIRT algorithm ("Simultaneous Iterative Reconstruction Technique") (P. Gilbert, J. Theor. Biol. 1972, 36, 105). Other reconstruction algorithms can be used, such as the FBP algorithm (J. Frank (Ed.), Electron tomography, methods for three-dimensional visualization of structures in the cell, 2006, pp. 245-274), which has the advantage of greater execution speed.

[0046] In step 203, the same tomographic reconstruction algorithm is applied as in step 202, but this time on only a portion of the set of measured projections, corresponding to a reduced angular range. The reduced angular range is, for example, [-60°; 60°] or [-70°; 70°]. It can also be undersampled relative to the full angular range. In other words, in this case, the angular increment chosen for the reduced angular range is greater than that chosen for the full angular range.

[0047] The subset of projections selected in step 203 can also be noisy via the addition of noise in order to simulate an acquisition degraded by a certain level of noise.

[0048] In general, the subset of projections selected in step 203 constitutes a degraded set of projections compared to the complete set used in step 202.

[0049] In step 203, a second, degraded set of tomographic reconstructions is thus obtained. This second set corresponds, for example, to an acquisition that would have been carried out by preparing a sample in the form of a slide and not a point, which has the disadvantage of an angular range as described in the introduction.

[0050] In step 204, an artificial intelligence model is trained to reconstruct the first complete set of tomographic reconstructions from the second degraded set of tomographic reconstructions.

[0051] The model used is, for example, a convolutional artificial neural network or a generative adversarial artificial neural network. For example, it could be a U-net type neural network.

[0052] A possible example of a U-Net architecture is a network comprising four layers. Each layer is a succession of two convolutional layers of 16, 32, 64, and 128 kernels (with a kernel of size 3x3) respectively, followed by normalization by batch and a ReLU activation function. Between each layer, 2x2 max-pooling is performed during the encoding phase and 2x2 oversampling during the decoding phase. Skip connections are added between the encoder and decoder to use the feature maps, learned during the encoding process, for decoding. Two convolutional layers with a single kernel (size 1x1), batch normalization, and a ReLU activation function are added to recover an image at the network output. Figure 3 illustrates the principle of training such a model. The model parameters are optimized to minimize the distance or cost function between the complete reconstructions of the first set and the images restored from the degraded reconstructions of the second set.In other words, the model 300 is trained to learn an artifact correction function that corrects a set of degraded reconstructions 302 provided as input to the model, generating as output a set of corrected reconstructions as close as possible to the complete set 301. The artifacts present in the reconstructions (2D cross-section images) are related to a limited angular range during the acquisition of the projections and / or the presence of measurement noise. In particular, the artifacts to be corrected are elongations and / or blurring that appears on the image in the direction of the missing angular area.

[0053] The cost function used to train the model is for example based on a PSNR (Peak Signal to Noise Ratio) criterion or an SSIM (Structure Similarity) metric between the images restored by the model and the first complete set of reconstructions 301.

[0054] For example, model 300 is optimized using a backpropagation gradient algorithm to update the parameters, or synaptic coefficients, of the neural network as training progresses.

[0055] For example, about ten reconstructions (2D sections) are excluded from the model training data but reserved for the model validation stage.

[0056] In an alternative embodiment, the model can be directly trained on training data that correspond to 3D volumes rather than 2D cross-sectional images.

[0057] Once the model has been trained, it can be used to correct incomplete reconstructions, in particular those obtained from projections measured for samples prepared in the form of a slide which have an angular range limitation.

[0058] Figure 4 describes the steps for implementing a 3D imaging method of a sample by electron tomography according to an embodiment of the invention.

[0059] The method is particularly applicable to a sample prepared in the form of a slide for which it is not possible to perform acquisitions for a complete angular range.

[0060] The process begins in step 401 by acquiring a series of projections using a transmission electron microscope for a sample of the same nature as that used to carry out acquisition step 201 of the training process described in [Fig.2].

[0061] The difference between step 401 and step 201 is that in step 401, the acquisition is performed over an incomplete or degraded angular range compared to that performed in step 201. This is due, for example, to the fact that the sample is prepared in the form of a slide, which does not allow acquisitions to be made for all possible angles of inclination of the specimen holder on which the slide is fixed relative to the direction of the electron beam. This limitation is mainly due to issues of size and internal operation of the microscope.

[0062] In step 402, the same tomographic reconstruction algorithm used in step 202 is applied to generate a first set of reconstructions. The resulting images are degraded due to the incomplete angular range during acquisition 401.

[0063] In step 403, the artificial intelligence model trained using the method described in [Fig.2] is executed in inference. The trained model takes as input the first set of tomographic reconstructions obtained in step 402 and produces as output a second set of reconstructions 404 corrected for artifacts related to the incomplete angular range.

[0064] In this way, it is possible to obtain artifact-free reconstructions for samples prepared in slide form, which benefit from a wider field of view than samples prepared in tip form. In other words, the resulting 2D section reconstructions have improved resolution with less blurring and distortion of details.

[0065] Figure 5 illustrates an example of the result obtained by applying the invention.

[0066] Image 501 is an example of reconstruction of a 2D section obtained in step 402, i.e. from a set of projections acquired for an incomplete angular range of [-60° ; 60°].

[0067] Image 501 is provided as input to the trained model to generate the restored image 502. The signal-to-noise ratio and similarity metrics are improved for the output image compared to the input image.

[0068] Figure 6 represents a highly simplified diagram of a device capable of implementing the invention. Such a device mainly comprises a transmission electron microscope (TEM) capable of performing a series of projection acquisitions for A sample prepared as a point or slide using a dedicated specimen holder. The device also includes a processing unit (PU) configured to implement the invention from the measured projections, specifically to execute the reconstruction algorithm and the automatic correction model. The device may also include a display screen or other interface for viewing the resulting 2D section reconstructions.

[0069] The processing unit TU can be implemented using hardware and / or software components, in particular one or more processors, one or more memories. The invention can thus be implemented as a computer program comprising instructions for its execution.

[0070] In one embodiment of the invention, the acquisitions made to generate the projections used as training data for the model (step 201) are made with a smaller angular step than that used for imaging a sample prepared in the form of a slide (step 401) in order to generate higher quality reconstructions for training the model.

[0071] In another embodiment of the invention, the model is trained on a sample of a first type of material, and then other models are trained from this first model using a transfer learning technique for samples of slightly different materials. An advantage of this embodiment is that it avoids the need to completely retrain each model for each new sample to be analyzed.

[0072] The invention makes it possible to correct artifacts related to a reduced angular range during the acquisition of projections, but it also makes it possible to correct artifacts related to measurement noise in a similar way.

Claims

Demands

1. A method for training an automatic artifact correction model in an electron tomography image of a sample, the method comprising the steps of: Performing (201) an acquisition of a first set of projection data of a sample using a transmission electron microscope capable of emitting an electron beam towards the sample, for a plurality of inclination angles of the sample relative to the electron beam belonging to a first angular range, Determining (202) a first set of tomographic reconstructions of the sample from the first set of projection data, Determining (203) a second set of degraded tomographic reconstructions of the sample from a subset of the first set of projection data associated with a subset of inclination angles of the first angular range,Train (204) the artifact correction model to correct the second set of degraded tomographic reconstructions in order to obtain the first set of tomographic reconstructions.

2. Method for training an automatic artifact correction model according to claim 1 wherein the model is a convolutional artificial neural network or a generative adversarial artificial neural network.

3. Method for training an automatic artifact correction model according to any one of the preceding claims wherein each tomographic reconstruction corresponds to a two-dimensional cross-section of a three-dimensional volume.

4. A method for training an automatic artifact correction model according to any one of the preceding claims wherein the first angular range is full and the subset of tilt angles corresponds to a reduced angular range or to an angular range undersampled from the first angular range.

5. Method for training an automatic artifact correction model according to any one of the preceding claims wherein the sample is prepared in the form of a tip.

6. A method for imaging a sample by electron tomography, the method comprising the steps of: Performing (401) an acquisition of a projection data set of a sample using a transmission electron microscope capable of emitting an electron beam, for a plurality of inclination angles of the sample relative to the electron beam, belonging to a reduced angular range; Determining (402) a first set of tomographic reconstructions of the sample from said projection data set; Receiving an automatic artifact correction model trained by means of the training method according to any one of the preceding claims to correct artifacts related to the reduced angular range.Execute (403) said model for the first set of tomographic reconstructions so as to generate (404) a second corrected set of tomographic reconstructions corresponding to an increased angular range,

7. A method for imaging a sample according to claim 6 wherein the sample is prepared in the form of a slide.

8. A method for imaging a sample according to any one of claims 6 or 7 wherein the sample to be imaged is identical to the sample used to train the automatic artifact correction model.

9. Electron tomography imaging device comprising a transmission electron microscope (TEM) and a processing unit (PU) adapted to perform the steps of one of the methods according to any one of claims 1 to 8.

10. Computer program comprising instructions that cause the imaging device according to claim 9 to perform the steps of one of the methods according to any one of claims 1 to 8.

Images