Acquisition and encoding of electron microscope generated images

Abstract

Solves the problems of conventional technology. [Solution] The method comprises configuring a transmission electron microscope according to a first acquisition setting which includes at least one of a first dose rate or a first operating mode. The method further comprises operating the electron microscope to image a radiosensitive sample according to the first acquisition setting during a first period. The method further comprises configuring the electron microscope according to a second acquisition setting which includes at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode after the first period. The method further comprises operating the electron microscope to image a radiosensitive sample according to the second acquisition setting during a second period. The method further comprises generating image data based on first and second data collected during the first and second periods, respectively.

Description

Technical Field

[0001] In charged particle microscopes such as transmission electron microscopes (TEM) and scanning electron microscopes (SEM), various detection techniques can be used to obtain information about a sample. SEM and TEM techniques can be used to image various types of samples, including the surface / inside of cells, the structure of protein molecules, the organization of molecules in viruses and cytoskeletal filaments, etc. The information obtained about the sample can be represented using a large amount of data (e.g., occupying a large amount of memory space).

Summary of the Invention

[0002] In some embodiments, a computer-implemented method includes configuring a transmission electron microscope according to a first acquisition setting that includes at least one of a first dose rate or a first operating mode. The computer-implemented method further includes operating the transmission electron microscope to image a radiation-sensitive sample according to the first acquisition setting during a first period. The computer-implemented method further includes configuring the transmission electron microscope according to a second acquisition setting that includes at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode after the first period. The computer-implemented method further includes operating the transmission electron microscope to image the radiation-sensitive sample according to the second acquisition setting during a second period. The computer-implemented method further includes generating image data based on first data and second data collected respectively during the first period and the second period.

[0003] In some embodiments, a non-temporary computer-readable storage medium containing instructions executable by one or more processors of a transmission electron microscope for triggering an operation includes configuring the transmission electron microscope according to a first acquisition setting which includes at least one of a first dose rate or a first operating mode. The operation further includes operating the transmission electron microscope to image a radiosensitive sample according to the first acquisition setting during a first period. The operation further includes configuring the transmission electron microscope according to a second acquisition setting which includes at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode after the first period. The operation further includes operating the transmission electron microscope to image a radiosensitive sample according to the second acquisition setting during a second period. The operation further includes generating image data based on first and second data collected during the first and second periods, respectively.

[0004] In some embodiments, the transmission electron microscope includes one or more memories for storing instructions and one or more processors configured to execute instructions to cause the transmission electron microscope to perform operations. The operation includes configuring the transmission electron microscope according to a first acquisition setting which includes at least one of a first dose rate or a first operating mode. The operation further includes operating the transmission electron microscope to image a radiosensitive sample according to the first acquisition setting during a first period. The operation further includes configuring the transmission electron microscope according to a second acquisition setting which includes at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode after the first period. The operation further includes operating the transmission electron microscope to image a radiosensitive sample according to the second acquisition setting during a second period. The operation further includes generating image data based on first and second data collected during the first and second periods, respectively. [Brief explanation of the drawing]

[0005] The aforementioned aspects of this disclosure and many associated advantages will be better understood and more readily apparent by referring to the following detailed description in conjunction with the accompanying drawings. [Figure 1] This is an illustrative diagram of a system for acquiring and encoding image data, according to several embodiments. [Figure 2] This is a schematic diagram of an electron microscope system according to several embodiments. [Figure 3] This chart shows different dose rate schedules according to several embodiments. [Figure 4] This chart shows the loss of detection quantum efficiency as a function of accumulated dose in several embodiments. [Figure 5] This is a schematic diagram showing examples of image processing systems according to several embodiments. [Figure 6] This is a schematic diagram illustrating examples of compression systems according to several embodiments. [Figure 7] This is a schematic diagram illustrating examples of generative adversarial networks for training image generation models, according to several embodiments. [Figure 8] This is a schematic diagram illustrating an exemplary architecture of an image generation model according to several embodiments. [Figure 9] This is a flowchart illustrating an exemplary process for training an image generation model using a generative adversarial network, according to several embodiments. [Figure 10] This is a flowchart illustrating an exemplary process for generating encoded image data according to several embodiments. [Figure 11] This is a flowchart illustrating an exemplary process for acquiring image data using several embodiments. [Figure 12] This is a flowchart illustrating an exemplary process for encoding image data according to several embodiments. [Figure 13] This is a flowchart illustrating an exemplary process for compressing image data according to several embodiments. [Figure 14]This is a flowchart illustrating an exemplary process for acquiring and encoding image data, according to several embodiments. [Figure 15] Schematic diagrams of exemplary computer systems usable in systems and methods according to some embodiments of this disclosure are shown.

[0006] In the drawings, similar reference numbers refer to the same parts throughout various drawings unless otherwise specified. Not all instances of elements are necessarily labeled to reduce confusion in the drawings where necessary. The drawings are not necessarily to scale; instead, the focus is on illustrating the principles being explained. [Modes for carrying out the invention]

[0007] Charged beam particle systems used in electron microscopy provide high-resolution imaging by detecting signal electrons (e.g., backscattered electrons, secondary electrons, etc.) generated by the elastic scattering of a beam of electrons emitted from an electron emitter interacting with atoms in a sample. In one example, electrons may be emitted from a cathode electrode heated by an electric current. The emitted electrons are attracted to an anode located downstream of the cathode electrode, forming an electron beam, which is directed towards and interacts with the sample. The current of signal electrons emitted from the electron beam interacting with the sample is measured by one or more electron detectors. This current is used to generate a high-resolution image of the sample and can be represented as image data.

[0008] In conventional charged beam particle systems, image data can be acquired by an image acquisition system over a period of time according to acquisition settings (e.g., dose rate, operating mode, duration, and / or magnification). Acquisition settings can allow for the acquisition of more or less image data. Furthermore, acquisition settings can degrade the sample faster or slower than other image acquisition settings. Additionally, acquisition settings can affect the time required to acquire a given amount and / or quality of image data. After image data is acquired, its storage, transmission, and / or use may be limited or require significant resources (e.g., storage resources, processing resources, network resources) due to its potentially large size.

[0009] This disclosure discloses techniques for using two or more acquisition settings over a period of time during which image data is collected in order to maximize the amount and quality of information obtained from a sample over a given period of time. This disclosure also discloses techniques for encoding and / or compressing image data in order to reduce the resources used by encoded image data compared to image data. The techniques disclosed herein may be used independently of or in combination with other techniques, which may enable more substantial benefits.

[0010] Generally, encoded data represents data in a different format. A basic example of encoding is the decimal number 10, which can be encoded into hexadecimal and represented as 0xA in hexadecimal representation. A decimal-to-hexadecimal encoding scheme makes it possible to represent a decimal representation in hexadecimal representation, and a hexadecimal-to-decimal encoding scheme makes it possible to represent a hexadecimal representation in decimal representation.

[0011] Compression is a type of encoding. Compression can be done to reduce the number of symbols used to represent a given piece of information. Compressed information can be decompressed. Compression can be lossy or lossy. In lossless compression, all data may remain after decompression. In other words, all information represented before the data was compressed can be restored after compression and subsequent decompression. Lossy compression, on the other hand, reduces the amount of data by permanently excluding certain information. In lossy compression, not all data may remain after decompression. In other words, all information represented before the data was compressed may not be restored after compression and subsequent decompression. For example, lossy compression may remove data that is not needed in the application area at hand (e.g., image or video compression where lossy compression results in artifacts in the decompressed image that are invisible or barely visible to a human observer).

[0012] In the first specific example, the acquisition settings of an electron-based imaging system may be adjusted between different time windows (e.g., based on a schedule, based on user input, etc.) to acquire one or more images of a sample. The settings may be dynamically adjusted based on time, the sample being imaged, the accumulated dose, and / or other factors. In the specific example, the dose rate of the acquisition settings may be adjusted over time so that the dose rate increases with time.

[0013] The techniques described herein, including those described as the first specific example, may offer several technical improvements, benefits, and advantages over conventional solutions. For example, conventional methods may use a single acquisition set to generate an image of a sample. The techniques described herein can optimize the imaging process by enabling the acquisition of the same image in a shorter time, or by enabling the acquisition of a better image in the same time. The image acquisition techniques described herein can also reduce the amount of noise in the acquired image compared to conventional techniques, thereby reducing the resources (e.g., network resources, processing resources, storage resources) used to process, transmit, and / or store the image.

[0014] In a second specific example, a data encoding scheme can be used to encode image data generated by an image acquisition system. The image data may or may not have been acquired using techniques for dynamically adjusting the image acquisition settings described above. The encoding scheme may be configured to encode the image data based on the acquisition settings used to generate the image data.

[0015] The techniques described herein, including those described as a second specific example, can provide multiple technical improvements, benefits, and advantages over conventional solutions. For example, conventional coding schemes may be static / pre-configured and thus all image data can be encoded using the same coding scheme. The techniques described herein can encode image data based on the acquisition settings used to generate the image data, so the coding scheme used can be selected based on the optimal coding scheme for encoding the image data. For example, if the image data contains information with little noise, the optimal coding scheme may be one with less loss. Further, if the image data contains information with a lot of noise, since there is room to discard more data while retaining the structural information contained in the image and reducing the memory space occupied by the encoded image data compared to the image data before encoding, a more irreversible coding scheme may be optimal. The coding technique can reduce the resources (e.g., network resources, processing resources, storage resources) used to process, transmit, and / or store the encoded image compared to the image before encoding.

[0016] One such coding scheme can be a machine learning model trained to encode image data. The model can be trained in various ways, such as using a Generative Adversarial Network (GAN). The model can be trained using the image data and the encoded image data as training data, and as a result, the model is trained to generate encoded image data based on the image data. The encoded image data generated by the model can occupy less memory space than the received image data, and as a result, less resources (e.g., memory, processing, and / or network) are used by the encoded image data generated by the model compared to the received image data.

[0017] In a third specific example, the encoded image data may then be compressed and further encoded to reduce the resources (e.g., network resources, processing resources, storage resources) used to process, transmit, and / or store the compressed image as compared to the image before compression. Compressing the encoded image data may be done after determining the reference information. The reference information may include the most likely particle positions and other sample information included in the image data. The reference information may be used to determine how to compress the image data based on the reference information retained in the compressed data.

[0018] The techniques described herein, including those described as a third specific example, may provide a number of technical improvements, benefits, and advantages over conventional solutions. For example, conventionally, a large amount of data is generated by an image acquisition system, and the number and / or size of the resources for storing, processing, and transmitting the image data can be large. The techniques described herein can reduce the resources for storing, processing, and transmitting the image data by compressing the image data so as to lose less valuable information and / or to further compress the information as compared to conventional compression techniques.

[0019] In the remainder of the specification, reference is made to a transmission electron microscope (TEM) as usual, but it will be readily understood by those skilled in the art that this technique is not limited thereto. This design can be used in other types of charged particle microscopes such as a scanning electron microscope (SEM), a scanning transmission electron microscope (STEM), a dual beam system including an ion beam source and an electron beam source, a reflection electron microscope (REM), a circuit editing microscope, etc. Therefore, the present disclosure and the claims should not be considered limited to any particular exemplary microscope considered, but can be widely applied to any number of electron microscopes that may exhibit some or all of the electrical or chemical characteristics of the examples considered.

[0020] Figure 1 is an exemplary diagram of a system 100 for acquiring and encoding image data 108, according to several embodiments. The system 100 may include a computing system 104, an image acquisition system 106, and an image processing system 110.

[0021] The computing system 104 may be a user device (e.g., a laptop, personal computer, telephone, etc.). The computing system 104 may also be a server. The computing system 104 may be capable of receiving input from user 102, for example, via a user interface. In certain embodiments, the input received by the computing system 104 may include instructions for an operating mode (e.g., counting mode, integration mode (also called sampling mode or linear mode)), one or more dose rates, duration, sample information (e.g., sample material), and / or schedule information. Based on the input, the computing system 104 may transmit image acquisition settings to the image acquisition system 106. The image acquisition settings may include any combination of user inputs.

[0022] The image acquisition system 106 may include an electron microscope system (EMS). The image acquisition system 106 can generate image data based on data collected over one or more time periods. The image acquisition system 106 may be configured to operate according to one or more acquisition settings. The acquisition settings for operating the image acquisition system 106 may be based on image acquisition settings received from the computing system 104. The acquisition settings may be based on predetermined acquisition settings that may have been previously configured (e.g., by the user and / or the system). The acquisition settings may be based on inputs received through the user interface of the image acquisition system 106, as described herein. In some embodiments, the acquisition settings may include at least a first acquisition setting that includes at least a first dose rate and / or a first operating mode. The acquisition settings may be based on user input, sample characteristics, the purpose of imaging the sample, and available resources (e.g., network resources, processing resources, memory resources).

[0023] The image acquisition system 106 may operate for a certain period of time to image a radiosensitive sample according to the acquisition settings. After this period, the image acquisition system 106 may be configured according to different acquisition settings that can be acquired, generated, or selected using the techniques described herein. The image acquisition system 106 may operate for a second period of time to image a radiosensitive sample according to different acquisition settings.

[0024] The image acquisition system 106 can generate image data 108 based on first sample data, second sample data, and / or other sample data collected during a first period, a second period, and / or any other period. The image data 108 may include one or more image stacks. An image stack may represent multiple images. Each image in an image stack may be an electronic counting frame. An electronic counting frame may be described using one or more electronic coordinates.

[0025] The image processing system 110 may generate encoded image data 112 using the image data 108. The image processing system 110 may receive the image data 108 from the image acquisition system 106 or another source of image data 108 (e.g., a data store, computing system 104). The image processing system 110 may process the image data 108 using one or more processing techniques to generate encoded image data 112.

[0026] In certain embodiments, the processing technique may include dynamically encoding portions of the image data 108 based on acquisition settings used in the process of acquiring each portion of the image data 108 using an image processing system 110. For example, if a first portion of the image data 108 is acquired using a first acquisition setting and a second portion of the image data 108 is acquired using a second acquisition setting, the first portion of the image data 108 may be encoded using a different encoding scheme than the one used to encode the second portion of the image data 108.

[0027] In certain embodiments, the processing technique may include using an image processing system 110 to encode image data 108 to generate intermediate encoded image data, and then compressing the intermediate encoded image data to generate compressed image data represented by encoded image data 112.

[0028] In certain embodiments, the processing technique may include the image processing system 110 encoding image data 108 using an artificial intelligence (AI) model (e.g., a machine learning (ML) model) trained to generate encoded image data 112.

[0029] Image data 108 and / or encoded image data 112 may be transmitted to a computing system 104 or a downstream consumer (e.g., a database). The processing described with respect to the system components of system 100 and other systems described herein may be performed using any combination of one or more devices. The processing may be performed locally and / or remotely on one or more devices.

[0030] Figure 2 is a schematic diagram of an electron microscope system (EMS) 200 according to several embodiments. The EMS 200 is an example of an image acquisition system (e.g., the image acquisition system 106 described herein). The EMS 200 may include a TEM 202. The TEM 202 can be used to acquire an image of a sample 206. In some examples, the TEM 202 may include an energy spectrometer 204. The energy spectrometer 204 may be used, for example, to acquire an EEL spectrum. The EMS 200 may include optical components to reduce or eliminate defocusing problems that occur during spectrum acquisition. In some examples, the optical components may be biased to a level based on the operating parameters of the EMS 200 and / or the energy spectrometer 204 so that they are dynamically refocused.

[0031] TEM202 may include a source 228, an irradiation system 230, a projection system 208, and various detectors 232, all of which may be controlled by a controller 226. Source 228 may be an electron source such as a Schottky source or a (cold) field emission gun (CFEG), which can provide an electron beam that propagates along the optical axis of TEM202 and interacts with the sample 206. Irradiation system 230 may include several electron-optical components for tuning the electron beam to deliver it to the sample 206. Tuning the electron beam may include collimation, astigmatism correction, and focusing the electron beam onto the sample surface. Projection system 208 may include various electrostatic / magnetic lenses, deflectors, correctors (e.g., astigmatism correctors), etc., which may be used to focus the electron beam emerging from the sample 206 onto one of the various detectors 232. The projection system 208 may be configured to focus an image of the sample 206 at a specific (adjustable) magnification on the detector 232, which is generally referred to as the “imaging mode” of the projection system. Alternatively, the projection system 208 may be configured to focus an image (or “diffraction pattern”) of the angular distribution of emerging electrons at a specific (adjustable) magnification on the detector 232. Such a diffraction pattern is formed at the back focal plane of a first magnifying lens (“objective lens,” not shown in Figure 2). This alternative mode of the projection system is generally referred to as the “diffraction mode,” and the magnification in this alternative mode is generally referred to as the “camera length.” However, for simplicity, when this disclosure refers to “image” or “focus,” it refers to both the “imaging mode” and the “diffraction mode” of the imaging system. Similarly, when this disclosure refers to “magnification setting,” it refers to both the “magnification setting” and the “camera length setting” of the imaging system.

[0032] In some examples, the projection system 208 focuses and adjusts the electron beam for delivery to the energy spectrometer 204. Various detectors 232 may be moved individually in and out of the optical path to provide different detection methods to the TEM 202. Various detectors 232 may include an imaging screen, a TEM camera, and a STEM camera.

[0033] The energy spectrometer 204 may include a dispersion element 212 (equipped with a bias tube 214), an optical component 234, a plurality of optical systems 220, and a detector 222. Many of the components of the energy spectrometer 204 are discussed in relation to Figure 1 and are not revisited for brevity. Other components of the energy spectrometer 204 shown in Figure 1 may be included but are excluded from Figure 2, although they are not specifically shown in Figure 2. As discussed, an additional component, namely the optical component 234, is included in the energy spectrometer 204 to correct the focus gradient of the electron energy loss spectrum across the detector. The optical component 234 may be a single element, such as a magnetic hexapole. Alternatively, the optical component 234 may be a plurality of multipole elements (such as a quadrupole, hexapole, or octupole) which may be electrical or magnetic in nature. In addition to spectral refocusing, the optical component 234 may perform multiple alternative functions simultaneously or non-simultaneously, such as image distortion correction or spectral distortion correction. In Figure 2, the optical component 234 is shown downstream of the dispersion element 212, but this is not the only position for placing the optical component 234. Generally, some elements of the optical component 234 can be placed upstream of the dispersion element 212 (but downstream of the sample 206), and some elements of the optical component 234 can be placed within or after multiple optical systems 220, downstream of the dispersion element 212.

[0034] The controller 226 may include one or more processing cores and memory for storing executable code. In addition, the controller 226 may provide operating voltages to some components of the EMS 200, or may be coupled to a voltage source (not shown) that can provide operating voltages in response to control signals provided by the controller 226. For example, the controller 226 may provide control and / or voltage to the irradiation system 230, the projection system 208, or the optical component 234. Furthermore, the controller 226 may control the operation of the detector 222 and / or receive data from the detector 222. In general, the controller 226 can set the operating parameters of the EMS 200 and adjust the electrical bias of the optical component 234 to dynamically focus the electron beam onto the detector 222 in response to changes in the operating conditions of the EMS 200, such as magnification.

[0035] The controller 226 may receive inputs that constitute the EMS200 (e.g., user input, input from a computing system). The EMS200 may be configured according to acquisition settings. Acquisition settings may include dose rate, operating mode, duration (e.g., 1 second, 5 seconds, 10 seconds, etc.), and / or magnification. The dose rate is the number of electrons received by the sample being imaged over a period of time. For example, the dose rate may be measured in electrons / second. The operating mode may include counting mode or integration mode.

[0036] In counting mode, individual electronic events can be identified, and digital events can be accumulated over time. The quality of image data 108 generated using counting mode may decrease as the dose rate increases over the operating range of counting mode. For example, if a dose rate that is too high is used, a high matching loss occurs, where all individual electrons cannot be localized because they are too close to each other in space and time.

[0037] In integral mode, the dose rate may differ from the dose rate in counting mode (e.g., it may be higher). In integral mode, the maximum dose rate may depend on how linearly the camera behaves within the range used and how pixels saturate to their maximum value. The image quality of the generated image data 108 may be constant as a function of the dose rate. The dose rate may have a constant image quality that is worse than the image quality obtained using counting mode (which can be called Detective Quantum Efficiency (DQE)). Integral mode may have a different operating range from that of counting mode (partially (e.g., overlapping) or entirely (disjointed)). Integral mode may be used during acquisition to accommodate very high doses. In an example where a total dose of 100 electrons per pixel is desired, with an electronic count dose of 0.025 electrons per pixel, integral mode may be used for the desired high dose.

[0038] The controller 226 may configure the EMS 200 to operate for a period of time according to acquisition settings in order to collect sample data of sample 206. Sample 206 may be a radiosensitive sample. The EMS 200 may generate image data 108 as a result of the sample data. In certain embodiments, image data 108 is sample data. In certain embodiments, first sample data and second sample data are combined to represent image data 108.

[0039] In certain embodiments, the controller 226 may configure the EMS 200 to image the sample 206 during a first period by operating according to a first acquisition setting. The controller 226 may then configure the EMS 200 to image the sample 206 during a second period by operating according to a second acquisition setting. During each period, sample data may be collected that can be used to generate image data 108. The first acquisition setting may be the same as or different from the second acquisition setting. The dose rate of the second acquisition setting may be different from (e.g., greater than) the dose rate of the first acquisition setting. The operating mode of the second acquisition setting may be different from the operating mode of the first acquisition setting. The second period may be a different length of time than the first period. The duration for operating the EMS 200 using the acquisition setting may be determined by the rate of radiation damage suffered by the sample 206. The duration for operating the EMS 200 according to the acquisition setting may be based on the previous acquisition setting, dose rate, and / or operating mode. Furthermore, the controller 226 may configure the frame rate used to acquire data using the EMS200. The frame rate may differ between the first and second periods.

[0040] In certain embodiments, the controller 226 may configure the EMS 200 to image the sample 206 by operating according to a first acquisition setting, a second acquisition setting, or other acquisition settings for any number of periods. The acquisition setting may be determined based on a time-dependent function (e.g., exponential, linear), user input, the purpose of imaging the sample 206, and / or the expected radiation damage characteristics of the sample 206.

[0041] If sample 206 is radiosensitive, sample 206 may degrade as the dose accumulates (e.g., as more electrons interact with the sample). Due to radiation damage, electrons detected at the end of the acquisition period may retain less information about the sample than those detected at the beginning of the acquisition period. In addition, events detected in high-dose-rate frames may retain less information about the imaged sample 206 than events detected in low-dose-rate frames. Image data 108 acquired from EMS200 may be more accurate at lower dose rates than at higher dose rates. The image quality (e.g., precision) of image data 108 may degrade as the dose rate increases. Certain embodiments can maximize the amount of information obtained from sample 206 over a given period.

[0042] For example, the controller 226 may change the acquisition settings while acquiring image data 108. For instance, the controller 226 may configure the EMS 200 to acquire first sample data in counting mode for 0.5 seconds at a dose rate of 4 electrons / second (e / p / s). After 0.5 seconds, the controller 226 may configure the EMS 200 to acquire second sample data in integration mode (e.g., single sampling mode) over two frames (e.g., 2 / 320 seconds) at a dose rate of 6080 electrons / second (e.g., 19 electrons / pixel / frame). The first sample data may represent the first frame or set of frames of image data 108, and the second sample data may represent the second frame or set of frames of image data. The first and second sample data may be combined (e.g., concatenated) to generate image data 108. The controller 226 may combine two or more sample data into image data 108.

[0043] To increase the speed at which acquisition settings can be changed, a shutter can be used (e.g., with a PWM-modulated fast blanker). The acquisition setting is set to the maximum desired dose rate, and the fast shutter can be made on and off at a given frequency (e.g., high frequency, low frequency) or camera frame rate (e.g., high camera rate, low camera rate). The shutter speed can be used to control the dose rate. For example, the shutter speed may be changed (e.g., by controller 226) from a first shutter speed to a second shutter speed that is greater or less than the first shutter speed (e.g., changed to 1 / 10 of the first shutter speed). By changing the shutter speed, the acquisition setting can be changed in a shorter period of time than other methods of changing the acquisition setting. For example, the shutter speed can be changed in less than a second, in contrast to other techniques that take several seconds to change the acquisition setting.

[0044] In embodiments where one or more acquisition settings (e.g., dose rate) are adjusted as a function of accumulated dose, sample 206 can be irradiated at a low dose rate (e.g., up to angstroms) when relatively little radiation damage occurs to sample 206. 2 (1 electron per unit). Furthermore, if the dose rate is adjusted as a function of the accumulated dose, sample 206 may be irradiated at a higher dose rate when an increase in radiation damage to sample 206 occurs. Such a dynamic dose rate schedule can be optimized so that the maximum amount of information (e.g., high-quality image data 108) is obtained from sample 206 over a fixed acquisition time. Such an acquisition method may produce high-quality image data 108 and image data 108 that occupy less memory space compared to other acquisition methods. Image data 108 that occupies less memory space can be transmitted and used by other systems that have less computational resource (e.g., network, processing) overhead.

[0045] The optical component 234 may be formed from a multipole element containing two or more conductive elements. In some examples, the optical component 234 may be formed from two opposing conductive elements, as well as from a deflector or shutter-type arrangement housed in a conductor. In other examples, the optical component 234 may be formed from a quadrupole or higher-order multipole element. The excitation of the optical component 234 may be electrical, magnetic, or a combination thereof, or more generally, electromagnetic. Regardless of the physical example, the optical component 234 may be excited to levels based on the operating parameters of the TEM 202 and / or energy spectrometer 204. Any combination of the dispersion element 212, the optical component 234, and the multiple optical systems 220 can be used. For example, in certain embodiments, none of the dispersion element 212, the optical component 234, and the multiple optical systems 220 are included in the EMS 200.

[0046] During operation, the electron beam generated by source 228 at primary energy may be projected toward sample 206, and the electron beam interacts with sample 206. Through this interaction, some electrons may lose energy by an amount related to the various material properties of sample 206. The electron beam then emerging from sample 206 may contain electrons of different energies across a range of energies. The emerging electron beam may then propagate along a different trajectory than the primary energy electrons, affecting the focal point, e.g., the crossover position, along the remaining optical path, including energy spectrometer 204 and TEM 202. In some examples, the operation of any component that can tune (direction-control) the electron beam may then result in defocusing within the spectral plane or in a focal gradient across the spectral plane. In this example, optical component 234 may be excited based on the current operating parameters to refocus the electron beam. In various examples, by exciting optical component 234, the crossover position can be aligned with the spectral plane, and therefore with detector 222.

[0047] Figure 3 is Chart 300, which shows different dose rate schedules according to several exemplary embodiments. Chart 300 has a Y-axis representing the dose rate in electrons / pixels / second. The X-axis is time. This chart shows three simple examples of dose rate schedules: a flat schedule, a linear schedule, and an exponential schedule. A flat dose rate schedule is not a dynamic dose rate schedule because the dose rate does not change over time. On the other hand, the linear dose rate schedule and the exponential dose rate schedule are examples of dynamic dose rate schedules because the dose rate changes over time.

[0048] Each of the exemplary dose rate schedules in Chart 300 accumulates approximately 40 electrons over a certain period (e.g., 4 seconds). The number of electrons accumulated over the dose rate schedule is equal to the area under each dose rate schedule curve. Chart 300 illustrates how electron accumulation is affected by the dose rate.

[0049] Using a flat dose-rate schedule of 10 electrons / pixel / second, 40 electrons accumulate over a period of 4 seconds. Using a linear dose-rate schedule starting at 0 (or slightly above 0) with a gradient of 5 electrons / pixel / second, 40 electrons accumulate over a period of 4 seconds. Using an exponential dose-rate schedule starting at 0, 40 electrons accumulate over a period of approximately 3.5 seconds.

[0050] Each of the dose rate schedules shown induces a different detection efficiency. The detection efficiency of the dose rate can be visualized as the detection quantum efficiency (DQE) as a function of the accumulated dose. The DQE as a function of the accumulated dose is shown for each of the dose rate schedules in Chart 400, as described below.

[0051] As an example, only the simplified dose rate schedule described above is shown in Chart 300. Those skilled in the art who are interested in this disclosure will recognize that other dose rate schedules are possible and may be functions of time or other variables. Furthermore, in certain embodiments, any combination of dose rates or other acquisition settings may vary as a function of time or based on a combination of variables.

[0052] Figure 4 is Chart 400, showing the detection quantum efficiency (DQE) loss as a function of accumulated dose for several exemplary embodiments. The dose rate schedules in Chart 300 induce different detection efficiencies, which are visualized as DQE loss as a function of the accumulated dose of the first two electrons in Chart 400 (expressed as percentage points (e.g., 0.15 = 15%)). The Y-axis of Chart 400 is the DQE loss measured in percentage points. The X-axis of Chart 400 is the accumulated dose measured per electron per pixel. The DQE loss may be the loss compared to the camera's optimal DQE (e.g., the DQE when using the lowest possible dose rate (which typically requires long exposure times)).

[0053] The earlier an electron interacts with the sample (e.g., sample 206) compared to other electrons interacting with the sample, the less degradation occurs, and therefore, higher quality data can be obtained from the sample by the electron. Thus, the first two electrons retain more frequency information about the sample (e.g., protein) (as evidenced by lower DQE loss) than the later electrons (as evidenced by higher DQE loss). The later an electron interacts with the sample compared to other electrons interacting with the sample, the more degradation occurs, and therefore, the electron does not retain as much information about the fine details of the sample. In certain embodiments, later electrons (e.g., electrons after the first and second electrons) may be used for particle picking even if the information retained by the later electrons may have higher DQE loss.

[0054] Chart 400 shows the DQE loss as a function of accumulated dose for the flat dose rate schedule, linear dose rate schedule, and exponential dose rate schedule, respectively, from Chart 300.

[0055] Chart 400 shows that for the first two electrons, the exponential dose-rate schedule is equivalent to a flat dose-rate schedule on a 70% faster camera. In other words, the amount of time required to acquire the same image data (e.g., image data 108) can be reduced by 70% using the exponential dose-rate schedule compared to the flat dose-rate schedule. Furthermore, the exponential dose-rate schedule can achieve the same performance (e.g., DQE loss) as the flat dose-rate schedule using conventional imaging with 40 electrons in a 5 times shorter exposure (0.8 seconds). Since the DQE loss is reduced by using a dynamic dose-rate schedule compared to the flat dose-rate schedule, this can result in reduced storage requirements for acquired image data.

[0056] In the case of a dynamic dose-rate schedule, the DQE loss for a given accumulated dose is lower compared to a flat schedule. Since the sample being imaged is already damaged by the end of the acquisition period (e.g., after previous electrons have interacted with the sample), a low DQE is not a problem. Information acquired during the latter half of the acquisition period may not be used to obtain high-resolution sample information. Information acquired during the latter half of the acquisition period may be used for intermediate processes such as particle picking.

[0057] Figure 5 is a schematic diagram showing examples of image processing systems 500 according to several embodiments. Image processing system 500 may be part of image processing system 110 described with respect to system 100. Image processing system 500 may include a first encoding system 504 and a compression system 516. In certain embodiments, image processing system 500 is used in combination with the dynamic image acquisition setting 502 as described above. Image processing system 500 may be used to generate encoded image data 514 and / or compressed image data 518.

[0058] The first encoding system 504 can receive image data 108 and acquisition settings 502. Image data 108 may be generated by an image acquisition system (e.g., image acquisition system 106, EMS200). Acquisition settings 502 may be used by EMS200 to generate image data 108. The first encoding system 504 may include an encoding selection system 506 that can select an encoding scheme 508 from one or more encoding schemes represented by a first encoding scheme 508a, a second encoding scheme 508b, ..., an Nth encoding scheme 508n. The first encoding system 504 can use the encoding scheme 508 to convert image data 108 into encoded image data 514.

[0059] The coding selection system 506 can receive image data 108 and acquisition settings 502 (e.g., dose rate). The coding selection system 506 can dynamically calculate the coding scheme 508 to use to represent the image data 108. Each coding scheme 508 may be used to encode the image data 108 into different representations. The coding scheme 508 used may be based on the acquisition settings 502 and / or the image data 108, so the representation and coding scheme 508 may be said to be dynamic. The calculation of the dynamic representation to use to represent the image data 108 may determine which of the available coding schemes 508 to select for use in encoding the image data 108. In certain embodiments, the acquisition settings 502 and / or the image data 108 are used to calculate the optimal coding scheme 508 for encoding the image data 108.

[0060] The coding selection system 506 can determine a subsequent coding scheme 508 to be used after a previous coding scheme 508 has been used, based on the total exposure of the sample (e.g., sample 206) to electrons and / or a portion of the total exposure of the sample, when the acquisition setting 502 changes. The ability of the coding selection system 506 to dynamically select the coding scheme 508 to use based on the acquisition setting 502 and / or other factors enables the first coding system 504 to generate dynamically / adaptive coded image data 514.

[0061] In one example, a first dynamic representation used to represent a first image data (a subset of image data from image data 108) is calculated by the coding selection system 506 based on at least a first acquisition setting. The first image data may be converted to a first spatiotemporal resolution representation using a first coding scheme 508a. Furthermore, a second dynamic representation used to represent a second image data (a subset of image data from image data 108) is calculated by the coding selection system 506 to represent the second image data based on at least a second acquisition setting. The second image data may be received by the first coding system after the first image data and may be part of the same sample imaging session (e.g., a different portion of image data 108 output from the image acquisition system). The second image data may be different from the first image data. The second image data may be converted to a second spatiotemporal resolution representation using a second coding scheme 508b. The second dynamic representation may be different from the first dynamic representation. In this example, the first encoding system 506 can dynamically encode image data such that the first image data is encoded using a different encoding scheme 508 than the second image data. The second encoding scheme 508b may have more loss than the first encoding scheme 508a. The first spatiotemporal resolution representation / first encoded image data may contain a higher spatiotemporal resolution compared to the second spatiotemporal resolution representation / second encoded image data.

[0062] By using different encoding schemes 508, high-resolution sample information that may be included in the first image data can be encoded differently (for example, in a higher spatiotemporal resolution representation) from low-resolution sample information that may be included in the second image data. The different encoding schemes 508 can reduce the amount of memory occupied by the encoded image data 514 and / or increase the spatiotemporal resolution of the encoded image data 514 compared to when a single encoding scheme 508 is used for both the first and second image data. The different encoding schemes 508 can minimize the amount of memory occupied by the encoded image data 514 while maximizing the spatiotemporal resolution of the encoded image data 514. In certain embodiments, the first and second image data can pass through one or more first encoding systems 504 in parallel. The first encoding systems 504 can reduce the amount of redundant information and / or discarded information, including associated image data 108.

[0063] The coding selection system 506 can select a coding scheme 508 based on the number of electrons emitted, the number of electrons that interacted with a given region of the sample, and / or the duration of time. The coding schemes 508 included in one or more sets of coding schemes, and / or the factors considered by the coding selection system 506 for selecting a coding scheme 508, may be determined by instructions received from a graphical user interface of a user device (e.g., a laptop, tablet, desktop, image acquisition system, etc.).

[0064] After the coding selection system 506 has computed a dynamic representation to be used to represent the image data 108, the coding selection system 506 may encode the image data 108 using an encoding scheme 508 from the set of encoding schemes (e.g., a first encoding scheme 508a). The coding selection system 506 may transmit the image data 108 to the selected encoding scheme 508. Otherwise, the coding selection system 506 may cause the selected encoding scheme 508 to receive the image data 108.

[0065] The selected encoding scheme 508 can encode image data 108 into encoded image data 514. The encoding scheme 508 can represent image data 108 in a different / encoded representation. The encoding scheme 508 may or may not cause encoded image data 514 to occupy less memory space than image data 108. The encoding scheme 508 can represent image data 108 in a spatiotemporal resolution representation. Different encoding schemes 508 can represent image data 108 in a higher spatiotemporal resolution representation and / or occupy less memory compared to other encoding schemes 508. In certain embodiments, image data 108 can be obtained from encoded image data 514 using an encoding scheme that is the reverse of the one used to generate encoded image data 514 using image data 108. In other words, the reverse of the encoding scheme (which may also be called the decoding scheme) can be used to decode encoded image data.

[0066] An example of the encoding scheme 508 may include an Electron Event Representation (EER). The EER may encode an electron event as (X, Y, time). The event location may be stored on a spatiotemporal sampling grid that becomes progressively coarser. In certain embodiments, if the amount of electrons at each sampling grid coordinate becomes too high, it may be more efficient to start encoding the image data 108 using a different encoding scheme (e.g., a dose-resolved scheme).

[0067] Another example of encoding scheme 508 is a dose-splitting scheme. A dose-splitting scheme may include a fixed and / or moving spatiotemporal resolution. A fixed spatiotemporal resolution can keep the spatiotemporal resolution constant for a selected dose-splitting scheme. A moving spatiotemporal resolution may allow adjustment of the spatiotemporal resolution based on the image size, the amount of camera frames per dose fraction, and / or other factors (e.g., acquisition settings 502). A dose-splitting scheme can compensate for local and global motion by dividing ("fractionating") a desired total exposure dose of a specific amount of electrons per pixel into partial exposures over time. For example, if 20 dose fractions are created using exposures of 20 electrons per pixel, it is possible to produce 20 images, each with 1 electron per pixel. The divisions can be selected independently of the camera's native frame rate, so 20 electrons per pixel exposure may result in 1000 native frames. Therefore, in this case, each dose fraction consists of 1000 / 20 frames = 50 frames.

[0068] Another example of the encoding scheme 508 is an image coding-decoding model. The image coding-decoding model may be trained to define encoded image data 514. The image coding-decoding model may be trained using a generative adversarial network (GAN). Embodiments of the image coding-decoding model are described in more detail below (for example, with respect to Figures 7 and 8, among other areas).

[0069] In certain embodiments, the set of coding schemes may include two or more of the types of coding schemes 508. For example, the set of coding schemes may include a first coding scheme 508a, which is a first dose-division scheme configured in a first way, and a second coding scheme 508b, which is a second dose-division scheme configured in a second way.

[0070] The encoded image data 514 may be the encoded image data 112 described in relation to system 100. The encoded image data 514 may represent a low-dose image frame or an electronic event. In a particular embodiment, the first encoded image data generated using the first image data represents an electronic event, and the second encoded image data generated using the second image data represents a low-dose image frame. In a particular embodiment, the encoded image data 514 is re-encoded by the compression system 516 to generate compressed image data 518. The compressed image data 518 may be the encoded image data 112 described in relation to system 100.

[0071] The compression system 516 can compress encoded image data 514 (for example, image data represented in spatiotemporal resolution representation) to generate compressed image data 518. In certain embodiments, the compression system 516 may generate compressed image data 518 based on a dynamic representation / encoding scheme 508 used with image data 108. Since the compression system 516 can generate compressed image data 518 based on a dynamic representation / encoding scheme 508 used with image data 108, the compression system 516 can dynamically generate a first compressed image data based on a first encoding scheme used, and dynamically generate a second compressed image data based on a second encoding scheme used. In other words, a first portion of image data 602 may be compressed using a different compression scheme than that used for a second portion of image data 602.

[0072] The compression system 516 can compress the encoded image data 514 into a compressed representation. In some embodiments, the compression system 516 can compress the encoded image data 514 using a compression scheme such as Lempel-Ziv-Welch (LZW) or a ZIP file. The compressed representation may be more or less irreversible depending on the encoding scheme 508 used with the image data 108 to generate the encoded image data 514. A second encoding system will be described in more detail with reference to Figure 6.

[0073] Figure 6 is a schematic diagram showing examples of the compression system 516 according to several embodiments. The compression system 516 may be used to generate compressed image data 634. Compressed image data 634 may be the same compressed image data as compressed image data 518. Compressed image data 634 may be a compressed format of image data 602. Image data 602 may be represented in a spatiotemporal representation and received by the compression system 516. Image data 602 may be encoded to spatiotemporal resolution (e.g., encoded image data 514). The compression system 516 may generate compressed image data 634 using image data 602. The compression system 516 may be used to generate data compression tailored to an application. The compression system 516 may include a dataset description generation system 604, a reference image generation system 620, and an information-based compression subsystem 632.

[0074] The compression system 516 may receive image data 602 from a first encoding system (e.g., a first encoding system 504), an image acquisition system (e.g., an image acquisition system 106), and / or another image data source. The image data 602 may include high-contrast image data 602 (e.g., image data acquired using TEM with phase plate or dark-field imaging techniques). The image data 602 may include an image stack. The image stack may include data representing multiple images.

[0075] The dataset description generation system 604 may generate a dataset descriptor 618 using image data 602. The dataset description generation system 604 may generate a dataset descriptor 618 using a drift correction system 606, a contrast transfer function (CTF) estimation and correction system 608, a particle picking system and / or a feature segmentation system 610, a particle classification system 612, a three-dimensional (3D) reconstruction system 614, an artificial intelligence model (not shown), and / or downsampled image data (not shown). In certain embodiments, the dataset descriptor 618 may include tomography tilt series alignment (e.g., generated by a tomography tilt series alignment method), downsampled images (e.g., generated by a downsampled image generator), and / or encoded image data (e.g., generated by an image generation model 704).

[0076] The drift correction system 606 can generate drift profiles for image data 602 and / or each image stack. The drift correction system 606 can measure the drift of image stacks (e.g., each image stack is independent of other image stacks). The drift may be field-of-view shift as a function of time. The drift correction system 606 may be used to estimate local sample deformation (e.g., sample movement induced by an electron beam).

[0077] The contrast transfer function (CTF) estimation and correction system 608 can determine how much phase shift and microscopic aberrations have damaged the image data 602. The parameters of the function can be determined by the CTF estimation and correction system 608. The CTF estimation and correction system 608 can process the given image data 602 with the estimated phase shift and microscopic aberrations so that their effects are reduced or eliminated.

[0078] A particle picking system can find the location in image data 602 where a particle of interest is likely to exist. Since the particle is actually a 3D object, but image data 602 is 2D, the particle picking system searches for a 2D projection of the 3D particle of interest. Various algorithms can be used by the particle picking system, such as general methods aimed at finding "blob-like" structures within a given size range. Template matching methods may also be used by the particle picking system, where a previously acquired 3D model is projected onto multiple 2D template images with different projection orientations, and these projected 2D template images are used as templates to be detected in image data 602. Artificial intelligence algorithms can also be used by the particle picking system.

[0079] A feature segmentation system can be used to detect features that are of interest to describe image data 602 but are not of interest for 3D particle reconstruction (e.g., contamination).

[0080] The particle classification system 612 groups the picked particles into classes of particles with identical appearances. Particles with identical appearances are likely to have the same projection orientation. By joining particles with the same projection orientation, a denoised particle projection image can be obtained, which can then be used for reconstruction (e.g., initial reconstruction).

[0081] The 3D reconstruction system 614 can estimate a 3D model of the image data 602 and / or each image stack. The 3D reconstruction system 614 can infer the relative angles between different particles or between particle classes. If the angles are known, tomographic reconstruction can be performed to obtain a 3D model.

[0082] An artificial intelligence model may be used to infer / generate a dataset descriptor 618 based on an input that includes one or more image stacks and / or other information obtained using one or more image stacks. The artificial intelligence model may be trained to generate a dataset descriptor 618 that yields the best compression. In certain embodiments, the artificial intelligence model may be an image generation model (e.g., image generation model 704).

[0083] Downsampled image data may be generated for each image stack. The summed and downsampled image stack may be generated as represented by the dataset descriptor 618. The reference image generation system 620 may then upsample the downsampled image stack to generate the reference stack 630.

[0084] The drift correction system 606, the CTF estimation and correction system 608, the particle picking system and / or feature segmentation system 610, the particle classification system 612, the three-dimensional (3D) reconstruction system 614, the artificial intelligence model, downsampled image data, and / or denoised image data may be used by the attribute extraction system 616 to generate a dataset descriptor 618. The attribute extraction system 616 can extract data relating to image data 602 from the data generated by the drift correction system 606, the CTF estimation and correction system 608, the particle picking system and / or feature segmentation system 610, the particle classification system 612, the three-dimensional (3D) reconstruction system 614, the artificial intelligence model, and / or downsampled image data. The extracted data may be included in the dataset descriptor 618.

[0085] In the example of attribute extraction system 616, the attribute extraction system 616 collects all compact description results from all processing steps (e.g., drift correction system 606, CTF estimation and correction system 608, etc.) and concatenates them into a single dataset descriptor 618. The compact description results may include a minimal description of what is generated by the drift correction system 606, CTF estimation and correction system 608, etc. For example, the compact description results may include particle picking, i.e., x, y coordinates and image numbers that do not include cropped and boxed particle images.

[0086] In one example, the dataset descriptor 618 includes particle coordinates and projection direction, defocus, drift profile, and / or location of stains on the sample (e.g., sample 206). The dataset descriptor 618 may be generated for image data 602 and / or each image stack contained in image data 602. The dataset descriptor 618 may be used to generate a probabilistic model for each spatiotemporal location within each image stack of image data 602. In other words, the dataset descriptor 618 may represent the most likely particle location and other sample information based on the information contained in the image stacks of image data 602. The dataset descriptor 618 may describe the dose used to acquire image data 602. The actual values ​​may be encoded using the information-based compression subsystem 632. In certain embodiments, the more accurate the dataset descriptor 618, the better / greater the expected compression of image data 602, which is then compressed and represented as compressed image data 634.

[0087] The reference image generation system 620 may generate a reference stack 630 using the dataset descriptor 618. The reference stack 630 may correspond to an image stack contained in the image data 602. Each reference stack 630 may represent the expected number of electrons for one or more points in spacetime. Each reference stack 630 may represent a 3D image. The reference image generation system 620 may perform the reverse operation compared to the operation performed by the dataset description generation system 604. The reference image generation system 620 may use the dataset descriptor 618 to regenerate an artificial micrograph that forms an ideal, noise-free estimate of the image stack contained in the image data 602.

[0088] The reference image generation system 620 includes a projection system 622, a particle placement system 624, a CTF application system 626, and a stack creation system 628, but different combinations of systems may be used (similar to how different combinations of systems can be included in the dataset description generation system 604).

[0089] The information-based compression subsystem 632 may receive a reference stack 630 generated by the reference image generation system 620, an image stack contained in the image data 602, and a dataset descriptor 618. The information-based compression subsystem 632 may include an entropy encoder. For example, the entropy encoder may be an arithmetic encoder, a Huffman encoder, a range variant of an asymmetric numeral system (rANS) encoder, or another encoder. The information-based compression subsystem 632 may generate compressed image data 634 representing the image data 602. The information-based compression subsystem 632 may generate compressed image data 634 using a Poisson distribution. The dataset descriptor 618, in combination with the running reference image generation system 620, can provide a Poisson distribution that gives the probability that N electrons land in each pixel (for example, by giving the expected number of electrons in each pixel).

[0090] The information-based compression subsystem 632 can use the reference stack 630 as prior knowledge to improve the compression ratio and generate compressed image data 634. Instead of encoding real data without prior knowledge, the information-based compression subsystem 632 may encode the difference between ideal data (represented, for example, by the reference stack 630) and real data (represented by the corresponding stack contained in the image data 602). The information-based compression subsystem 632 may use the pixel intensity of the reference stack 630 as the lambda parameter of a Poisson distribution that gives the probability that N electrons land on each pixel.

[0091] The information-based compression subsystem 632 may also output a dataset descriptor 618 and / or other information generated and / or used by the compression system 516 to generate the compressed image data 634. For example, the information-based compression subsystem 632 may output a dataset descriptor 618, which may later be used to decompress the compressed image data 634.

[0092] The compression system 516 may enable additional compression compared to other compression techniques. The compression system 516 may be suitable for compressing image data 602 obtained by image data acquisition methods that provide images with large contrast differences, such as TEM imaging with phase plates, dark-field imaging techniques, and STEM iDPC. In conventional TEM imaging, the average number of electrons per pixel is large compared to the expected spread of the number of electrons per pixel, resulting in a limiting gain of the compression factor when encoding electron events or small dose fractions. The compression system 516 may reduce the memory space / resources used to represent the image data 602 and the network resources used to transmit the information representing the image data 602. The compression system 516 may be reversible in that it preserves the information necessary to obtain the highest possible reconstruction resolution.

[0093] Figure 7 is a schematic diagram showing an example of a generative adversarial network (GAN) 700 for training an image generation model 704, according to several embodiments. The image generation model 704 may be an example of an image coding-decoding model. The image coding-decoding model can be trained to define / generate coded image data 706 based on a latent space 702 representing received image data or image data (e.g., actual coded image data 708). GAN 700 is a deep learning architecture. GAN 700 trains two neural networks, an image generation model 704 and a discriminative model 710, to compete against each other.

[0094] The image generation model 704 may be used to generate encoded image data 706 based on a latent space 702. In certain embodiments, the generated encoded image data 706 can be generated from the latent space 702, which may represent actual encoded image data 708. In certain embodiments, the generated encoded image data 706 can be generated using a fixed-length random vector from the latent space 702 representing image data as input. The fixed-length random vector may represent image data. The fixed-length random vector may contain image data that has been randomly noised based on an arbitrary noise distribution. The noise distribution may be sampled to obtain a batch of noise samples (noise vectors) to be used as input image data to the image generation model 704.

[0095] The generated encoded image data 706 can be represented in a spatiotemporal resolution representation. The image data can be input to an image generation model 704. The image generation model 704 may be capable of integrating damaged electronic counting frames into a single denoising image. The image generation model 704 can generate encoded image data 706 with reduced data compared to the input image data received by the image generation model 704, and can do so without sacrificing the spatiotemporal resolution of the image data represented by latent space.

[0096] The image generation model 704 and the discrimination model 710 are trained in an adversarial game, the image generation model attempts to generate encoded image data 706, and the discrimination model 710 attempts to predict whether the encoded image data is false / generated encoded image data 706 or real encoded image data 708. The real encoded image data 708 may include image data 108. The real encoded image data 708 may be image data generated by an image acquisition system (e.g., image acquisition system 106).

[0097] The identification model 710 analyzes the actual encoded image data 708 and determines its attributes independently of the image generation model 704. The output of the ground truth and the identification model 710 may be used by the loss comparison system 712 to determine how to adjust the weights of the image generation model 704 and / or the identification model 710. The loss comparison system 712 may send a first weight adjustment signal 714 to the identification model 710 to adjust its weights in order to reduce the error of the identification model 710. The loss comparison system 712 may send a second weight adjustment signal 716 to the image generation model 704 to adjust its weights in order to reduce the error of the image generation model 704.

[0098] After the image generation model 704 modifies several data attributes in the latent space 702 by adding noise (or random changes) to specific attributes, the image generation model 704 can pass the generated encoded image data 706 to the discriminator model 710. The discriminator model 710 calculates the probability that the generated encoded image data 706 belongs to the original dataset of real encoded image data 708. The discriminator model 710 provides guidance to the image generation model 704 by performing weight adjustments on its parameters using a second weight adjustment signal 716 to reduce the randomization of the noise vector in the next cycle. The image generation model 704 attempts to maximize the probability of error by the discriminator model 710, but the discriminator model 710 attempts to minimize the probability of error by using a loss comparison system 712 that sends a first weight adjustment signal 714 to the discriminator model 710 to update the weights used by the discriminator model 710. During training iterations, both the image generation model 704 and the discrimination model 710 have modified weights based on weight adjustment signals transmitted by the loss comparison system 712, and continuously evolve and are compared with each other. Training iterations may continue until the image generation model 704 and the discrimination model 710 reach an equilibrium state. In the equilibrium state, the discrimination model 710 may no longer recognize synthetic data. At this point, the training process may be completed.

[0099] In certain embodiments, GAN700 is a basic GAN architecture in which the image generation model 704 generates data fluctuations with little or no feedback from the discriminator model 710. In certain embodiments, GAN700 is a conditional GAN ​​architecture in which the image generation model 704 and the discriminator model 710 receive additional information such as class labels or some other form of conditioned data. Those skilled in the art who are interested in this disclosure will be aware of other GAN architectures that can be used to train the image generation model 704 (e.g., deep convolutional GANs, super-resolution GANs, etc.).

[0100] In certain embodiments, the image generation model 704 may be trained using other training techniques, or it may not be trained using a GAN. For example, the image generation model 704 may be a diffusion model, a stable diffusion model, a variational encoder, and / or an autoregressive model. In certain embodiments, the image generation model 704 is trained to generate encoded image data 706 using at least a portion of the acquisition settings (e.g., acquisition settings 502).

[0101] The image generation model 704 can be trained to receive image data and / or embed the image data in latent space 702 and output the generated encoded image data 706. The image generation model 704 can be trained to partially or completely omit image data frames obtained from electrons interacting with the sample that are obtained later than other image data frames obtained from electrons interacting with the sample, thereby reducing the volume of image data in the generated image data 708 compared to the volume of image data. The image generation model 704 can use the image data to generate encoded image data 708 that occupies less memory space than the image data without compromising the spatiotemporal resolution of the generated encoded image data 708.

[0102] Figure 8 is a schematic diagram showing exemplary architectures of an image generation model 704 according to several embodiments. The image generation model 704 may be trained according to the techniques described with respect to Figure 7. The image generation model 704 may be trained to output encoded image data 804 based on received image data 802.

[0103] Image data 802 may include image data 108. Image data 802 may be received from an encoding selection system (e.g., encoding selection system 506) or an image acquisition system (e.g., image acquisition system 106). In certain embodiments, an image generation model 704 receives and uses one or more acquisition settings (e.g., acquisition settings 502) to generate encoded image data 804. In certain embodiments, encoded image data 804 is a compressed representation of image data 802.

[0104] The image generation model 704 may include any combination of layers. For example, the image generation model 704 may include an input layer 806, an encoder block 808, a time context module 810, a decoder block 812, a quantization layer 814, and / or an output layer 816. In a particular embodiment, the number of each layer included in the architecture of the image generation model 704 may be more or less (e.g., zero or more).

[0105] The input layer 806 can accept image data 802. Image data 802 can be represented as one or more image stacks (e.g., raw 3D slices from a cryoEM stack image acquisition system). The input layer 806 can apply a Fourier transform to separate the frequency components.

[0106] Encoder block 808 may include a 3D convolutional layer. Encoder block 808 may include a Swish activation function and spatial dropout. The Swish activation function and spatial dropout may be performed after the 3D convolutional layer processing. Encoder block 808 can encode the input layer data into a low-dimensional latent space.

[0107] The time context module 810 can use one or more recurrent neural networks (RNNs) to capture temporal correlations between consecutive frames and enhance the ability of the image generation model 704 to distinguish between noise and structured data.

[0108] Decoder block 812 may utilize 3D transposed convolution, Swish activation functions, and / or batch normalization. Decoder block 812 may reconstruct the denoised image from the latent space.

[0109] The quantization layer 814 can quantize the denoised image to further reduce the file size (e.g., occupied memory space) without sacrificing important structural details of the image data 802.

[0110] The output layer can generate encoded image data 804. Encoded image data 804 may represent high-quality and / or denoised image data that retains the essential structural information of image data 802. Encoded image data 804 may be part of an encoded image stack (e.g., an encoded 3D image slice).

[0111] The architecture of the image generation model 704 and / or the training techniques used to train the image generation model 704 can result in significant data reduction compared to other methods of representing image data generated by an image acquisition system. The image generation model 704 can compress large sets of image data, such as reducing a 2TB dataset to 200GB, achieving a 90% data reduction. The image generation model 704 can preserve the quality of image data 802 in encoded image data 804, and can do so despite the size reduction it can achieve. Preserving the quality of image data 802 in encoded image data 804 means that important structural information contained in image data 802 can be preserved, and data quality can not be degraded.

[0112] Using the image generation model 704 can increase the speed and efficiency of generating encoded (e.g., compressed) image data 802 compared to other techniques for encoding image data 804 acquired from an image acquisition system. Computational, network, and / or memory resources can be reduced by the image generation model 704. For example, network and / or memory resources can be reduced by the generated compressed encoded image data 804. In another example, the computational resources required to generate encoded image data 804 from image data 802 may be less than other methods for encoding image data 802 to achieve similar encoded image data. The image generation model 704 can be hosted on a local server and / or a remote server (e.g., in the cloud).

[0113] Processes 900, 1000, 1100, 1200, 1300, and 1400, shown in the flowcharts and any other figures, may be implemented using hardware in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, or a combination thereof. The software may be stored in a non-temporary storage medium (e.g., a memory device). Processes 900, 1000, 1100, 1200, 1300, and 1400, as well as the methods presented in the other figures and described herein, are intended to be illustrative and non-limiting. Processes 900, 1000, 1100, 1200, 1300, and 1400, as well as the other figures, illustrate various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in some different order, or some steps may be performed in parallel. In alternative embodiments, please understand that processes 900, 1000, 1100, 1200, 1300, and 1400, as well as other figures, may include more or fewer steps than those shown in their respective figures.

[0114] Figure 9 is a flowchart of an exemplary process 900 for using a generative adversarial network (GAN) (e.g., GAN700) together with an image generation model and a discriminative model to train an image generation model (e.g., image generation model 704) according to several embodiments. The image generation model and the discriminative model can be trained using a training dataset that includes multiple batches of training examples.

[0115] In 902, a latent space, a first parameter set, and a first weight set may be initialized. The first parameter set and the first weight set may be for an image generation model and may influence the output produced by the image generation model. Each weight may be a value assigned to the corresponding parameter. In one example, the image generation model may include a neural network. In one example, the image generation model may be an image coding-decoding model. In one example, the image generation model may receive a fixed-length random vector representing image data as input and generate coded image data. The fixed-length random vector may represent image data. The fixed-length random vector may contain image data that has been randomly noised based on an arbitrary noise distribution. The noise distribution may be sampled to obtain a batch of noise samples (noise vectors) to be used as input image data to the image generation model. The image generation model may be trained to generate coded image data using the received image data. The coded image data may be compressed image data. The coded image data may retain the quality of the image data within the coded image data and may retain important structural information contained in the image data.

[0116] In 904, a second parameter set and a second weight set may be initialized. The second parameter set and the second weight set may be for a discriminative model and may influence the output produced by the discriminative model. Each weight may be a value assigned to the corresponding parameter. The discriminative model may be a neural network. Each training example used by the discriminative model may include real encoded image data.

[0117] In 906, a fixed-length random vector can be input to an image generation model. The image generation model can then generate encoded image data based on the fixed-length random vector. The image generation model can output the generated encoded image data.

[0118] In step 908, the generated encoded image data may be sent to a discriminative model. One or more encoded image data may be sent to the discriminative model in the training phase (e.g., batch size) after being generated. After one or more encoded image data have been sent, each encoded image data may be input into the discriminative model.

[0119] In 910, the identification model may be configured to generate classifications of real-encoded image data or generated-encoded image data. The identification model may generate two or more classifications for two or more real-encoded image data and / or generated-encoded image data (for example, depending on the batch size). The classifications may indicate whether the identification model classified the received-encoded image (real or generated) as real-encoded image data or as generated-encoded image data.

[0120] In 912, the classification determined by the identification model can be transmitted to the loss comparison system.

[0121] In 914, the loss comparison system may compare the classification output by the discriminative model with ground truth data to determine whether the classification was correct. The loss comparison system may use any appropriate loss function. In one embodiment, the image generation model and the discriminative model are trained adversarially using a loss function (e.g., Mean Square Error (MSE)).

[0122] Adversarial training means sequentially and iteratively training a discriminative model and an image generation model. The discriminative model can be trained with both generated encoded image data received from the image generation model and real encoded image data from training examples. As the discriminative model improves to recognize the generated encoded image data as such (i.e., as a "fake" image), the image generation model loss may increase. The image generation model is then trained to reduce this loss.

[0123] In 916, the loss comparison system may send a weight adjustment signal to the image generation model or the identification model based on the classification and ground truth output by the identification model.

[0124] Image generation and discrimination models can be trained by repeatedly updating a first weight set corresponding to a first parameter set (e.g., by using backpropagation) and / or updating a second weight set corresponding to a second parameter set. In certain embodiments, one or more weights of the image generation model are adjusted if the discrimination model correctly classifies the generated encoded image as a generated encoded image (e.g., not a real encoded image).

[0125] Image generation and discrimination models can be trained by repeatedly updating the parameter weights for the image generation and discrimination models until the loss function for the discrimination output is maximized for the discrimination model and minimized for the image generation model. Those skilled in the art who are interested in this disclosure will recognize that other techniques may be used to train machine learning models to generate encoded image data.

[0126] Steps 906-916 may be repeated depending on how long training continues and / or the loss function used by the loss comparison system. Once the image generation model is trained, it can be used to generate encoded image data using the image data.

[0127] Figure 10 is a flowchart of an exemplary process 1000 for generating encoded image data according to several embodiments. The encoded image data may be generated using an image generation model (e.g., image generation model 704) during inference time.

[0128] In 1002, an image acquisition system (e.g., image acquisition system 106) may be operated to image a sample (e.g., a radiosensitive sample). The image acquisition system may generate sample data based on data collected over a period of time. The image acquisition system may be configured to operate according to one or more acquisition settings. The acquisition settings used to operate may be based on pre-configured acquisition settings, a sample, and / or user input. As an example, the acquisition settings may include a first dose rate and / or a first operating mode. The sample data collected during the period may be image data and / or may be used to generate image data.

[0129] In step 1004, image data is generated based on sample data collected during the period. In certain embodiments, such as when the sample data is the same as the image data, this step may be omitted. The image data may include one or more electronic counting frames.

[0130] In 1006, image data can be encoded. Image data can be encoded using an encoding scheme. In certain embodiments, the encoding scheme can encode image data using a machine learning model (e.g., an image encoding-decoding model). The machine learning model can compress the image data. In certain embodiments, the machine learning model may be trained to encode image data (e.g., trained using a GAN training architecture). By encoding image data, the network, memory, and / or processing resources used by the system on and / or with the image data can be reduced. For example, the image data before encoding may have an image resolution represented using a first number of bits (e.g., a first amount of space in memory), and the encoded image data may have the same image resolution but use a second number of bits which is less than the first number of bits. In other words, the encoded image may occupy less space in memory than the image data, thereby reducing the network, memory, and / or processing resources used by the system on and / or with the encoded image data compared to the image data before encoding.

[0131] In a particular embodiment, the encoding scheme used in step 1006 is one of several encoding schemes available for encoding the image data. In a particular embodiment, the encoding scheme used in step 1006 is one of several encoding schemes used for encoding each portion of the image data. For example, the encoding scheme performed during step 1006 may be performed on a first portion of the image data, and a different encoding scheme may be performed on a second different portion of the image data.

[0132] Figure 11 is a flowchart of an exemplary process 1100 for acquiring image data according to several embodiments. In certain embodiments, process 1100 may be performed by an image acquisition system (e.g., image acquisition system 106). As described above, the image acquisition system may include an electron microscope. The image acquisition system may be used to acquire sample data and / or image data of an imaged sample (e.g., a radiosensitive sample).

[0133] In 1102, the image acquisition system may be configured according to a first acquisition setting. In certain embodiments, the first acquisition setting may include a first dose rate and / or a first operating mode (e.g., counting mode, integration mode). The first acquisition setting, which works in conjunction with other settings, may be based on an image acquisition setting received from another system, device, user interface, etc. The first acquisition setting, which works in conjunction with other settings, may be based on a predetermined acquisition setting that may have been previously configured. In certain embodiments, the acquisition setting may additionally or alternatively include duration and / or magnification.

[0134] In 1104, the acquisition system can operate according to a first acquisition setting during a first period. The first period may be a portion of the total period over which the sample is imaged by the image acquisition system. The length of the first period may be determined based on user input, the sample to be imaged (expected and / or actual radiation damage to the sample), the operating mode, the dose rate used, the dose rate schedule, and / or the total time the sample is imaged, user input, the purpose of imaged the sample, a time-dependent function (e.g., exponential, linear), etc.

[0135] Sample data obtained from a sample by an image acquisition system may have different characteristics during the first period compared to a different second period, which is a portion of the total period over which the sample is imaged. For example, if the sample is radiosensitive, the sample may degrade as the dose accumulates (e.g., as more electrons interact with the sample). Due to radiation damage, electrons detected at the end of the acquisition period may retain less information about the sample than those detected at the beginning of the acquisition period. Sample data acquired by operating during the first period can be used to generate image data.

[0136] In 1106, the image acquisition system may be configured according to a second acquisition setting. The image acquisition system may be configured according to a second acquisition setting after a first period. The second acquisition setting may differ from the first acquisition setting. In certain embodiments, the second acquisition setting may include a higher dose rate, a different operating mode, and / or other setting differences compared to the first acquisition setting. The second acquisition setting may be determined according to a time-dependent function, the purpose of imaging the sample, and / or the sample (e.g., the expected radiation damage characteristics of the sample).

[0137] In 1108, the acquisition system may operate according to a second acquisition setting during a second period. The second period may be a portion of the total period during which the sample is imaged by the image acquisition system. The length of the second period may be determined based on user input, the sample to be imaged, the operating mode, the dose rate used, the dose rate schedule, and / or the total time the sample is imaged. The second sample data obtained from the sample by the image acquisition system may have different characteristics from the sample data obtained from the sample by the image acquisition system during the first period.

[0138] In 1110, the first and second sample data acquired by the image acquisition system during the first and second periods, respectively, can be combined to generate image data. For simplicity, the illustrated process 1100 includes the first and second periods. However, in certain embodiments, three or more periods are included in the total period. In certain embodiments, three or more acquisition settings are used over the total period. In certain embodiments, the image data is then encoded (e.g., into a spatiotemporal representation, into a compressed representation).

[0139] As described above, if a sample is radiosensitive, it may degrade as the dose accumulates (e.g., as more electrons interact with the sample). Due to radiation damage, electrons detected at the end of the acquisition period may retain less information about the sample than those detected at the beginning of the acquisition period. Furthermore, events detected in high-dose-rate frames may retain less information about the imaged sample than events detected in low-dose-rate frames. In addition, image data acquired from the EMS200 may be more accurate at lower dose rates than at higher dose rates, and the image quality (e.g., precision) of the image data may degrade as the dose rate increases. With the above in mind, certain embodiments of this specification (e.g., adjusting the acquisition settings over the period over which sample data is acquired) can increase the amount of information obtained from the sample over a given period. Certain embodiments can maximize the amount of information obtained from the sample over a given period.

[0140] Figure 12 is a flowchart of an exemplary process 1200 for encoding image data according to several embodiments. Process 1200 may be performed by an image processing system (e.g., image processing system 110). Process 1200 may be performed by a first encoding system (e.g., first encoding system 504). Process 1200 may be performed on image data generated by an image acquisition system (e.g., image acquisition system 106).

[0141] In 1202, image data may be received. The image data may be generated using an image acquisition system. The image data may be generated based on the image acquisition system using acquisition settings. The image data may represent image data obtained during the total period over which the sample was imaged by the image acquisition system. The image data may represent image data obtained during a portion of the total period over which the sample was imaged by the image acquisition system. The image data may represent image frames or electronic events.

[0142] In step 1204, a dynamic representation of the image data is calculated. The dynamic representation used to represent the image data may be calculated based on at least the acquisition settings used to obtain the image data. The dynamic representation may be an encoding scheme from a set of encoding schemes dynamically selected based on at least the acquisition settings used to obtain the image data. In other words, the image data may be encoded using an encoding scheme based on the acquisition settings used to obtain the image data. Step 1204 may be performed by an encoding selection system (e.g., encoding selection system 506).

[0143] In step 1206, the image data may be converted to a spatiotemporal resolution representation using a dynamic representation computed during step 1204. The image data may be converted to spatiotemporal resolution using an EER coding scheme, a dose-division coding scheme, an image coding-decoding model, or another coding scheme capable of coding the image data. The spatiotemporal resolution representation is sometimes referred to as coded image data.

[0144] Using different encoding schemes can reduce the amount of memory occupied by encoded image data and / or increase the spatiotemporal resolution of encoded image data compared to using different encoding schemes. By using an encoding scheme based on acquisition settings, it is possible to minimize the amount of memory occupied by encoded image data and, at the same time, maximize the spatiotemporal resolution of the encoded image data.

[0145] In certain embodiments, process 1200 may be used for two distinct parts of image data. For example, process 1200 may be used for a first part of image data to encode the first part using a first encoding scheme. The first part of image data may represent a low-dose image frame or an electronic event. Furthermore, process 1200 may be used for a second part of image data that is different from the first part of image data, and the second encoding scheme may be different from the first encoding scheme. The second part of image data may represent a low-dose image frame or an electronic event. In certain embodiments, the first part of image data may have a different (e.g., higher) resolution than the second part of image data (e.g., due to the acquisition settings used to generate the image data, such as the accumulated dose to the sample). The second encoding scheme can generate an encoded image independently of the first encoding scheme. In certain embodiments, the second encoding scheme may generate the second encoded part using an encoding scheme that is more lossy than the first encoding scheme.

[0146] Figure 13 is a flowchart of an exemplary process 1300 for compressing image data according to several embodiments. Compression can result in a reduction of the amount of memory space used to represent the image data. Compression can be reversible in that it preserves all the information necessary to obtain the highest possible reconstruction resolution.

[0147] In 1302, image data may be received. The image data may include an image stack. The image stack may include data representing a set of images. The image data may be received from an image acquisition system (e.g., image acquisition system 106), an encoding system (e.g., a first encoding system 504), and / or memory, etc.

[0148] In 1304, the image data may be encoded. In certain embodiments, the image data may be encoded to spatiotemporal resolution using an image coding-decoding model, or another coding scheme that can encode the image data, using an EER coding scheme, a dose-division coding scheme. In certain embodiments, the image data may be encoded based on the acquisition settings used to acquire the image data.

[0149] In 1306, in certain embodiments, a dataset descriptor of encoded image data may be generated. The dataset descriptor may be generated using drift correction, contrast transfer function correction, particle picking, feature segmentation, particle classification, 3D reconstruction, artificial intelligence models, downsampled image data, and / or denoised image data.

[0150] Dataset descriptors can be generated for image data and / or each image stack contained within the image data. Dataset descriptors can be used to generate probabilistic models for each spatiotemporal location within each image stack of the image data. In other words, a dataset descriptor can represent the most likely particle location and other sample information based on the information contained in the image stacks of the image data. In certain embodiments, the more accurate the dataset descriptor, the better / more effective the expected compression will be for the image data that is compressed and represented as a second encoded image data.

[0151] In step 1308, in certain embodiments, compressed image data may be generated. Compressed image data may be generated using the dataset descriptor and entropy encoder generated in step 1306. Compressed image data may include a compressed image stack representing a set of one or more images. Generating compressed image data using a dataset descriptor may include generating a three-dimensional reference image using an encoded dataset descriptor and / or encoded image. The three-dimensional reference image may include the expected number of electrons for one or more points in spacetime. In certain embodiments, the entropy encoder may be a range variation of an arithmetic encoder, a Huffman encoder, or an asymmetric numeral (rANS) encoder.

[0152] Figure 14 is a flowchart of an exemplary process 1400 for acquiring and encoding image data, according to several embodiments.

[0153] Steps 1402, 1404, 1406, 1408, and 1410 can be performed in the same manner as steps 1102, 1104, 1106, 1108, and 1110 described above. In certain embodiments, steps 1406 and 1408 are not performed, and only a single acquisition setting set is used to generate image data.

[0154] In step 1412, the image data generated in step 1410 can be encoded. The image data can be encoded using any encoding scheme described herein. For example, the image data can be encoded using the EER encoding scheme, the dose-division encoding scheme, the image encoding-decoding model, or the image generation model (e.g., image generation model 704). As described above, in a particular embodiment, different parts of the image data can be encoded using an encoding scheme based on the acquisition settings used to obtain the sample data.

[0155] In 1414, encoded image data may be further encoded by compressing the encoded image data. Exemplary compression schemes may include using a compression system (e.g., compression system 516), an arithmetic encoder, a Huffman encoder, or a range variation of an asymmetric numeral system (rANS) encoder, or another encoder.

[0156] In step 1416, the image data generated in step 1410 may be encoded in a way that compresses it. An exemplary compression scheme may include using an image generation model (e.g., image generation model 704) and / or a compression system (e.g., compression system 516).

[0157] Figure 15 shows a schematic diagram of an exemplary computer system that can be used in systems and methods according to some embodiments of the present disclosure.

[0158] Any of the computer systems referred to herein (e.g., computing system 104, image acquisition system 106, etc.) may utilize any number of appropriate subsystems. An example of such subsystems is shown in computer system 1510 in Figure 15. In some embodiments, the computer system includes a single computer device, and subsystems may be components of the computer device. In other embodiments, the computer system may include multiple computer devices, each having internal components, and each being a subsystem. The computer system may include desktop and laptop computers, tablets, mobile phones, and other mobile devices.

[0159] The subsystems shown in Figure 15 are interconnected via a system bus 1575. Additional subsystems are shown, including a printer 1574, a keyboard 1578, a storage device 1579, and a monitor 1576 (e.g., a display screen such as an LED) coupled to a display adapter 1582. Peripherals and I / O devices coupled to the input / output (I / O) controller 1571 can be connected to the computer system by any number of means known in the art, such as input / output (I / O) ports 1577 (e.g., USB, FireWire®). For example, the computer system 1510 can be connected to a wide area network such as the Internet, a mouse input device, or a scanner using I / O ports 1577 or an external interface 1581 (e.g., Ethernet, Wi-Fi, etc.). The interconnection via the system bus 1575 allows the central processor 1573 to communicate with each subsystem and control the execution of multiple instructions from system memory 1572 or storage device 1579 (e.g., a hard drive or a fixed disk such as an optical disk), as well as the exchange of information between subsystems. The system memory 1572 and / or storage device 1579 can embody a computer-readable medium. Another subsystem is a data acquisition device 1585 such as a camera, microphone, or accelerometer. Any of the data referred to herein can be output from one component to another and to the user.

[0160] A computer system may include multiple identical components or subsystems that are connected together, for example, by an external interface 1581, by an internal interface, or via removable storage devices that are connected from one component to another and can be removed. In some embodiments, computer systems, subsystems, or devices may communicate over a network. In such examples, one computer may be considered a client and another a server, and each may be part of the same computer system. Each client and server may include multiple systems, subsystems, or components.

[0161] Aspects of the embodiments can be implemented in the form of control logic using hardware circuits (e.g., application-specific integrated circuits or field-programmable gate arrays) and / or computer software stored in memory having a generally programmable processor in a modular or integrated manner. Thus, the processor may include memory for storing software instructions constituting the hardware circuits, and an FPGA or ASIC having configuration instructions. As used herein, the processor may include a single-core processor, a multi-core processor on the same integrated chip, or multiple processing units on a single circuit board or networked, and dedicated hardware. Based on the disclosures and teachings provided herein, those skilled in the art will know and understand other ways and / or methods for implementing embodiments of the embodiments of the present disclosure using hardware and combinations of hardware and software.

[0162] Any of the software components or functions described in this application, such as processes 900, 1000, 1100, 1200, 1300, and / or 1400, may be implemented as software code executed by a processor using any suitable computer language such as Java, C, C++, C#, Objective-C, Swift, or a scripting language such as Perl or Python using prior art or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer-readable medium for storage and / or transmission. Suitable non-temporary computer-readable media may include random access memory (RAM), read-only memory (ROM), magnetic media such as hard drives or floppy disks, optical media such as compact disks (CDs), DVDs (digital versatile disks), or Blu-ray discs, flash memory, etc. The computer-readable media may be any combination of such devices. Furthermore, the order of operations may be rearranged. A process may terminate when its operation is complete, but may have additional steps not shown in the figures. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination can correspond to the function returning to its calling function or the main function.

[0163] Such programs may also be encoded and transmitted using carrier signals adapted for transmission over wired, optical, and / or wireless networks compliant with various protocols, including the Internet. Therefore, computer-readable media may be created using data signals encoded by such programs. Computer-readable media encoded with program code may be packaged with compatible devices or provided separately from other devices (e.g., via internet download). Any such computer-readable media may reside on or within a single computer product (e.g., a hard drive, CD, or an entire computer system), or on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing the user with any of the results referred to herein.

[0164] Any of the methods described herein can be performed in whole or in part using a computer system comprising one or more processors that can be configured to perform the steps. Any operation performed using a processor (e.g., alignment, determination, comparison, computing, calculation) may be performed in real time. The term “real time” may refer to a computing operation or process that is completed within a specific time constraint. The time constraint may be one minute, one hour, one day, or seven days. Thus, embodiments may cover a computer system configured to perform any of the steps of the methods described herein using different components that perform each step or each group of steps, where possible. Although presented as numbered steps, the steps of the methods herein may be performed simultaneously, at different times, or in different orders. In addition, parts of these steps may be used in conjunction with parts of other steps from other methods. Furthermore, all or part of the steps may be optional. Moreover, any of the steps of any of the methods may be performed using modules, units, circuits, or other means of a system for performing these steps.

[0165] In the aforementioned specification, embodiments of the disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative, not restrictive. The sole and exclusive indicator of the scope of the disclosure, and what the applicant intends to include, is the literal and equivalent scope of the set of claims issued from this application, including any subsequent amendments, in the particular form in which such claims are issued. Specific details of particular embodiments can be combined in any preferred manner without departing from the spirit and scope of the embodiments of the disclosure.

[0166] As used herein, the terms “and,” “or,” and “and / or” may have a variety of meanings, which may also depend at least partially on the context in which such terms are used. Typically, when “or” is used to relate a list such as A, B, or C, it is intended to mean A, B, and C, used here in an inclusive sense, as well as A, B, or C, used here in an exclusive sense. In addition, as used herein, the term “one or more” may be used to describe any singular feature, structure, or characteristic, or any combination of features, structures, or characteristics. However, it should be noted that these are merely examples, and the subject matter claimed is not limited to these examples. Furthermore, when the term “at least one of” is used to relate a list such as A, B, or C, it may be interpreted to mean any combination of A, B, and / or C, such as A, B, C, AB, AC, BC, AA, AAB, ABC, AABBCCC, etc.

[0167] Throughout this specification, references to “one example,” “a particular example,” or “exemplary implementation” mean that any particular feature, structure, or characteristic described in relation to a feature and / or example may be included in at least one feature and / or example of the claimed subject matter. Therefore, occurrences of phrases such as “in one example,” “in one example,” “a particular example,” “in a particular implementation,” or other similar phrases in various parts of this specification do not necessarily all refer to the same feature, example, and / or limitation. Furthermore, any particular feature, structure, or characteristic may be combined in one or more examples and / or features.

[0168] In some implementations, operation or processing may involve the physical manipulation of physical quantities. While not always the case, such quantities may take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, or otherwise manipulated. It has sometimes proven convenient, primarily due to common usage, to refer to such signals as bits, data, values, elements, symbols, characters, terms, digits, numerals, etc. However, it should be understood that all of these or similar terms should be associated with appropriate physical quantities and are merely convenient labels. Unless otherwise specified, as will be evident from the discussions herein, discussions using terms such as “processing,” “computing,” “calculating,” and “determining” throughout this specification are understood to refer to the operation or processes of dedicated devices such as dedicated computers, dedicated computing devices, or similar dedicated electronic computing devices. Therefore, in the context of this specification, a dedicated computer or similar dedicated electronic computing device can manipulate or transform signals that are typically represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmitting devices, or display devices of the dedicated computer or similar dedicated electronic computing device.

[0169] In the detailed description above, numerous specific details are provided to provide a complete understanding of the claimed subject matter. However, it will be understood by those skilled in the art that the claimed subject matter can be implemented without these specific details. In other cases, methods and apparatus known to those skilled in the art are not described in detail so as not to obscure the claimed subject matter. Therefore, the claimed subject matter should not be limited to the specific examples disclosed, and such claimed subject matter is also intended to include all embodiments and equivalents thereof that fall within the scope of the appended claims.

[0170] The terms and expressions used herein are intended to be descriptive rather than restrictive, and in using such terms and expressions, no equivalents of any illustrated or described features or parts thereof are intended to be excluded, although it should be recognized that various modifications are possible within the scope of the claims. Accordingly, while this disclosure includes specific embodiments and optional features, modifications and variations of the concepts disclosed herein may be made by those skilled in the art, and such modifications and variations should be understood to be within the scope of the appended claims.

[0171] When a term is used without explicit definition, it is understood that its ordinary meaning is intended unless the term has a special and / or specific meaning in the field of charged particle microscopy systems or other related fields. The terms “approximately,” “same,” “about,” “similar,” and “substantially” are used to indicate that deviations from a described characteristic or numerical value have little or no effect on the corresponding function, characteristic, or attribute of the described structure. In the illustrated example, where a dimensional parameter is described as “substantially equal” or “approximately” another dimensional parameter, the terms “substantially” or “approximately” are intended to reflect that the two dimensions being compared may not be equal within tolerances such as manufacturing tolerances. Similarly, where geometric parameters such as alignment or angular orientation are described as “almost” perpendicular, “substantially” perpendicular, or “substantially” parallel, “approximately” equal, or “approximately” the same, the terms “almost,” “substantially,” or “approximately” equal are intended to reflect that the alignment or angular orientation may differ from the precisely described state within tolerances (e.g., not exactly perpendicular). With respect to dimensional values ​​such as diameter, length, and width, the term "approximately" can be understood to describe a deviation of up to ±10% from the stated value. For example, a dimension of "approximately 10 mm" may represent a dimension between 9 mm and 11 mm. In this disclosure, "partial range" refers to a range of values ​​between two stated ranges and / or including one of the two stated ranges.

[0172] This specification provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the description following the exemplary embodiments provides instructions that enable those skilled in the art to implement various embodiments. It will be understood that various modifications can be made to the function and configuration of the elements without departing from the spirit and scope set forth in the appended claims.

[0173] Specific details are given herein to provide a complete understanding of the embodiments. However, it will be understood that embodiments may be carried out without these specific details. To avoid obscuring embodiments with unnecessary details, for example, specific system components, systems, processes, and other elements of this disclosure may be shown in the form of schematic diagrams or omitted from illustrative diagrams. In other cases, well-known circuits, processes, components, structures, and / or techniques may be shown without unnecessary details.

[0174] As used in this application and claims, the singular forms "a," "an," and "the" include the plural form unless the context specifically indicates otherwise. Additionally, the term "includes" means "comprises." Furthermore, the term "coupled" does not exclude the presence of intermediate elements between coupled items.

[0175] The systems, apparatus, and methods described herein should not be construed as limiting. Rather, this disclosure covers all novel and non-obvious features and aspects of the various disclosed embodiments, individually, and in various combinations and partial combinations thereof. The disclosed systems, methods, and apparatus are not limited to any particular aspects or features or combinations thereof, and the disclosed systems, methods, and apparatus do not require the existence of any one or more particular advantages or the resolution of any problem. Any operating theories provided are for the sake of explanation, but the disclosed systems, methods, and apparatus are not limited to such operating theories.

[0176] While some of the operations of the disclosed methods are described in a specific sequential order for convenience, it should be understood that the format of the specification is inclusive of reordering unless a specific ordering is required by the specific wording set forth below. For example, operations described sequentially may, in some cases, be reordered or performed simultaneously. Furthermore, for simplicity, the accompanying figures may not show the various ways in which the disclosed systems, methods, and apparatus can be used with other systems, methods, and apparatus. In addition, this specification may use terms such as “produce” and “provide” to describe the disclosed methods. These terms are high-level abstractions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation and will be readily recognizable to those skilled in the art.

Claims

1. A transmission electron microscope is configured according to a first acquisition setting that includes at least one of a first dose rate or a first operating mode, During the first period, the transmission electron microscope is operated to image a radiosensitive sample according to the first acquisition settings, After the first period, the transmission electron microscope is configured according to a second acquisition setting which includes at least one of a second dose rate different from the first dose rate, or a second operating mode different from the first operating mode. During the second period, the transmission electron microscope is operated to image the radiosensitive sample according to the second acquisition settings, This includes generating image data based on first data and second data collected during the first and second periods, respectively. Computer implementation method.

2. The computer implementation method according to claim 1, wherein the first operating mode includes a counting mode or an integration mode.

3. The computer implementation method according to claim 1, wherein the first acquisition setting includes the first dose rate, and the second acquisition setting includes the second dose rate.

4. The computer implementation method according to claim 1, wherein the second dose rate is greater than the first dose rate.

5. The computer implementation method according to claim 1, wherein the second acquisition setting is determined according to at least one of the time-dependent function of the radiation-sensitive sample or the expected radiation damage characteristics.

6. The computer implementation method according to claim 1, wherein the first period is determined based on the rate of radiation damage to the radiation-sensitive sample.

7. The computer implementation method according to claim 1, wherein the first period is determined based on at least one of the first dose rate or the first operating mode.

8. The computer implementation method according to claim 1, wherein the first data is collected using a first frame rate, and the second data is collected using a second frame rate lower than the first frame rate.

9. The computer implementation method according to claim 1, wherein at least one of the first acquisition setting or the second acquisition setting further includes at least one of duration or magnification.

10. The aforementioned image data includes a plurality of electronic counting frames, and the computer implementation method is The further includes encoding the image data using an image coding-decoding model trained to define encoded image data, The computer implementation method according to claim 1.

11. The computer implementation method according to claim 10, wherein the image data includes an image resolution represented using a first number of bits, and the encoded image data includes an image resolution using a second number of bits less than the first number of bits.

12. A non-temporary computer-readable storage medium containing instructions executable by one or more processors of a transmission electron microscope, The aforementioned instruction is, The transmission electron microscope is configured according to a first acquisition setting which includes at least one of a first dose rate or a first operating mode, During the first period, the transmission electron microscope is operated to image a radiosensitive sample according to the first acquisition settings, After the first period, the transmission electron microscope is configured according to a second acquisition setting which includes at least one of a second dose rate different from the first dose rate, or a second operating mode different from the first operating mode. During the second period, the transmission electron microscope is operated to image the radiosensitive sample according to the second acquisition settings, This includes generating image data based on first data and second data collected during the first and second periods, respectively. To make it perform an action, Non-temporary computer-readable storage medium.

13. The aforementioned operation is, At least the first dynamic representation of the first data is calculated based on the first acquisition setting, Converting the first data to a first spatiotemporal resolution representation using the first dynamic representation, At least the calculation of a second dynamic representation of the second data based on the second acquisition setting, Using the second dynamic representation, the second data is converted into a second spatiotemporal resolution representation, The first compressed image data is generated using the first spatiotemporal resolution representation and the first compression method, The present invention further includes generating a second compressed image data using a second compression method with the second spatiotemporal resolution representation, The non-temporary computer-readable storage medium according to claim 12.

14. The non-temporary computer-readable storage medium according to claim 13, wherein the first resolution of the first spatiotemporal resolution representation is greater than the second resolution of the second spatiotemporal resolution representation.

15. The non-temporary computer-readable storage medium according to claim 13, wherein the first compression method is determined based on the first dynamic representation, and the second compression method is determined based on the second dynamic representation.

16. The non-temporary computer-readable storage medium according to claim 13, wherein the second compression method is more irreversible than the first compression method.

17. Converting the first data to the first spatiotemporal resolution representation using the first dynamic representation is: This includes performing encoding using at least one of the following: an electronic event representation, a dose-splitting scheme, or an entropy coder and a probabilistic model. Converting the second data to the second spatiotemporal resolution representation using the second dynamic representation is: The process includes performing a second encoding using at least one of the electronic event representation, the dose division scheme, or the entropy coder and the probabilistic model. The non-temporary computer-readable storage medium according to claim 13.

18. The non-temporary computer-readable storage medium according to claim 13, wherein the first data represents a low-dose image frame or electronic event, and the second data represents a low-dose image frame or electronic event.

19. The non-temporary computer-readable storage medium according to claim 13, wherein at least one instruction from the first dynamic representation or the second dynamic representation is received from the graphical user interface of a user device.

20. The aforementioned operation is, Encoding a second image data set that includes at least one of the first spatiotemporal resolution representation or the second spatiotemporal resolution representation generates an encoded image dataset. The encoding image dataset and at least one of the following: drift correction, contrast transfer function correction, particle picking, feature segmentation, particle classification, 3D reconstruction, or artificial intelligence model, are used to generate an encoding dataset descriptor. A three-dimensional reference image is generated using the encoded dataset descriptor and the encoded image dataset. The further step includes generating a compressed second image data using the three-dimensional reference image, the encoded dataset descriptor, and the entropy encoder. The non-temporary computer-readable storage medium according to claim 13.

21. A transmission electron microscope comprising one or more memories for storing instructions, The system comprises one or more processors configured to execute the instructions for causing the transmission electron microscope to operate, The aforementioned operation is, The transmission electron microscope is configured according to a first acquisition setting which includes at least one of a first dose rate or a first operating mode, During the first period, the transmission electron microscope is operated to image a radiosensitive sample according to the first acquisition settings, After the first period, the transmission electron microscope is configured according to a second acquisition setting which includes at least one of a second dose rate different from the first dose rate, or a second operating mode different from the first operating mode. During the second period, the transmission electron microscope is operated to image the radiosensitive sample according to the second acquisition settings, This includes generating image data based on first data and second data collected during the first and second periods, respectively. Transmission electron microscope.

22. The aforementioned operation is, The above image data is encoded to generate an encoded image dataset, The coding of the image dataset and generating an coded dataset descriptor using at least one of the following: drift correction, contrast transfer function correction, particle picking, feature segmentation, particle classification, 3D reconstruction, artificial intelligence model, downsampled image data, or denoised image data. The further includes generating compressed image data using the encoded dataset descriptor and the entropy encoder, The transmission electron microscope according to claim 21.

23. Using the encoded dataset descriptor to generate the compressed image data is, This includes generating a three-dimensional reference image using the encoded dataset descriptor and the encoded image dataset. The transmission electron microscope according to claim 22.

24. The transmission electron microscope according to claim 23, wherein the three-dimensional reference image includes the expected number of electrons for one or more points in spacetime.

25. The transmission electron microscope according to claim 22, wherein the image data includes an image stack containing data representing multiple images.

Images