Acquisition and coding of electron microscope generated images

Abstract

A method includes configuring a transmission electron microscope according to a first acquisition setting, the first acquisition setting including at least one of a first dose rate or a first mode of operation. The method further includes operating the electron microscope to image the radiation-sensitive sample according to the first acquisition settings during the first time period. The method further includes configuring the electron microscope according to a second acquisition setting after the first time period, the second acquisition setting including at least one of a second dose rate different from the first dose rate or a second mode of operation different from the first mode of operation. The method further includes operating the electron microscope to image the radiation-sensitive sample according to the second acquisition settings during a second time period. The method further includes generating image data based on the first data and the second data collected during the first time period and the second time period, respectively.

Description

BACKGROUND

[0001] Charged particle microscopy methods such as transmission electron microscopy (TEM) and scanning electron microscopy (SEM) can use a range of detection techniques to obtain information about a sample. SEM and TEM techniques can be used to image a variety of types of samples including surfaces / interiors of cells, structures of protein molecules, organization of molecules in viruses, and cytoskeletal fibers, among others. The information obtained about the sample can be represented using a large amount of data (e.g., occupying a large amount of memory space). SUMMARY

[0002] In some embodiments, a computer-implemented method includes configuring a transmission electron microscope according to first acquisition settings, the first acquisition settings including at least one of a first dose rate or a first operating mode. The computer-implemented method also includes operating the transmission electron microscope according to the first acquisition settings to image a radiation-sensitive sample during a first time period. The computer-implemented method also includes configuring the transmission electron microscope according to second acquisition settings after the first time period, the second acquisition settings including at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode. The computer-implemented method also includes operating the transmission electron microscope according to the second acquisition settings to image the radiation-sensitive sample during a second time period. The computer-implemented method also includes generating image data based on first data and second data collected during the first time period and the second time period, respectively.

[0003] In some embodiments, a non-transitory computer-readable storage medium includes instructions executable by one or more processors of a transmission electron microscope to cause operations including configuring the transmission electron microscope according to first acquisition settings, the first acquisition settings including at least one of a first dose rate or a first operating mode. The operations also include operating the transmission electron microscope according to the first acquisition settings to image a radiation-sensitive sample during a first time period. The operations also include configuring the transmission electron microscope according to second acquisition settings after the first time period, the second acquisition settings including at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode. The operations also include operating the transmission electron microscope according to the second acquisition settings to image the radiation-sensitive sample during a second time period. The operations also include generating image data based on first data and second data collected during the first time period and the second time period, respectively.

[0004] In some embodiments, a transmission electron microscope includes one or more memories storing instructions and one or more processors configured to execute the instructions to cause the transmission electron microscope to perform operations. The operations include configuring the transmission electron microscope according to first acquisition settings, the first acquisition settings including at least one of a first dose rate or a first operating mode. The operations also include operating the transmission electron microscope to image a radiation-sensitive sample according to the first acquisition settings during a first time period. The operations also include configuring the transmission electron microscope according to second acquisition settings after the first time period, the second acquisition settings including at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode. The operations also include operating the transmission electron microscope to image the radiation-sensitive sample according to the second acquisition settings during a second time period. The operations also include generating image data based on first data and second data collected during the first time period and the second time period, respectively. BRIEF DESCRIPTION OF DRAWINGS

[0005] The foregoing aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings.

[0006] Figure 1 is an example illustration of a system for acquiring and encoding image data, in accordance with some embodiments.

[0007] Figure 2 is a schematic diagram of an electron microscope system, in accordance with some embodiments.

[0008] Figure 3 is a chart illustrating different dose rate schedules, in accordance with some embodiments.

[0009] Figure 4 is a chart illustrating loss of detection quantum efficiency as a function of cumulative dose, in accordance with some embodiments.

[0010] Figure 5 is a schematic diagram depicting an example of an image processing system, in accordance with some embodiments.

[0011] Figure 6 is a schematic diagram depicting an example of a compression system, in accordance with some embodiments.

[0012] Figure 7 is a schematic diagram depicting an example of a generative adversarial network for training an image generation model, in accordance with some embodiments.

[0013] Figure 8 is a schematic diagram depicting an example architecture of an image generation model, in accordance with some embodiments.

[0014] Figure 9 is a flowchart of an example process for training an image generation model using a generative adversarial network, in accordance with some embodiments.

[0015] Figure 10 is a flowchart of an example process for generating encoded image data, in accordance with some embodiments.

[0016] Figure 11 is a flowchart of an example process for acquiring image data, in accordance with some embodiments.

[0017] Figure 12 is a flowchart of an example process for encoding image data, in accordance with some embodiments.

[0018] Figure 13 is a flowchart of an example process for compressing image data, in accordance with some embodiments.

[0019] Figure 14 is a flowchart of an example process for acquiring image data and encoding the image data, in accordance with some embodiments.

[0020] Figure 15 depicts a schematic diagram of an example computer system that can be used with systems and methods in accordance with some embodiments of the present disclosure.

[0021] In the drawings, like reference numerals refer to like parts throughout the various views unless otherwise indicated. Not all elements of a device need be labeled in order to reduce clutter in the drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the described embodiments. DETAILED DESCRIPTION

[0022] Charged particle beam systems used in electron microscopy provide high resolution imaging by detecting signal electrons (e.g., backscattered electrons, secondary electrons, etc.) produced by elastic scattering of an electron beam emitted from an electron emitter interacting with atoms of a sample. In one example, electrons can be emitted from a cathode electrode heated by an electric current. The emitted electrons are attracted to an anode placed downstream of the cathode electrode, forming an electron beam that is directed to and interacts with the sample. The current of signal electrons emitted from the electron beam interacting with the sample is measured by one or more electron detectors. This current can be used to generate a high resolution image of the sample and represented as image data.

[0023] In conventional charged particle beam systems, image data is acquired by the image acquisition system over a period of time based on acquisition settings (e.g., dose rate, operating mode, duration, and / or magnification). Acquisition settings can result in acquiring more or less image data. Additionally, acquisition settings can cause the sample to degrade faster or slower than other image acquisition settings. Furthermore, acquisition settings can affect how long it takes to acquire a predetermined amount and / or quality of image data. After image data is acquired, storage, transmission, and / or use of the image data may be limited or require many resources (e.g., storage resources, processing resources, network resources) because the image data size can be large.

[0024] This disclosure discloses techniques for using two or more acquisition settings within a time period for collecting image data to maximize the amount and quality of information obtained from samples within a given time period. This disclosure also discloses techniques for encoding and / or compressing image data to reduce the resources used by encoded image data compared to the original image data. The techniques disclosed herein can be used independently of other techniques or in combination, which can achieve even greater benefits.

[0025] Generally speaking, encoded data represents data in different forms. As a basic example of encoding, the decimal number 10 can be encoded as hexadecimal and represented as 0xA in hexadecimal representation. Decimal-to-hexadecimal encoding schemes enable decimal numbers to be represented in hexadecimal, and hexadecimal-to-decimal encoding schemes enable hexadecimal numbers to be represented in decimal.

[0026] Compression is a form of encoding. Compression can be performed to reduce the number of symbols used to represent a given piece of information. Compressed information can be decompressed / uncompressed. Compression can be lossy or lossless. With lossless compression, all data is preserved after the data is decompressed. In other words, all the information represented by the data before compression is recovered after compression and subsequent decompression. On the other hand, in the case of lossy compression, the amount of data is reduced by permanently eliminating some information. In the case of lossy compression, all data may not be preserved after decompression. In other words, all the information represented by the data before compression may not be recovered after compression and subsequent decompression. For example, lossy compression can eliminate data that is not needed in the current application domain (e.g., image or video compression, where lossy compression results in artifacts in the decompressed image that are invisible or nearly invisible to a human observer).

[0027] In a first specific example, the acquisition settings of an electronically based imaging system can be adjusted (e.g., based on scheduling, user input, etc.) during different time windows of acquiring one or more images of the sample. The settings can be dynamically adjusted based on time, the sample as an image, cumulative dose, and / or other factors. In a specific example, the dose rate of the acquisition settings can be adjusted over time such that the dose rate increases over time.

[0028] The techniques described herein (including those described as first specific examples) offer numerous technical improvements, benefits, and advantages over existing solutions. For example, conventionally, a single acquisition setup can be used to generate images of a sample. The techniques described herein optimize the imaging process, such as by enabling the acquisition of the same image in less time, or by enabling the acquisition of better images in the same amount of time. Compared to conventional techniques, the image acquisition techniques described herein also reduce the amount of noise in the acquired images, thereby reducing the resources (e.g., network resources, processing resources, storage resources) required for processing, transmitting, and / or storing images.

[0029] In a second specific example, a data encoding scheme can be used to encode the image data generated by the image acquisition system. The image data may or may not have been acquired using the techniques described above for dynamically adjusting image acquisition settings. The encoding scheme can be configured to encode the image data based on the acquisition settings used to generate the image data.

[0030] The techniques described herein (including those described as a second specific example) offer numerous technical improvements, benefits, and advantages over existing solutions. For example, conventionally, encoding schemes may be static / pre-configured, and therefore the same encoding scheme can be used to encode all image data. Since the techniques described herein encode image data based on the acquisition settings used to generate the image data, the encoding scheme used can be selected based on what is optimal for encoding the image data. For example, when the image data includes information with little noise, the optimal encoding scheme may be one with lower loss. Furthermore, when the image data includes information with a large amount of noise, a more lossy encoding scheme may be optimal because more data can be discarded while preserving the same structural information included in the image, and simultaneously reducing the memory space occupied by the encoded image data compared to the image data before encoding. Encoding techniques can reduce the resources (e.g., network resources, processing resources, storage resources) used for processing, transmitting, and / or storing the encoded image compared to the image before encoding.

[0031] One such encoding scheme could be a machine learning model already trained to encode image data. The model may have been trained in various ways, such as using a Generative Adversarial Network (GAN). The model may have been trained using both the image data and the encoded image data as training data, such that the model is trained to generate encoded image data based on the image data. The encoded image data generated by the model can occupy less memory space than the received image data, thus using fewer resources (e.g., memory, processing, and / or network) compared to the received image data.

[0032] In the third specific example, the encoded image data can be subsequently encoded to compress it, further reducing the resources (e.g., network resources, processing resources, storage resources) required for processing, transmitting, and / or storing the compressed image compared to the image before compression. Compression of the encoded image data can occur after determining reference information. The reference information may include the most probable particle positions and other sample information included in the image data. The reference information can be used to determine how to compress the image data based on the reference information to be retained in the compressed data.

[0033] The techniques described herein (including those described as a third specific example) offer numerous technological improvements, benefits, and advantages over existing solutions. For example, typically, large amounts of data are generated by image acquisition systems, and the resources used for storing, processing, and transmitting image data can be substantial in quantity and / or size. The techniques described herein can reduce the resources required for storing, processing, and transmitting image data by compressing it in a manner that results in less loss of valuable information and / or further compression of information compared to conventional compression techniques.

[0034] While the remainder of this specification will conventionally refer to transmission electron microscopy (TEM), those skilled in the art will readily understand that the technique is not limited thereto. The invention is designed for use with other types of charged particle microscopes, such as scanning electron microscopy (SEM), scanning transmission electron microscopy (STEM), dual-beam systems including ion beam sources and electron beam sources, reflection electron microscopy (REM), circuit editing microscopes, etc. Therefore, this disclosure and claims should not be considered as limiting to any particular example microscope discussed, but can be used broadly with any number of electron microscopes capable of exhibiting some or all of the electrical or chemical properties of the discussed examples.

[0035] Figure 1 This is an example illustration of a system 100 for acquiring image data 108 and encoding the image data according to some implementation schemes. System 100 may include a computing system 104, an image acquisition system 106, and an image processing system 110.

[0036] The computing system 104 may be a user device (e.g., a laptop, personal computer, telephone, etc.). The computing system 104 may be a server. The computing system 104 may be able to receive input from user 102 via, for example, a user interface. In some embodiments, the input received by the computing system 104 includes indications of operating modes (e.g., counting mode, integration mode (also known as sampling mode or linear mode)), one or more dose rates, duration, sample information (e.g., sample material), and / or scheduling information. Input may enable the computing system 104 to transmit image acquisition settings to the image acquisition system 106. The image acquisition settings may include any combination of user input.

[0037] Image acquisition system 106 may include an electron microscope system (EMS). Image acquisition system 106 may generate image data based on data collected over one or more time periods. Image acquisition system 106 may be configured to operate according to one or more acquisition settings. Acquisition settings for operating image acquisition system 106 may be based on image acquisition settings received from computing system 104. Acquisition settings may be based on predetermined acquisition settings that may have been previously configured (e.g., by a user and / or the system). Acquisition settings may be based on input received via a user interface of image acquisition system 106, as described herein. In some embodiments, acquisition settings may include at least a first acquisition setting that includes at least a first dose rate and / or a first operating mode. Acquisition settings may be configured based on user input, sample characteristics, the purpose of imaging the sample, and available resources (e.g., network resources, processing resources, memory resources).

[0038] Image acquisition system 106 can operate according to acquisition settings to image radiation-sensitive samples during a time period. After this time period, image acquisition system 106 can be configured with different acquisition settings, which can be acquired, generated, or otherwise selected using the techniques described herein. Image acquisition system 106 can operate according to these different acquisition settings during a second time period to image radiation-sensitive samples.

[0039] Image acquisition system 106 can generate image data 108 based on first sample data, second sample data, and / or other sample data collected during a first time period, a second time period, and / or any other time period. Image data 108 may include one or more image stacks. Image stacks may represent multiple images. Each image in the image stack may be an electronic counting frame. One or more electronic coordinates may be used to describe an electronic counting frame.

[0040] Image processing system 110 can use image data 108 to generate encoded image data 112. Image processing system 110 can receive image data 108 from image acquisition system 106 or another source of image data 108 (e.g., data storage, computing system 104). Image processing system 110 can use one or more processing techniques to process image data 108 to generate encoded image data 112.

[0041] In some embodiments, the processing techniques may include using the image processing system 110 to dynamically encode corresponding portions of the image data 108 based on the acquisition settings used during the acquisition of portions of the image data 108. For example, if a first portion of the image data 108 was acquired using a first acquisition setting and a second portion of the image data 108 was acquired using a second acquisition setting, the first portion of the image data 108 may be encoded using an encoding scheme different from the encoding scheme used to encode the second portion of the image data 108.

[0042] In some implementations, the processing techniques may include encoding image data 108 using image processing system 110 to generate intermediate encoded image data, and then compressing the intermediate encoded image data to generate compressed image data represented by encoded image data 112.

[0043] In some implementations, the processing techniques may include an image processing system 110 using an artificial intelligence (AI) model (e.g., a machine learning (ML) model) to encode image data 108, the AI ​​model being trained to generate encoded image data 112.

[0044] Image data 108 and / or encoded image data 112 may be transmitted to computing system 104 or downstream consumers (e.g., databases). Any combination of one or more devices may be used to perform the processes described for system components of system 100 and other systems described herein. Processing may be performed locally and / or remotely from one or more devices.

[0045] Figure 2This is a schematic diagram of an electron microscope system (EMS) 200 according to some embodiments. The EMS 200 is an example of an image acquisition system (e.g., the image acquisition system 106 described herein). The EMS 200 may include a TEM 202. The TEM 202 can be used to acquire images of the sample 206. In some examples, the TEM 202 may include an energy dispersive spectrometer 204. For example, the energy dispersive spectrometer 204 can be used to obtain an EEL spectrum. The EMS 200 may include optical components to reduce or eliminate any defocusing problems that occur during spectrum acquisition. In some examples, the optical components may be biased to a certain level based on the operating parameters of the EMS 200 and / or the energy dispersive spectrometer 204, such that refocusing is dynamic.

[0046] TEM 202 may include a source 228, an illumination system 230, a projection system 208, and various detectors 232, all of which may be controlled by a controller 226. The source 228 may be an electron source, such as a Schottky source or a (cold) field emission gun (CFEG), and may provide an electron beam that propagates along the optical axis of TEM 202 to interact with the sample 206. The illumination system 230 may include multiple electro-optical components to adjust the electron beam delivered to the sample 206. Adjusting the electron beam may include collimation, astigmatism correction, and focusing the electron beam onto the sample plane. The projection system 208 may include various electrostatic / magnetic lenses, deflectors, correctors (e.g., astigmatism correction devices), etc., which may be used to focus the electron beam emitted from the sample 206 onto one of the various detectors 232. The projection system 208 may be configured to focus an image of the sample 206 onto the detector 232 at a specific (adjustable) magnification; this is generally referred to as the "imaging mode" of the projection system. Alternatively, the projection system 208 can be configured to focus an image (or “diffraction pattern”) of the angular distribution of the emitted electrons onto the detector 232 at a specific (adjustable) magnification. This diffraction pattern is formed on a first magnifying lens (“objective”). Figure 2 In the back focal plane (not shown). This alternative mode of the projection system is generally referred to as the "diffraction mode", and the magnification in this alternative mode is generally referred to as the "camera length". However, for simplicity, when the term "image" or "focus" is used in this disclosure, it refers to both the "imaging mode" and the "diffraction mode" of the imaging system. Similarly, when the term "magnification setting" is used in this disclosure, it refers to both the "magnification setting" and the "camera length setting" of the imaging system.

[0047] In some examples, the projection system 208 focuses and adjusts the electron beam delivered to the energy dispersive spectrometer 204. Various detectors 232 can be individually moved into and out of the optical path to provide different detection schemes for the TEM 202. The various detectors 232 may include an imaging screen, a TEM camera, and a STEM camera.

[0048] The energy dispersive spectrometer 204 may include a dispersive element 212 (with a bias tube 214), optical components 234, multiple optical devices 220, and a detector 222. About Figure 1 Many components of the energy dispersive spectrometer 204 have been discussed, and will not be repeated for the sake of brevity. Figure 1 Other components of the spectrometer 204 described herein, but not specifically shown in the spectrometer 204, are included, but... Figure 2 The [specific element] is omitted. An additional component, namely optical component 234, is included in the energy dispersive spectrometer 204 to correct the focusing gradient of the electron energy loss spectrum across the detector, as discussed. Optical component 234 can be a single element, such as a magnetic hexapole. Alternatively, optical component 234 can include multiple multipole elements (tetrapole, hexapole, octapole, etc.), which can be electric or magnetic in their operating nature. Furthermore, in addition to refocusing the spectrum, optical component 234 can perform several alternative functions simultaneously or not, such as correcting image distortion or correcting spectral distortion. Figure 2 In this example, optical component 234 is shown downstream of dispersive element 212, but this is not the only location where optical component 234 can be arranged. Typically, some elements of optical component 234 may be arranged upstream of dispersive element 212 (but downstream of sample 206), and some elements of optical component 234 may be arranged downstream of dispersive element 212 within or after multiple optical devices 220.

[0049] Controller 226 may include one or more processing cores and memory storing executable code. Additionally, controller 226 may provide operating voltages to some components of EMS 200 or be coupled to a voltage supply (not shown) that provides operating voltages in response to control signals provided by controller 226. For example, controller 226 may provide control and / or voltages to lighting system 230, projection system 208, or optical components 234. Furthermore, controller 226 may control the operation of detector 222 and / or receive data from detector 222. Typically, controller 226 may set operating parameters of EMS 200 and may adjust the electrical bias of optical components 234 in response to changes in operating conditions of EMS 200, such as magnification, to dynamically focus the electron beam onto detector 222.

[0050] The controller 226 can receive inputs for configuring the EMS 200 (e.g., user input, input from a computing system). The EMS 200 can be configured according to acquisition settings. Acquisition settings may include dose rate, operating mode, duration (e.g., 1 second, 5 seconds, 10 seconds, etc.), and / or magnification. Dose rate is the number of electrons received by the imaged sample within a time period. For example, dose rate can be measured in electrons per second. Operating modes may include counting mode or integrating mode.

[0051] In counting mode, individual electron events can be identified and digital events can be accumulated over time. Within the operating range of counting mode, the quality of image data 108 generated using counting mode can decrease with increasing dose rate. For example, if an excessively high dose rate is used, high overlap loss occurs, where all individual electrons cannot be located because they are too close to each other in space and time.

[0052] In integral mode, the dose rate may differ from (e.g., be greater than) that in counting mode. In integral mode, the maximum dose rate may depend on the linearity of the camera's performance within its operating range and how well the pixels saturate to their maximum value. The image quality of the generated image data 108 may be constant with respect to the dose rate. The dose rate may have a fixed image quality that is worse than the image quality obtained using counting mode (which may be referred to as detection quantum efficiency (DQE)). The integral mode may have an operating range that differs from that of the counting mode (partially (e.g., overlapping) or entirely (non-intersecting)). Integral mode can be used during acquisition to accommodate very large doses. In an example where a total dose of 100 electrons per pixel is desired and the electron count dose is 0.025 electrons per pixel, integral mode can be used due to the high desired dose.

[0053] The controller 226 can configure the EMS 200 to operate over a period of time according to acquisition settings to collect sample data of sample 206. Sample 206 may be a radiation-sensitive sample. The EMS 200 may generate image data 108 as a result of the sample data. In some embodiments, image data 108 is sample data. In some embodiments, first sample data and second sample data are combined to represent image data 108.

[0054] In some implementations, controller 226 may configure EMS 200 to operate according to a first acquisition setting of image sample 206 during a first time period. Controller 226 may subsequently configure EMS 200 to operate according to a second acquisition setting of image sample 206 during a second time period. During each time period, sample data that can be used to generate image data 108 may be collected. The first acquisition setting may be the same as or different from the second acquisition setting. The dose rate of the second acquisition setting may be different from (e.g., greater than) the dose rate of the first acquisition setting. The operating mode of the second acquisition setting may be different from the operating mode of the first acquisition setting. The second time period may be a different duration than the first time period. The time period for which EMS 200 operates using the acquisition settings may be determined by the rate of radiation damage suffered by sample 206. The time period for which EMS 200 operates according to the acquisition settings may be based on previous acquisition settings, dose rate, and / or operating mode. Furthermore, controller 226 may configure the frame rate for acquiring data using EMS 200. The frame rate may be different during the second time period compared to the first time period.

[0055] In some implementations, controller 226 may configure EMS 200 to operate according to a first acquisition setting, a second acquisition setting, or other acquisition settings during any number of time periods to image sample 206. The acquisition settings may be determined based on a time-related function (e.g., an exponential function, a linear function), user input, the purpose of imaging sample 206, and / or the expected radiation damage characteristics of sample 206.

[0056] When sample 206 is radiation sensitive, it can degrade with cumulative dose (e.g., as more electrons interact with the sample). Due to radiation damage, electrons detected at the end of the acquisition period may carry less information about the sample than electrons detected at the beginning of the acquisition period. Additionally, events detected in high-dose-rate frames may carry less information about the imaged sample 206 than events detected in low-dose-rate frames. Image data 108 obtained from EMS 200 may be more accurate at lower dose rates than at higher dose rates. The image quality (e.g., accuracy) of image data 108 may decrease with increasing dose rate. Certain embodiments may maximize the amount of information obtained from sample 206 within a given time period.

[0057] In one example, controller 226 may cause a change in acquisition settings during the acquisition of image data 108. For example, controller 226 may configure EMS 200 to acquire first sample data for 0.5 seconds in counting mode at a dose rate of 4 electrons per second (e / p / s). After 0.5 seconds, controller 226 may configure EMS 200 to acquire second sample data for 2 frames (e.g., 2 / 320 seconds) in integration mode (e.g., single-sampling mode) at a dose rate of 6080 electrons per second (e.g., 19 electrons per pixel per frame). The first sample data may represent a first frame or a first set of frames of image data 108, and the second sample data may represent a second frame or a second set of frames of image data. The first and second sample data may be combined (e.g., concatenated) to generate image data 108. Controller 226 may combine two or more sample data to form image data 108.

[0058] To increase the speed at which acquisition settings can be changed, a shutter (e.g., a fast blanking device with PWM modulation) can be used. The acquisition settings can be set to the maximum desired dose rate, allowing the fast shutter to open and close at a given frequency (e.g., high frequency, low frequency) or at the camera frame rate (e.g., high camera rate, low camera rate). The shutter rate can be used to control the dose rate. For example, the shutter rate can be changed (e.g., by controller 226) from a first shutter rate to a second shutter rate greater than or less than the first shutter rate (e.g., changing it to 1 / 10 of the first shutter rate). By changing the shutter rate, the acquisition settings can be changed in a shorter time period than other methods of changing acquisition settings. For example, while other techniques require several seconds to change the acquisition settings, the shutter rate can be changed in less than one second.

[0059] In embodiments where one or more acquisition settings (e.g., dose rate) are adjusted based on cumulative dose, when sample 206 has been subjected to relatively small radiation damage (e.g., up to 1 electron per angstrom) 2 When the dose rate is adjusted based on the cumulative dose, sample 206 can be irradiated at a low dose rate. Furthermore, when sample 206 has already suffered increased radiation damage, sample 206 can be irradiated at a higher dose rate. This dynamic dose rate scheduling can be optimized to obtain maximum information (e.g., high-quality image data 108) from sample 206 within a fixed acquisition period. This type of acquisition method can generate high-quality image data 108 and image data 108 that occupies less memory space compared to other acquisition methods. Image data 108 that occupies less memory space can be transmitted and used by other systems with less computational resource (e.g., network, processing) overhead.

[0060] Optical component 234 may be formed of a multipole element comprising two or more conductive elements. In some examples, optical component 234 is formed of two opposing conductive elements housed in a conductor, similar to a deflector-type or shutter-type arrangement. In other examples, optical component 234 is formed of a quadrupole or higher-order multipole element. The excitation of optical component 234 may be electrical, magnetic, or a combination of these properties, or more generally, electromagnetic. Regardless of the physical example, optical component 234 may be excited to a level based on the operating parameters of TEM 202 and / or energy dispersive spectrometer 204. Any combination of dispersive element 212, optical component 234, and multiple optical devices 220 may be used. For example, in some embodiments, dispersive element 212, optical component 234, and multiple optical devices 220 are not included in EMS 200.

[0061] In operation, an electron beam of primary energy generated by source 228 can be projected toward sample 206, where the electron beam interacts with sample 206. This interaction can cause some electrons to lose a certain amount of energy, which is related to various material properties of sample 206. The electron beam emitted from sample 206 can then include electrons of different energies within a certain energy range. The emitted electron beam can then propagate along a different trajectory than the primary energy electrons, which can affect the focus, such as the crossover position, along the rest of the optical path including spectrometer 204 and TEM 202. In some examples, operation of any component that can adjust (manipulate) the electron beam can subsequently result in defocusing in the spectral plane or can result in a focusing gradient on the spectral plane. In this example, to refocus the electron beam, optics 234 can be excited based on the current operating parameters. In various examples, exciting optics 234 can align the crossover position with the spectral plane, and thus with detector 222.

[0062] Figure 3 This is a graph 300 illustrating different dose rate schedules according to some exemplary embodiments. Graph 300 has a Y-axis representing the electronic dose rate per pixel per second. The X-axis is time. This graph illustrates three simple examples of dose rate scheduling: flat scheduling, linear scheduling, and exponential scheduling. Flat dose rate scheduling is not dynamic dose rate scheduling because the dose rate does not change over time. On the other hand, linear dose rate scheduling and exponential dose rate scheduling are examples of dynamic dose rate scheduling because the dose rate changes over time.

[0063] Each example dose rate schedule in the example dose rate schedules in Figure 300 accumulates approximately 40 electrons over a time period (e.g., 4 seconds). The number of electrons accumulated within a dose rate schedule is equal to the area under the corresponding dose rate schedule curve. Figure 300 illustrates how the accumulation of electrons is affected by the dose rate.

[0064] Using a flat dose rate schedule of 10 electrons per second per pixel, 40 electrons were accumulated over a 4-second time period. Using a linear dose rate schedule starting from 0 (or slightly above 0) with a slope of 5 electrons per second per pixel, 40 electrons were accumulated over a 4-second time period. Using an exponential dose rate schedule starting from 0, 40 electrons were accumulated over a time period of approximately 3.5 seconds.

[0065] Each dose rate schedule shown results in a different detection efficiency. The detection efficiency of the dose rate can be visualized as the detection quantum efficiency (DQE) as a function of the cumulative dose. The DQE as a function of the cumulative dose is illustrated in Figure 400 for each dose rate schedule, as described below.

[0066] For illustrative purposes, only the simple dose rate schedule described above is shown in Figure 300. Those skilled in the art who benefit from this disclosure will recognize that other dose rate schedules are possible and can be functions of time or another variable. Furthermore, in some embodiments, any combination of dose rates or other acquisition settings may be varied based on a function of time or a combination of variables.

[0067] Figure 4 This is a graph 400 illustrating the detection quantum efficiency (DQE) loss as a function of cumulative dose according to some exemplary embodiments. The dose rate scheduling in graph 300 results in different detection efficiencies, which are visualized in graph 400 as the DQE loss as a function of the cumulative dose of the first two electrons (expressed as a percentage point (e.g., 0.15 = 15%)). The Y-axis of graph 400 is the DQE loss measured as a percentage point. The X-axis of graph 400 is the cumulative dose measured as the number of electrons per pixel. The DQE loss can be the loss compared to the camera's optimal DQE (e.g., the DQE when using the lowest possible dose rate, which typically requires a large exposure time).

[0068] Compared to other electrons interacting with the sample (e.g., sample 206), the earlier an electron interacts with the sample, the less degradation occurs, and therefore higher quality data can be obtained from the sample via the electron. Thus, the first two electrons carry more high-frequency information about the sample (e.g., protein) (as demonstrated by the lower DQE loss) compared to later electrons (as demonstrated by the higher DQE loss). The later an electron interacts with the sample, the more degradation occurs, and therefore the electron will not carry as much information about the finer details of the sample. In some embodiments, later electrons can be used for particle selection even if the information carried by them (e.g., electrons after the first and second electrons) may have a higher DQE loss.

[0069] Figure 400 illustrates the DQE loss as a function of the cumulative dose in each of the flat dose rate schedules, linear dose rate schedules, and exponential dose rate schedules from Figure 300.

[0070] Figure 400 illustrates that for the first two electrons, exponential dose rate scheduling is equivalent to 70% faster flat dose rate scheduling on a flat camera. In other words, compared to flat dose rate scheduling, using exponential dose rate scheduling reduces the amount of time required to acquire the same image data (e.g., image data 108) by 70%. Furthermore, exponential dose rate scheduling can achieve the same performance (e.g., DQE loss) as flat dose rate scheduling using 40 electrons with five times shorter exposures (0.8 seconds). Because using dynamic dose rate scheduling reduces DQE loss compared to flat dose rate scheduling, it can lead to a reduction in the storage requirements of the acquired image data.

[0071] For dynamic dose rate scheduling, the DQE loss for a given cumulative dose is lower compared to flat scheduling. Low DQE is not a problem because the sample to be imaged is already damaged at the end of the acquisition period (e.g., after previous electrons have already interacted with the sample). Information acquired during the later part of the acquisition period may not be used to obtain high-resolution sample information. Information acquired during the later part of the acquisition period can be used for intermediate steps such as particle selection.

[0072] Figure 5 This is a schematic diagram depicting an example of an image processing system 500 according to some embodiments. The image processing system 500 may be part of the image processing system 110 as described with respect to system 100. The image processing system 500 may include a first encoding system 504 and a compression system 516. In some embodiments, the image processing system 500 is used in conjunction with a dynamic image acquisition setup 502 as described above. The image processing system 500 can be used to generate encoded image data 514 and / or compressed image data 518.

[0073] The first encoding system 504 may receive image data 108 and acquisition settings 502. Image data 108 may have been generated by an image acquisition system (e.g., image acquisition system 106, EMS 200). Acquisition settings 502 may have been used by EMS 200 to generate image data 108. The first encoding system 504 may include an encoding selection system 506 that can select encoding scheme 508 from one or more encoding schemes represented by a first encoding scheme 508a, a second encoding scheme 508b, up to an Nth encoding scheme 508n. The first encoding system 504 may use encoding scheme 508 to convert image data 108 into encoded image data 514.

[0074] The encoding selection system 506 can receive image data 108 and acquisition settings 502 (e.g., dose rate). The encoding selection system 506 can dynamically calculate an encoding scheme 508 for representing the image data 108. Each encoding scheme 508 can be used to encode the image data 108 into a different representation. The representation and encoding scheme 508 can be described as dynamic because the encoding scheme 508 used can be based on the acquisition settings 502 and / or the image data 108. The calculation of the dynamic representation for representing the image data 108 can determine which encoding scheme among the available encoding schemes 508 will be selected to encode the image data 108. In some embodiments, the optimal encoding scheme 508 for encoding the image data 108 is calculated using the acquisition settings 502 and / or the image data 108.

[0075] The encoding selection system 506 can determine the subsequent encoding scheme 508 to be used after using the previous encoding scheme 508, based on the total length of electron exposure of the sample (e.g., sample 206) and / or a portion of the total exposed length of the sample, after a predetermined time period when the acquisition settings 502 have changed. The ability of the encoding selection system 506 to dynamically select the encoding scheme 508 to be used based on the acquisition settings 502 and / or other factors enables the first encoding system 504 to generate dynamic / adaptive encoded image data 514.

[0076] In one example, a first dynamic representation for representing first image data (a subset of image data from image data 108) is calculated by the encoding selection system 506 based at least on a first acquisition setting. The first image data can be converted into a first spatiotemporal resolution representation using a first encoding scheme 508a. Furthermore, a second dynamic representation for representing second image data (a subset of image data from image data 108) is calculated by the encoding selection system 506 based at least on a second acquisition setting to represent the second image data. The second image data may be received by the first encoding system after the first image data and may be part of the same sample imaging session (e.g., a different portion of image data 108 output from the image acquisition system). The second image data may be different from the first image data. The second image data can be converted into a second spatiotemporal resolution representation using a second encoding scheme 508b. The second dynamic representation may differ from the first dynamic representation. In this example, the first encoding system 506 is capable of dynamically encoding image data such that the first image data is encoded using an encoding scheme 508 different from the second image data. The second encoding scheme 508b may have greater loss than the first encoding scheme 508a. Compared to the second spatiotemporal resolution representation / second encoded image data, the first spatiotemporal resolution representation / first encoded image data may include a higher spatiotemporal resolution.

[0077] By using different encoding schemes 508, high-resolution sample information that may be included in the first image data can be encoded differently from lower-resolution sample information that may be included in the second image data (e.g., with a higher spatiotemporal resolution representation). Different encoding schemes 508 can reduce the amount of memory occupied by the encoded image data 514 and / or increase the spatiotemporal resolution of the encoded image data 514 compared to using a single encoding scheme 508 for the first and second image data. Different encoding schemes 508 can minimize the amount of memory occupied by the encoded image data 514 while maximizing the spatiotemporal resolution of the encoded image data 514. In some embodiments, the first and second image data can be passed in parallel through one or more first encoding systems 504. The first encoding system 504 can reduce redundant information and / or the amount of discarded information including related image data 108.

[0078] The coding selection system 506 may select a coding scheme 508 based on the number of emitted electrons, the number of electrons that have interacted with a given region of the sample, and / or the time period, etc. The coding scheme 508 included in a set of one or more coding schemes and / or the factors considered by the coding selection system 506 to select the coding scheme 508 may be determined by instructions received from a graphical user interface of a user device (e.g., a laptop, tablet, desktop computer, image acquisition system, etc.).

[0079] After the encoding selection system 506 calculates the dynamic representation for the image data 108, the encoding selection system 506 can cause the image data 108 to be encoded using an encoding scheme 508 (e.g., a first encoding scheme 508a) from a set of encoding schemes. The encoding selection system 506 can then transmit the image data 108 to the selected encoding scheme 508. The encoding selection system 506 can also otherwise cause the selected encoding scheme 508 to receive the image data 108.

[0080] The selected encoding scheme 508 encodes image data 108 into encoded image data 514. Encoding scheme 508 can represent image data 108 in different / encoded representations. Encoding scheme 508 may or may not cause the encoded image data 514 to occupy less space in memory than image data 108. Encoding scheme 508 can represent image data 108 in a spatiotemporal resolution representation. Different encoding schemes 508 may result in image data 108 being represented at a higher spatiotemporal resolution and / or occupying less memory compared to other encoding schemes 508. In some embodiments, an inverse encoding scheme corresponding to the encoding scheme used to generate encoded image data 514 from image data 108 can be used to obtain image data 108 from encoded image data 514. In other words, the inverse of the encoding scheme (which may also be referred to as a decoding scheme) can be used to decode the encoded image data.

[0081] Examples of encoding scheme 508 may include an electronic event representation (EER). An EER can encode electronic events as (X, Y, time). Event locations can be stored on a progressively coarser spatiotemporal sampling grid. In some embodiments, it may be more efficient to begin encoding the image data 108 using a different encoding scheme (e.g., a dose-segmentation scheme) when the amount of electrons in each sampling grid coordinate becomes too high.

[0082] Another example of encoding scheme 508 may include a dose segmentation scheme. A dose segmentation scheme may include a fixed and / or moving spatiotemporal resolution. A fixed spatiotemporal resolution keeps the spatiotemporal resolution constant for the selected dose segmentation scheme. A moving spatiotemporal resolution allows the spatiotemporal resolution to be adjusted based on image size, the number of camera frames per dose fraction, and / or other factors (e.g., acquisition settings 502). A dose segmentation scheme divides (“segments”) the total expected exposure dose of a certain amount of electrons per pixel into partial exposures over time to correct for local and global motion. For example, if 20 dose fractions are generated using an exposure of 20 electrons per pixel, 20 images can be generated, each with 1 electron per pixel. The segmentation can be selected independently of the camera’s native frame rate, so an exposure of 20 electrons per pixel could result in 1000 native frames; in this case, each dose fraction would therefore consist of 1000 / 20 frames = 50 frames.

[0083] Another example of encoding scheme 508 may include an image encoding-decoding model. The image encoding-decoding model can be trained to define encoded image data 514. The image encoding-decoding model may have already been trained using a generative adversarial network (GAN). Implementation schemes for the image encoding-decoding model are described in more detail below (e.g., regarding...). Figure 7 and Figure 8 (and other parts).

[0084] In some embodiments, the encoding scheme set may include more than one type of encoding scheme 508. For example, the encoding scheme set may include a first encoding scheme 508a as a first dose segmentation scheme configured in a first manner, and may also include a second encoding scheme 508b as a second dose segmentation scheme configured in a second manner.

[0085] Encoded image data 514 may be encoded image data 112 as described with respect to system 100. Encoded image data 514 may represent a low-dose image frame or an electronic event. In some embodiments, first encoded image data generated using first image data represents an electronic event, and second encoded image data generated using second image data represents a low-dose image frame. In some embodiments, encoded image data 514 is re-encoded by compression system 516 to generate compressed image data 518. Compressed image data 518 may be encoded image data 112 as described with respect to system 100.

[0086] Compression system 516 can compress encoded image data 514 (e.g., image data represented in spatiotemporal resolution) to generate compressed image data 518. In some embodiments, compression system 516 can generate compressed image data 518 based on a dynamic representation / encoding scheme 508 used with image data 108. Since compression system 516 can generate compressed image data 108 based on the dynamic representation / encoding scheme 508 used for image data 518, compression system 516 can dynamically generate first compressed image data based on a first encoding scheme used, and dynamically generate second compressed image data based on a second encoding scheme used. In other words, a different compression scheme than the second portion of image data 602 can be used to compress the first portion of image data 602.

[0087] Compression system 516 can compress the encoded image data 514 into a compressed representation. In some embodiments, compression system 516 may use compression schemes such as Lempel-Ziv-Welch (LZW) or ZIP files to compress the encoded image data 514. Depending on the encoding scheme 508 used with image data 108 to generate the encoded image data 514, the loss in the compressed representation may be greater or less. Reference Figure 6 The second coding system will be described in more detail.

[0088] Figure 6This is a schematic diagram depicting an example of a compression system 516 according to some embodiments. Compression system 516 can be used to generate compressed image data 634. Compressed image data 634 can be the same compressed image data as compressed image data 518. Compressed image data 634 can be a compressed form of image data 602. Image data 602 can be represented in a spatiotemporal representation and received by compression system 516. Image data 602 may have been encoded to a spatiotemporal resolution (e.g., encoded image data 514). Compression system 516 can use image data 602 to generate compressed image data 634. Compression system 516 can be used to generate application-customized data compression. Compression system 516 may include a dataset description generation system 604, a reference image generation system 620, and an informed compression subsystem 632.

[0089] Compression system 516 may receive image data 602 from a first encoding system (e.g., first encoding system 504), an image acquisition system (e.g., image acquisition system 106), and / or another image data source. Image data 602 may include high-contrast image data 602 (e.g., image data obtained using TEM or dark-field imaging techniques with a phase plate). Image data 602 may include an image stack. The image stack may include data representing multiple images.

[0090] The dataset description generation system 604 can use image data 602 to generate a dataset descriptor 618. The dataset description generation system 604 can use a drift correction system 606, a contrast transfer function (CTF) estimation and correction system 608, a particle picking system and / or a feature segmentation system 610, a particle classification system 612, a three-dimensional (3D) reconstruction system 614, an artificial intelligence model (not shown), and / or downsampled image data (not shown) to generate the dataset descriptor 618. In some embodiments, the dataset descriptor 618 may include tomographic tilt series alignment (e.g., generated by a tomographic tilt series alignment method), downsampled images (e.g., generated by a downsampled image generator), and / or encoded image data (e.g., generated by an image generation model 704).

[0091] The drift correction system 606 can generate drift profiles for image data 602 and / or each image stack. The drift correction system 606 can measure the drift of the image stack (e.g., each image stack is measured independently of other image stacks). The drift can be a field-of-view shift as a function of time. The drift correction system 606 can be used to estimate local sample deformation (e.g., electron beam-induced sample motion).

[0092] The contrast transfer function (CTF) estimation and correction system 608 determines the extent to which phase shift and microscopic aberrations have corrupted the image data 602. The parameters of the function can be determined by the CTF estimation and correction system 608. Given estimated phase shift and microscopic aberrations, the CTF estimation and correction system 608 can process the image data 602 such that the effects of phase shift and microscopic aberrations are reduced or eliminated.

[0093] A particle selection system can locate positions in image data 602 where particles of interest may exist. Since particles are actually 3D objects, while image data 602 is 2D, the particle selection system searches for 2D projections of the 3D particles of interest. The particle selection system can use various algorithms, such as general methods designed to find “blob-like” structures within a given size range. The particle selection system can use template matching methods, where an earlier obtained 3D model is projected into multiple 2D template images with different projection orientations, and those projected 2D template images are used as templates to be detected in image data 602. The particle selection system can also use artificial intelligence algorithms.

[0094] Feature segmentation systems can be used to detect features that are not of interest for 3D particle reconstruction but are of interest for describing image data 602 (e.g., contamination).

[0095] The particle classification system 612 groups selected particles into particle categories with the same appearance. Particles with the same appearance may have the same projection orientation. By combining particles with the same projection orientation, a denoised particle projection image can be obtained, which can then be used for reconstruction (e.g., initial reconstruction).

[0096] The three-dimensional (3D) reconstruction system 614 can estimate a 3D model for image data 602 and / or each image stack. The three-dimensional (3D) reconstruction system 614 can infer the relative angles between different particles or particle categories. When the angles are known, tomographic reconstruction can be performed to obtain the 3D model.

[0097] An artificial intelligence model can be used to infer / generate a dataset descriptor 618 based on input including one or more image stacks and / or other information obtained using one or more image stacks. The artificial intelligence model may have been trained to generate a dataset descriptor 618 that produces optimal compression. In some implementations, the artificial intelligence model may be an image generation model (e.g., image generation model 704).

[0098] Downsampled image data can be generated for each image stack. A summed and downsampled image stack can be generated, represented by a dataset descriptor 618. The reference image generation system 620 can then upsample the downsampled image stack to generate a reference stack 630.

[0099] A drift correction system 606, a CTF estimation and correction system 608, a particle selection system and / or a feature segmentation system 610, a particle classification system 612, a 3D reconstruction system 614, an artificial intelligence model, downsampled image data, and / or denoised image data can be used by an attribute extraction system 616 to generate a dataset descriptor 618. The attribute extraction system 616 can extract data about image data 602 from the data generated by the drift correction system 606, the CTF estimation and correction system 608, the particle selection system and / or the feature segmentation system 610, the particle classification system 612, the 3D reconstruction system 614, the artificial intelligence model, and / or the downsampled image data. The extracted data can be included in the dataset descriptor 618.

[0100] In the example of attribute extraction system 616, attribute extraction system 616 collects all compact descriptive results from all processing steps (e.g., drift correction system 606, CTF estimation and correction system 608, etc.) and concatenates them into a dataset descriptor 618. The compact descriptive results may include a minimal description of what was produced by drift correction system 606, CTF estimation and correction system 608, etc. For example, the compact descriptive results may include particle selection: x, y coordinates and image numbering, but exclude cropped and bounded particle images.

[0101] In one example, dataset descriptor 618 includes the coordinates and projection orientation of the particles, defocus, drift profile, and / or the location of dirt on the sample (e.g., sample 206). Dataset descriptor 618 can be generated for image data 602 and / or each image stack included in image data 602. Dataset descriptor 618 can be used to generate a probabilistic model for each spatiotemporal location in each image stack of image data 602. In other words, dataset descriptor 618 can represent the most probable particle location and other sample information based on information included in the image stacks of image data 602. Dataset descriptor 618 can describe the dose used to acquire image data 602. The actual values ​​can be encoded using informed compression subsystem 632. In some embodiments, the more accurate the dataset descriptor 618, the better / larger the expected compression of image data 602, which is compressed and represented as compressed image data 634.

[0102] The reference image generation system 620 can use a dataset descriptor 618 to generate a reference stack 630. The reference stack 630 may correspond to an image stack included in the image data 602. Each reference stack 630 may represent a desired number of electrons at one or more points in spacetime. Each reference stack 630 may represent a 3D image. The reference image generation system 620 can perform inverse processing compared to the processing performed by the dataset descriptor generation system 604. The reference image generation system 620 can use the dataset descriptor 618 to regenerate an artificial photomicrograph that forms an idealized, noise-free estimate of the image stack included in the image data 602.

[0103] Although the reference image generation system 620 includes a projection system 622, a particle placement system 624, a CTF application system 626, and a stack creation system 628, different combinations of the systems can be used (just as different combinations of the systems can be included in the dataset description generation system 604).

[0104] The informed compression subsystem 632 may receive a reference stack 630 generated by the reference image generation system 620, an image stack included in image data 602, and a dataset descriptor 618. The informed compression subsystem 632 may include an entropy encoder. For example, the entropy encoder may be a range variant of an arithmetic encoder, a Huffman encoder, or an asymmetric digital system (rANS) encoder, or another encoder. The informed compression subsystem 632 may generate compressed image data 634 representing image data 602. The informed compression subsystem 632 may use a Poisson distribution to generate the compressed image data 634. The dataset descriptor 618, in conjunction with the running reference image generation system 620, may provide a Poisson distribution, thus giving the probability that N electrons will fall on the corresponding pixels (e.g., by giving the expected electron count for each pixel).

[0105] The informed compression subsystem 632 can use the reference stack 630 as prior knowledge to generate compressed image data 634 to improve the compression ratio. Instead of encoding the real data without prior knowledge, the informed compression subsystem 632 can encode the difference between ideal data (e.g., represented by the reference stack 630) and real data (represented by the corresponding stack included in the image data 602). The informed compression subsystem 632 can use the pixel intensity of the reference stack 630 as the λ parameter of a Poisson distribution giving the probability that N electrons will land on the corresponding pixel.

[0106] The informed compression subsystem 632 may also output a dataset descriptor 618 and / or other information generated and / or used by the compression system 516 to generate compressed image data 634. For example, the informed compression subsystem 632 may output a dataset descriptor 618. The dataset descriptor 618 may then be used to decompress the compressed image data 634.

[0107] Compression system 516 provides additional compression compared to other compression techniques. Compression system 516 is suitable for compressing image data 602 acquired through image data acquisition methods that deliver images with large contrast differences, such as TEM imaging with a phase plate, dark-field imaging techniques, and STEM iDPC. In conventional TEM imaging, the average number of electrons per pixel is relatively large relative to the expected distribution of electrons per pixel, resulting in a marginal gain in the compression factor when encoding electron events or small dose fractions. Compression system 516 can reduce the memory space / resources used to represent image data 602 and the network resources used to transmit information representing image data 602. Compression system 516 can be lossless in retaining the information required to obtain the highest possible reconstruction resolution.

[0108] Figure 7 This is a schematic diagram illustrating an example of a Generative Adversarial Network (GAN) 700 used to train an image generation model 704 according to some implementation schemes. The image generation model 704 can be an example of an image encoding-decoding model. The image encoding-decoding model can be trained to define / generate encoded image data 706 based on received image data or a latent space 702 representing the image data (e.g., real encoded image data 708). GAN 700 is a deep learning architecture. GAN 700 trains two neural networks, namely the image generation model 704 and the discriminative model 710, to compete against each other.

[0109] Image generation model 704 can be used to generate encoded image data 706 based on latent space 702. In some embodiments, the generated encoded image data 706 can be generated from latent space 702, and latent space 702 can represent real encoded image data 708. In some embodiments, the generated encoded image data 706 can be generated by using a fixed-length random vector representing image data from latent space 702 as input, and generating encoded image data 706. The fixed-length random vector can represent image data. The fixed-length random vector can include image data with noise randomly added based on an arbitrary noise distribution. The noise distribution can be sampled to obtain a batch of noise samples (noise vectors) to be used as input image data for image generation model 704.

[0110] The generated encoded image data 706 can be represented in a spatiotemporal resolution representation. The image data can be input into an image generation model 704. The image generation model 704 may be able to merge damaged electron count frames into a single denoised image. The image generation model 704 can generate encoded image data 706 with reduced data compared to the input image data received by the image generation model 704, and can do so without sacrificing the spatiotemporal resolution of the image data represented by the latent space.

[0111] Image generation model 704 and discriminator model 710 are trained in an adversarial competition, where the image generation model attempts to generate encoded image data 706, and the discriminator model 710 attempts to predict whether the encoded image data is fake / generated encoded image data 706 or real encoded image data 708. Real encoded image data 708 may include image data 108. Real encoded image data 708 may be image data generated by an image acquisition system (e.g., image acquisition system 106).

[0112] The discriminant model 710 analyzes the real coded image data 708 independently of the image generation model 704 and distinguishes between attributes. The loss comparison system 712 uses the ground truth and the output of the discriminant model 710 to determine how to adjust the weights of the image generation model 704 and / or the discriminant model 710. The loss comparison system 712 may transmit a first weight adjustment signal 714 to the discriminant model 710 to adjust its weights for the purpose of reducing its error. The loss comparison system 712 may transmit a second weight adjustment signal 716 to the image generation model 704 to adjust its weights for the purpose of reducing its error.

[0113] After image generation model 704 modifies some data attributes in latent space 702 by adding noise (or random variations) to certain attributes, it can pass the generated encoded image data 706 to discriminative model 710. Discriminative model 710 calculates the probability that the generated encoded image data 706 belongs to the original dataset of the true encoded image data 708. Discriminative model 710 provides guidance to image generation model 704 by using a second weight adjustment signal 716 to perform weight adjustments on the parameters of image generation model 704 to reduce noise vector randomization in the next cycle. Image generation model 704 attempts to maximize the error probability of discriminative model 710, but discriminative model 710 uses a loss comparison system 712 to attempt to minimize the error probability, which transmits a first weight adjustment signal 714 to discriminative model 710 to update the weights used by discriminative model 710. During training iterations, the weights of both the image generation model 704 and the discriminator model 710 are changed based on the weight adjustment signal transmitted by the loss comparison system 712, and are made to continuously evolve and compete with each other. Training iterations can continue until the image generation model 704 and the discriminator model 710 reach an equilibrium state. In the equilibrium state, the discriminator model 710 may no longer recognize synthetic data. At this point, the training process can be completed.

[0114] In some implementations, GAN 700 is a basic GAN architecture in which the image generation model 704 generates data variations with little or no feedback from the discriminator model 710. In some implementations, GAN 700 is a conditional GAN ​​architecture in which the image generation model 704 and the discriminator model 710 receive additional information, such as class labels or some other form of conditional data. Those skilled in the art who benefit from this disclosure will recognize other GAN architectures (e.g., deep convolutional GANs, super-resolution GANs, etc.) that can be used to train the image generation model 704.

[0115] In some implementations, the image generation model 704 may be trained using other training techniques and may not require a GAN. For example, the image generation model 704 may include a diffusion model, a stable diffusion model, a variational encoder, and / or an autoregressive model. In some implementations, the image generation model 704 is trained to generate encoded image data 706 using at least a portion of an acquisition setup (e.g., acquisition setup 502).

[0116] Image generation model 704 can be trained to receive image data and / or image data embeddings in latent space 702, and output generated encoded image data 706. Image generation model 704 can be trained such that image data frames obtained from electrons interacting with the sample later than other image data frames obtained from electrons interacting with the sample are partially or completely omitted, and the volume of image data in the generated image data 708 is reduced compared to the volume of the image data. Image generation model 704 can use the image data to generate encoded image data 708 that occupies less space in memory than the image data itself, without compromising the spatiotemporal resolution of the generated encoded image data 708.

[0117] Figure 8 This is a schematic diagram depicting an example architecture of an image generation model 704 according to some implementation schemes. The image generation model 704 may have been based on... Figure 7 The described technique has been trained. The image generation model 704 may have been trained to output encoded image data 804 based on the received image data 802.

[0118] Image data 802 may include image data 108. Image data 802 may be received from an encoding selection system (e.g., encoding selection system 506) or an image acquisition system (e.g., image acquisition system 106). In some embodiments, image generation model 704 receives and uses one or more acquisition settings (e.g., acquisition settings 502) to generate encoded image data 804. In some embodiments, encoded image data 804 is a compressed representation of image data 802.

[0119] Image generation model 704 may include any combination of layers. In an example, image generation model 704 may include input layer 806, encoder block 808, temporal context module 810, decoder block 812, quantization layer 814, and / or output layer 816. In some embodiments, the architecture of image generation model 704 may include more or fewer (e.g., zero or more) layers per layer.

[0120] Input layer 806 accepts image data 802. Image data 802 may be represented as one or more image stacks (e.g., raw 3D slices from a cryoEM stack image acquisition system). Input layer 806 may apply Fourier transform to isolate frequency components.

[0121] Encoder block 808 may include 3D convolutional layers. Encoder block 808 may include swish activation and spatial dropout. Swish activation and spatial dropout may be performed after processing the 3D convolutional layers. Encoder block 808 may encode input layer data into a lower-dimensional latent space.

[0122] The temporal context module 810 can use one or more recurrent neural networks (RNNs) to capture the temporal correlation between consecutive frames, thereby enhancing the ability of the image generation model 704 to distinguish between noise and structured data.

[0123] Decoder block 812 can utilize 3D transposed convolution, swish activation, and / or batch normalization. Decoder block 812 can reconstruct a denoised image from the latent space.

[0124] The quantization layer 814 can quantize the denoised image to further reduce the file size (e.g., the memory space occupied) without sacrificing the critical structural details of the image data 802.

[0125] The output layer can produce encoded image data 804. Encoded image data 804 can represent high-quality and / or denoised image data that retains the basic structural information of image data 802. Encoded image data 804 can be part of an encoded image stack (e.g., an encoded 3D image slice).

[0126] Compared to other methods of representing image data generated by an image acquisition system, the architecture of image generation model 704 and / or the training techniques used to train image generation model 704 can lead to significant data reduction. Image generation model 704 can compress large image datasets, such as reducing a 2TB dataset to 200GB, achieving a 90% data reduction. Image generation model 704 can maintain the quality of image data 802 within encoded image data 804, and it can do so even when its size is reduced. Maintaining the quality of image data 802 within encoded image data 804 preserves key structural information included in image data 802 without degrading data quality.

[0127] Compared to other techniques for encoding image data 802 obtained from an image acquisition system, using image generation model 704 improves the speed and efficiency of generating encoded (e.g., compressed) image data 804. Image generation model 704 can reduce computational, network, and / or storage resources. For example, network and / or storage resources can be reduced by generating compressed encoded image data 804. In another example, the computational resources required to generate encoded image data 804 from image data 802 may be less than other methods of encoding image data 802 to achieve similar encoded image data. Image generation model 704 can be hosted on a local server and / or a remote server (e.g., in the cloud).

[0128] Processes 900, 1000, 1100, 1200, 1300, and 1400 depicted in the flowcharts and any other accompanying figures may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, implemented using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., stored on a memory device). The methods presented in processes 900, 1000, 1100, 1200, 1300, and 1400, and other accompanying figures, as well as those described herein, are intended to be illustrative and not limiting. Although processes 900, 1000, 1100, 1200, 1300, and 1400, and other accompanying figures depict various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In some alternative embodiments, the processes may be performed in some different order, or some steps may be performed in parallel. It should be understood that, in alternative embodiments, processes 900, 1000, 1100, 1200, 1300, and 1400, as well as the other accompanying drawings, may include more or fewer steps than those depicted in the respective drawings.

[0129] Figure 9 This is a flowchart of an example process 900 for training an image generation model (e.g., image generation model 704) using a generative adversarial network (GAN) (e.g., GAN 700) with an image generation model and a discriminative model, according to some implementation schemes. The image generation model and the discriminative model can be trained using a training dataset that includes multiple batches of training examples.

[0130] At position 902, the latent space, a first set of parameters, and a first set of weights can be initialized. The first set of parameters and weights can be used by the image generation model and can influence the output generated by the image generation model. Each weight can be a value assigned to the corresponding parameter. In one example, the image generation model can include a neural network. In one example, the image generation model can be an image encoder-decoder model. In one example, the image generation model can receive a fixed-length random vector representing image data as input and generate encoded image data. The fixed-length random vector can represent image data. The fixed-length random vector can include image data with noise randomly added based on an arbitrary noise distribution. The noise distribution can be sampled to obtain a batch of noise samples (noise vectors) to be used as input image data for the image generation model. The image generation model can be trained to generate encoded image data using the received image data. The encoded image data can be compressed image data. The encoded image data can preserve the quality of the image data in the encoded image data and can retain key structural information included in the image data.

[0131] At position 904, a second set of parameters and a second set of weights can be initialized. These parameters and weights are used by the discriminative model and can influence the output generated by it. Each weight can be a value assigned to the corresponding parameter. The discriminative model can be a neural network. Each training example used by the discriminative model can include real-world encoded image data.

[0132] At position 906, a fixed-length random vector can be input into the image generation model. The image generation model can then generate coded image data based on the fixed-length random vector. The image generation model can then output the generated coded image data.

[0133] At position 908, the generated encoded image data can be transferred to the discriminative model. One or more encoded image data sets can be generated before being sent to the discriminative model during the training phase (e.g., batch size). One or more encoded image data sets can be transferred before each encoded image data set is input into the discriminative model.

[0134] At 910, the discriminant model can be configured to generate either real or generated coded image data. The discriminant model can generate more than one classification for more than one real and / or generated coded image data (e.g., depending on the batch size). The classification indicates whether the discriminant model has classified the received coded image (real or generated) as either real or generated coded image data.

[0135] At position 912, the classification determined by the discriminant model can be transferred to the loss comparison system.

[0136] At point 914, the loss comparison system compares the classification output by the discriminative model with ground truth data to determine whether the classification is correct. The loss comparison system can use any suitable loss function. In one implementation, the image generation model and the discriminative model are trained adversarially using a loss function (e.g., mean squared error (MSE)).

[0137] Adversarial training refers to iteratively training a discriminative model and an image generation model sequentially. The discriminative model can be trained based on both generated coded image data received from the image generation model and real coded image data from training examples. The image generation model's loss may increase as the discriminative model improves to identify generated coded image data as such (i.e., as "forged" images). Therefore, the image generation model is trained to reduce this loss.

[0138] At position 916, the loss comparison system can transmit weight adjustment signals to the image generation model or the discriminant model based on the classification and ground truth output by the discriminant model.

[0139] The image generation model and the discriminative model can be trained by repeatedly updating a first set of weights corresponding to a first set of parameters and / or updating a second set of weights corresponding to a second set of parameters. In some implementations, one or more weights of the image generation model are adjusted when the discriminative model correctly classifies a generated coded image as a generated coded image (e.g., not a real coded image).

[0140] The image generation and discriminative models can be trained by repeatedly updating the weights of the parameters of both models until the loss function of the discriminative output is maximized relative to the discriminative model and minimized relative to the image generation model. Those skilled in the art who benefit from this disclosure will recognize that other techniques can be used to train machine learning models to generate encoded image data.

[0141] Steps 906 to 916 can be repeated based on the training duration and / or the loss function used by the loss comparison system. Once the image generation model is trained, it can be used to generate encoded image data from image data.

[0142] Figure 10 This is a flowchart of an example process 1000 for generating coded image data according to some implementation schemes. The coded image data can be generated using an image generation model (e.g., image generation model 704) during inference time.

[0143] At 1002, an image acquisition system (e.g., image acquisition system 106) is operable to image a sample (e.g., a radiation-sensitive sample). The image acquisition system can generate sample data based on data collected over a time period. The image acquisition system can be configured to operate according to one or more acquisition settings. The acquisition settings to be used for operation can be based on pre-configured acquisition settings, the sample, and / or user input. As an example, the acquisition settings may include a first dose rate and / or a first operating mode. The sample data collected during this time period can be image data and / or data that can be used to generate image data.

[0144] At point 1004, image data is generated based on sample data collected during that time period. In some implementations, such as when the sample data and image data are identical, this step may be omitted. The image data may include one or more electronically counted frames.

[0145] At position 1006, image data can be encoded. An encoding scheme can be used to encode the image data. In some embodiments, the encoding scheme may use a machine learning model (e.g., an image encoder-decoder model) to encode the image data. The machine learning model can compress the image data. In some embodiments, the machine learning model may have been trained to encode the image data (e.g., trained using a GAN training architecture). Encoding the image data reduces the network, memory, and / or processing resources used by systems that manipulate and / or utilize the image data. For example, image data may have an image resolution represented using a first number of bits (e.g., a first amount of space in memory) before encoding, and encoded image data may have the same image resolution but use a second number of bits less than the first number. In other words, the encoded image may occupy less space in memory than the image data, which reduces the network, memory, and / or processing resources used by systems that manipulate and / or utilize the encoded image data (compared to the image data before encoding).

[0146] In some embodiments, the encoding scheme used at step 1006 is one of a plurality of encoding schemes that can be used to encode image data. In some embodiments, the encoding scheme used at step 1006 is one of a plurality of encoding schemes used to encode a corresponding portion of the image data. For example, the encoding scheme performed during step 1006 may be performed for a first portion of the image data, and different encoding schemes may be performed for a second and different portion of the image data.

[0147] Figure 11 This is a flowchart of an example process 1100 for acquiring image data according to some embodiments. In some embodiments, process 1100 may be performed by an image acquisition system (e.g., image acquisition system 106). As described above, the image acquisition system may include an electron microscope. The image acquisition system can be used to obtain sample data and / or image data of an imaging sample (e.g., a radiation-sensitive sample).

[0148] At 1102, the image acquisition system can be configured according to a first acquisition setting. In some embodiments, the first acquisition setting may include a first dose rate and / or a first operating mode (e.g., a counting mode, an integrating mode). The first acquisition setting to be used for operation may be based on image acquisition settings received from another system, device, user interface, etc. The first acquisition setting to be used for operation may be based on predetermined acquisition settings that may have been previously configured. In some embodiments, the acquisition settings may additionally or alternatively include duration and / or magnification.

[0149] At 1104, the acquisition system can operate during a first time period according to a first acquisition setting. The first time period can be a portion of the total time period during which the image acquisition system images the sample. The length of the first time period can be determined based on user input, the sample to be imaged (the expected and / or actual radiation damage to the sample), the operating mode, the dose rate used, dose rate scheduling, and / or the total time for imaging the sample, user input, the purpose of imaging the sample, time-related functions (e.g., exponential functions, linear functions), etc.

[0150] The sample data acquired by the image acquisition system from the sample during the first time period may have different characteristics compared to a different second time period, which is part of the total time period for imaging the sample. For example, when the sample is radiation sensitive, it may degrade with the accumulation of dose (e.g., as more electrons interact with the sample). Due to radiation damage, electrons detected at the end of the acquisition period may carry less information about the sample than electrons detected at the beginning of the acquisition period. The sample data acquired by operating during the first time period can be used to generate image data.

[0151] At 1106, the image acquisition system can be configured according to a second acquisition setting. The image acquisition system can be configured according to the second acquisition setting after the first time period. The second acquisition setting may differ from the first acquisition setting. In some embodiments, the second acquisition setting may include a higher dose rate, a different operating mode, and / or other setting differences compared to the first acquisition setting. The second acquisition setting may be determined based on a time-related function, the purpose of imaging the sample, and / or the sample (e.g., the expected radiation damage characteristics of the sample).

[0152] At point 1108, the acquisition system can operate during a second time period according to the second acquisition settings. The second time period can be a portion of the total time period during which the image acquisition system images the sample. The length of the second time period can be determined based on user input, the sample to be imaged, the operating mode, the dose rate used, dose rate scheduling, and / or the total time for imagerization of the sample, etc. The second sample data obtained from the sample by the image acquisition system may have different characteristics than the sample data obtained from the sample by the image acquisition system during the first time period.

[0153] At 1110, the first sample data and the second sample data acquired by the image acquisition system during the first time period and the second time period, respectively, can be combined to generate image data. For simplicity, the illustrated process 1100 includes the first time period and the second time period. However, in some embodiments, more than two time periods are included in the total time period. In some embodiments, more than two acquisition settings are used within the total time period. In some embodiments, the image data is subsequently encoded (e.g., encoded as a spatiotemporal representation, encoded as a compressed representation).

[0154] As mentioned above, when a sample is radiation sensitive, it may degrade with cumulative dose (e.g., as more electrons interact with the sample). Due to radiation damage, electrons detected at the end of the acquisition period may carry less information about the sample than electrons detected at the beginning of the acquisition period. Additionally, events detected in high-dose-rate frames may carry less information about the imaged sample than events detected in low-dose-rate frames. Furthermore, image data acquired from the EMS200 is more accurate at lower dose rates than at higher dose rates, and the image quality (e.g., accuracy) of the image data may decrease with increasing dose rate. Considering the above, certain embodiments described herein (e.g., embodiments that adjust acquisition settings during the period of sample data acquisition) can increase the amount of information obtained from the sample within a given time period. Certain embodiments can maximize the amount of quantitative information obtained from the sample within a given time period.

[0155] Figure 12 This is a flowchart of an example process 1200 for encoding image data according to some embodiments. Process 1200 may be performed by an image processing system (e.g., image processing system 110). Process 1200 may be performed by a first encoding system (e.g., first encoding system 504). Process 1200 may be performed on image data generated by an image acquisition system (e.g., image acquisition system 106).

[0156] At 1202, image data can be received. The image data may have already been generated using an image acquisition system. The image data may have already been generated based on the image acquisition system using acquisition settings. The image data may represent image data acquired during the total time period during which the image acquisition system images the sample. The image data may represent image data acquired during a portion of the total time period during which the image acquisition system images the sample. The image data may represent image frames or electronic events.

[0157] At 1204, a dynamic representation for the image data is calculated. This dynamic representation can be calculated based at least on the acquisition settings used to acquire the image data. The dynamic representation can be an encoding scheme from a set of encoding schemes, which is dynamically selected based at least on the acquisition settings used to acquire the image data. In other words, the image data can be encoded using an encoding scheme based on the acquisition settings used to acquire the image data. Step 1204 can be performed by an encoding selection system (e.g., encoding selection system 506).

[0158] At 1206, the image data can be converted into a spatiotemporal resolution representation using the dynamic representation calculated during step 1204. The image data can be converted to spatiotemporal resolution using an EER coding scheme, a dose segmentation coding scheme, an image coding-decoding model, or another coding scheme capable of encoding the image data. The spatiotemporal resolution representation may be referred to as coded image data.

[0159] Compared to using different encoding schemes, these different encoding schemes can reduce the amount of memory occupied by the encoded image data and / or increase the spatiotemporal resolution of the encoded image data. By using the encoding scheme based on the acquisition settings, the amount of memory occupied by the encoded image data can be minimized while maximizing the spatiotemporal resolution of the encoded image data.

[0160] In some embodiments, process 1200 can be used for two separate portions of the image data. For example, process 1200 can be used for a first portion of the image data to encode the first portion using a first encoding scheme. The first portion of the image data may represent a low-dose image frame or an electronic event. Furthermore, process 1200 can be used for a second portion of the image data that differs from the first portion, and the second encoding scheme may differ from the first encoding scheme. The second portion of the image data may represent a low-dose image frame or an electronic event. In some embodiments, the first portion of the image data may have a different resolution (e.g., larger) than the second portion of the image data (e.g., due to the acquisition settings used to generate the image data, due to the cumulative dose received by the sample, etc.). The second encoding scheme may produce an encoded image independent of the first encoding scheme. In some embodiments, the second encoding scheme may use an encoding scheme with greater loss than the first encoding scheme to generate the second encoded portion.

[0161] Figure 13 This is a flowchart of an example process 1300 for compressing image data according to some implementation schemes. Compression can result in a reduction in the amount of memory space used to represent image data. Compression can be lossless in terms of retaining all the information needed to obtain the highest possible reconstructed resolution.

[0162] At 1302, image data can be received. The image data may include an image stack. The image stack may include data representing a set of images. The image data may have been received from an image acquisition system (e.g., image acquisition system 106), an encoding system (e.g., first encoding system 504), and / or a memory, etc.

[0163] At position 1304, the image data can be encoded. In some embodiments, the image data can be encoded to spatiotemporal resolution using an EER encoding scheme, a dose segmentation encoding scheme, an image code-decode model, or another encoding scheme capable of encoding the image data. In some embodiments, the image data can be encoded based on the acquisition settings used to acquire the image data.

[0164] At 1306, in some implementations, a dataset descriptor can be generated for the coded image data. The dataset descriptor can be generated using drift correction, contrast transfer function correction, particle picking, feature segmentation, particle classification, 3D reconstruction, artificial intelligence models, downsampled image data, and / or denoised image data.

[0165] A dataset descriptor can be generated for each image data and / or each image stack included in the image data. The dataset descriptor can be used to generate a probabilistic model for each spatiotemporal location within each image stack of the image data. In other words, the dataset descriptor can represent the most probable particle locations and other sample information based on information included in the image stacks of the image data. In some implementations, the more accurate the dataset descriptor, the better / greater the expected compression of the image data, when compressed and represented as a second encoded image data.

[0166] At 1308, in some embodiments, compressed image data may be generated. The compressed image data may be generated using the dataset descriptor and entropy encoder generated at step 1306. The compressed image data may include a stack of compressed images representing a set of one or more images. Generating compressed image data using the dataset descriptor may include generating a three-dimensional reference image using an encoded dataset descriptor and / or encoded images. The three-dimensional reference image may include a predetermined number of electrons at one or more points in spacetime. In some embodiments, the entropy encoder may be a range variant of an arithmetic encoder, a Huffman encoder, or an asymmetric digital system (rANS) encoder.

[0167] Figure 14 This is a flowchart of an example process 1400 for acquiring and encoding image data according to some implementation schemes.

[0168] Steps 1402, 1404, 1406, 1408, and 1410 can be performed in a manner similar to that described above for steps 1102, 1104, 1106, 1108, and 1110. In some embodiments, steps 1406 and 1408 are not performed, and image data is generated using only a single set of acquisition settings.

[0169] At 1412, the image data generated in step 1410 can be encoded. Any encoding scheme described herein can be used to encode the image data. For example, an EER encoding scheme, a dose segmentation encoding scheme, an image encoding-decoding model, or an image generation model (e.g., image generation model 704) can be used to encode the image data. As described above, in some embodiments, different portions of the image data can be encoded using an encoding scheme based on the acquisition settings used to obtain the sample data.

[0170] At 1414, the encoded image data can be further encoded by compressing the encoded image data. Example compression schemes may include using a compression system (e.g., compression system 516), an arithmetic encoder, a Huffman encoder, or a range variant of an asymmetric digital system (rANS) encoder, or another encoder.

[0171] At 1416, the image data generated in step 1410 may be encoded such that it is compressed. Example compression schemes may include the use of an image generation model (e.g., image generation model 704) and / or a compression system (e.g., compression system 516).

[0172] Figure 15 A schematic diagram of an example computer system capable of being used with systems and methods according to some embodiments of this disclosure is depicted.

[0173] Any of the computer systems mentioned herein (e.g., computing system 104, image acquisition system 106, etc.) may utilize any appropriate number of subsystems. Examples of such subsystems are... Figure 15 The computer system 1510 is shown in the diagram. In some embodiments, the computer system includes a single computer device, wherein a subsystem may be a component of the computer device. In other embodiments, the computer system may include multiple computer devices, each of which is a subsystem having internal components. The computer system may include desktop and laptop computers, tablet computers, mobile phones, and other mobile devices.

[0174] Figure 15The subsystems shown are interconnected via system bus 1575. Additional subsystems are shown, such as printer 1574, keyboard 1578, storage device 1579, monitor 1576 (e.g., display such as an LED) coupled to display adapter 1582, etc. Peripheral devices and input / output (I / O) devices coupled to I / O controller 1571 can be accessed via ports such as I / O ports 1577 (e.g., USB, FireWire). ® Any number of components known in the art, such as I / O port 1577 or external interface 1581 (e.g., Ethernet, Wi-Fi, etc.), can be used to connect computer system 1510 to a wide area network such as the Internet, a mouse input device, or a scanner. Interconnection via system bus 1575 allows central processing unit 1573 to communicate with each subsystem and control the execution of multiple instructions from system memory 1572 or storage device 1579 (e.g., a fixed hard disk or optical disk such as a hard disk drive) and the exchange of information between subsystems. System memory 1572 and / or storage device 1579 may be embodied in a computer-readable medium. Another subsystem is data collection device 1585, such as a camera, microphone, accelerometer, etc. Any data mentioned herein can be output from one component to another and can be output to a user.

[0175] A computer system may include multiple identical components or subsystems, for example, connected together via an external interface 1581, an internal interface, or via a removable storage device that allows connection to and removal from one component from another. In some embodiments, the computer system, subsystem, or apparatus may communicate over a network. In such instances, one computer may be considered a client, and another computer may be considered a server, wherein the client and the server may each be part of the same computer system. The client and the server may each include multiple systems, subsystems, or components.

[0176] Various aspects of the implementation scheme may be implemented using hardware circuitry (e.g., application-specific integrated circuits or field-programmable gate arrays) and / or in a modular or integrated manner using computer software stored in memory having a general programmable processor in the form of control logic, and thus the processor may include memory storing software instructions for configuring the hardware circuitry and an FPGA or ASIC having configuration instructions. As used herein, the processor may include a single-core processor, a multi-core processor on the same integrated chip, or multiple processing units on a single board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, those skilled in the art will recognize and appreciate other ways and / or methods of implementing the embodiments of this disclosure using hardware and combinations of hardware and software.

[0177] Any of the software components or functions described in this application (such as processes 900, 1000, 1100, 1200, 1300, and / or 1400) can be implemented as software code to be executed by a processor using any suitable computer language (such as, for example, Java, C, C++, C#, Objective-C, Swift) or scripting language (such as Perl or Python using, for example, conventional or object-oriented techniques). The software code can be stored as a series of instructions or commands on a computer-readable medium for storage and / or transmission. Suitable non-transitory computer-readable media may include random access memory (RAM), read-only memory (ROM), magnetic media (such as hard disk drives or floppy disks) or optical media (such as compact discs (CDs) or DVDs (Digital Universal Optical Discs) or Blu-ray discs), flash memory, etc. The computer-readable medium can be any combination of such devices. Furthermore, the order of operations can be rearranged. A process may terminate when its operations are completed, but may have additional steps not included in the figures. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a procedure corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0178] Such programs can also be encoded and transmitted using carrier signals suitable for transmission over wired, optical, and / or wireless networks (including the Internet) conforming to various protocols. In this way, computer-readable media can be created using data signals encoded with such programs. Computer-readable media encoded with program code can be packaged with a compatible device or provided separately from other devices (e.g., downloaded via the Internet). Any such computer-readable media can reside on or within a single computer product (e.g., a hard disk drive, CD, or an entire computer system) and can exist on or within different computer products within a system or network. The computer system may include a monitor, printer, or other suitable display for providing a user with any of the results mentioned herein.

[0179] Any method described herein can be executed, in whole or in part, by a computer system comprising one or more processors configured to perform the steps. Any operation performed by a processor (e.g., alignment, determination, comparison, calculation, measurement) can be performed in real time. The term "real time" can refer to a computational operation or process completed within a certain time constraint. The time constraint could be 1 minute, 1 hour, 1 day, or 7 days. Therefore, embodiments may involve a computer system configured to perform the steps of any method described herein, potentially having different components performing the respective steps or groups of steps. Although presented as numbered steps, the steps of the methods herein can be performed simultaneously or at different times or in different orders. Furthermore, portions of these steps can be used in conjunction with portions of other steps from other methods. Additionally, all or part of the steps can be optional. Furthermore, any step in any method of any method can be performed using modules, units, circuits, or other components of a system for performing these steps.

[0180] In the foregoing description, numerous specific details have been described with reference to embodiments of this disclosure, which may vary depending on the embodiments. Therefore, the description and drawings are to be considered illustrative rather than restrictive. The unique and exclusive indication of the scope of this disclosure, and the applicant's intention to define its scope, is the literal and equivalent scope of the set of claims published in the specific form that comprises these claims in this application, including any subsequent amendments. Specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of this disclosure.

[0181] As used herein, the terms “and,” “or,” and “and / or” can have a variety of meanings, which are also expected to depend at least in part on the context in which such terms are used. Generally, when used with a list of associations such as A, B, or C, “or” is intended to mean A, B, and C (used herein in an inclusive sense) and A, B, or C (used herein in an exclusive sense). Furthermore, as used herein, the term “one or more” can be used to describe any feature, structure, or property in the singular form, or to describe a combination of features, structures, or properties. However, it should be noted that this is merely an illustrative example, and the claimed subject matter is not limited to this example. Additionally, when used with a list of associations such as A, B, or C, the term “at least one of” can be interpreted to mean any combination of A, B, and / or C, such as A, B, C, AB, AC, BC, AA, AAB, ABC, AABBCCC, etc.

[0182] Throughout this specification, references to “an example,” “example,” “some examples,” or “exemplary embodiment” mean that a particular feature, structure, or characteristic described in connection with that feature and / or example may be included in at least one feature and / or example of the claimed subject matter. Therefore, the appearance of the terms “in an example,” “example,” “in some examples,” “in some embodiments,” or other similar terms throughout this specification does not necessarily refer to all the same feature, example, and / or limitation. Furthermore, a particular feature, structure, or characteristic may be combined in one or more examples and / or features.

[0183] In some specific implementations, operation or processing may involve the physical manipulation of physical quantities. Typically, but not always, such quantities may take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, or otherwise manipulated. Primarily for common usage, it has proven convenient to sometimes refer to such signals as bits, data, values, elements, symbols, characters, items, quantities, numbers, etc. However, it should be understood that all these or similar terms will be associated with the appropriate physical quantity and are merely convenient notations. Unless otherwise specifically stated, as will be apparent from the discussion herein, it should be understood that throughout this specification, the use of terms such as “processing,” “calculation,” “estimation,” “determination,” etc., refers to the actions or processes of a particular device (such as a dedicated computer, dedicated computing device, or similar dedicated electronic computing device). Therefore, in the context of this specification, a dedicated computer or similar dedicated electronic computing device is capable of manipulating or transforming signals that are typically represented as physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of the dedicated computer or similar dedicated electronic computing device.

[0184] In the foregoing specific embodiments, numerous specific details have been set forth to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods and apparatus that would be understood by a person of ordinary skill have not been described in detail so as not to obscure the claimed subject matter. Therefore, it is intended that the claimed subject matter be limited to the specific examples disclosed, but rather that it may also include all aspects falling within the scope of the appended claims and their equivalents.

[0185] The terms and expressions used herein are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof; rather, it should be recognized that various modifications are possible within the scope of the claims. Therefore, it should be understood that while this disclosure includes particular embodiments and optional features, modifications and alterations to the concepts disclosed herein can be made by those skilled in the art, and such modifications and alterations are considered to be within the scope of the appended claims.

[0186] When terms are used without explicit definition, it should be understood that unless the term has a special and / or specific meaning in the field of charged particle microscopy systems or other related fields, the term refers to its general meaning. The terms “approximately,” “identical,” “about,” “similar,” or “substantially” are used to indicate a deviation from the stated property or numerical value, where such deviation has little or no effect on the corresponding function, property, or attribute of the described structure. In the illustrated example where a dimensional parameter is described as “substantially equal to” or “approximately” to another dimensional parameter, the terms “substantially” or “approximately” are intended to reflect that the two dimensions being compared may be unequal within permissible limits (such as manufacturing tolerances). Similarly, where geometric parameters (such as alignment or angular orientation) are described as “approximately” perpendicular, “substantially” perpendicular, or “substantially” parallel, “approximately” equal, or “approximately” identical, the terms “approximately,” “substantially,” or “approximately” are intended to reflect that the alignment or angular orientation may differ from the precisely stated condition (e.g., not precisely perpendicular) within permissible limits. For dimensional values ​​(such as diameter, length, width, etc.), the term "about" can be understood as describing a deviation of up to ±10% from the stated value. For example, a dimension "about ten mm" can describe a dimension from nine to eleven mm. In this disclosure, a "subrange" refers to a range of values ​​between and / or including one of the stated ranges.

[0187] This description provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0188] Specific details are given in the description to provide a thorough understanding of the implementation scheme. However, it should be understood that the implementation scheme may be practiced without these specific details. For example, specific system components, systems, processes, and other elements of this disclosure may be shown schematically or omitted from the illustrations to avoid obscuring the implementation scheme with unnecessary details. In other cases, well-known circuits, processes, components, structures, and / or techniques may be shown without unnecessary details.

[0189] As used in this application and claims, the singular forms “a,” “an,” and “the” include the plural forms unless the context clearly indicates otherwise. Additionally, the term “comprising” means “including.” Furthermore, the term “coupled” does not exclude the existence of intermediate elements between coupled items.

[0190] The systems, apparatus, and methods described herein should not be construed as limiting in any way. Rather, this disclosure relates to all novel and non-obvious features and aspects of the various disclosed embodiments, whether individually or in various combinations and sub-combinations formed with each other. The disclosed systems, methods, and apparatus are not limited to any particular aspect or feature or combination thereof, nor are they required to possess any one or more particular advantages or problems solved. Any operational theory is provided for ease of interpretation, but the disclosed systems, methods, and apparatus are not limited to such operational theories.

[0191] Although some of the methods disclosed are described in a specific order for ease of presentation, it should be understood that this descriptive approach includes rearrangement unless the specific language described below requires a particular order. For example, operations described sequentially may be rearranged or performed simultaneously in some cases. Furthermore, for simplicity, the accompanying drawings may not show the various ways in which the disclosed systems, methods, and apparatus can be used in conjunction with other systems, methods, and apparatus. Additionally, this description sometimes uses terms such as “produce” and “provide” to describe the disclosed methods. These terms are high-level abstractions of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation and can be readily identified by one of skill in the art.

Claims

1. A computer-implemented method, the computer-implemented method comprising: The transmission electron microscope is configured according to a first acquisition setting, the first acquisition setting including at least one of a first dose rate or a first operating mode; During the first time period, the transmission electron microscope is operated according to the first acquisition settings to image the radiation-sensitive sample; After the first time period, the transmission electron microscope is configured according to a second acquisition setting, the second acquisition setting including at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode; During the second time period, the transmission electron microscope is operated according to the second acquisition settings to image the radiation-sensitive sample; as well as Image data is generated based on first data and second data collected during the first time period and the second time period, respectively.

2. The computer-implemented method according to claim 1, wherein the first operating mode includes a counting mode or an integration mode.

3. The computer-implemented method according to claim 1, wherein the first acquisition setting includes the first dose rate, and wherein the second acquisition setting includes the second dose rate.

4. The computer-implemented method according to claim 1, wherein the second dose rate is greater than the first dose rate.

5. The computer-implemented method of claim 1, wherein the second acquisition settings are determined based on at least one of the following: a time correlation function or the expected radiation damage characteristics of the radiation-sensitive sample.

6. The computer-implemented method according to claim 1, wherein the first time period is determined based on the radiation damage rate of the radiation-sensitive sample.

7. The computer-implemented method of claim 1, wherein the first time period is determined based on at least one of the first dose rate or the first operating mode.

8. The computer-implemented method of claim 1, wherein the first data is collected using a first frame rate, and the second data is collected using a second frame rate lower than the first frame rate.

9. The computer-implemented method according to claim 1, wherein at least one of the first acquisition settings or the second acquisition settings further includes at least one of the following: duration or magnification.

10. The computer-implemented method of claim 1, wherein the image data comprises a plurality of electronically counted frames, the method further comprising: The image data is encoded using an image encoder-decoder model trained to define the encoded image data.

11. The computer-implemented method of claim 10, wherein the image data includes an image resolution represented using a first number of bits, and the encoded image data includes the image resolution using a second number of bits less than the first number of bits.

12. A non-transitory computer-readable storage medium comprising instructions executable by one or more processors of a transmission electron microscope to induce an operation comprising: The transmission electron microscope is configured according to a first acquisition setting, the first acquisition setting including at least one of a first dose rate or a first operating mode; During the first time period, the transmission electron microscope is operated according to the first acquisition settings to image the radiation-sensitive sample; After the first time period, the transmission electron microscope is configured according to a second acquisition setting, the second acquisition setting including at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode; During the second time period, the transmission electron microscope is operated according to the second acquisition settings to image the radiation-sensitive sample; as well as Image data is generated based on first data and second data collected during the first time period and the second time period, respectively.

13. The non-transitory computer-readable storage medium of claim 12, wherein the operation further comprises: A first dynamic representation for representing the first data is calculated based at least on the first acquisition settings; The first data is converted into a first spatiotemporal resolution representation using the first dynamic representation; The second dynamic representation for representing the second data is calculated based at least on the second acquisition settings; The second data is converted into a second spatiotemporal resolution representation using the second dynamic representation; The first spatiotemporal resolution is used to generate first compressed image data using the first compression scheme. as well as The second spatiotemporal resolution is used to generate the second compressed image data using the second compression scheme.

14. The non-transitory computer-readable storage medium of claim 13, wherein the first spatiotemporal resolution represents a first resolution greater than the second spatiotemporal resolution represents a second resolution.

15. The non-transitory computer-readable storage medium of claim 13, wherein the first compression scheme is determined based on the first dynamic representation, and the second compression scheme is determined based on the second dynamic representation.

16. The non-transitory computer-readable storage medium of claim 13, wherein the second compression scheme has greater loss than the first compression scheme.

17. The non-transitory computer-readable storage medium of claim 13, wherein converting the first data into the first spatiotemporal resolution representation using the first dynamic representation comprises: Use at least one of the following to perform encoding: electronic event representation, dose segmentation scheme, or entropy encoder and probabilistic model; and The process of converting the second data into the second spatiotemporal resolution representation using the second dynamic representation includes: The second encoding is performed using at least one of the following: the electronic event representation, the dose segmentation scheme, or the entropy encoder and the probability model.

18. The non-transitory computer-readable storage medium of claim 13, wherein the first data represents a low-dose image frame or electronic event, and the second data represents a low-dose image frame or electronic event.

19. The non-transitory computer-readable storage medium of claim 13, wherein an instruction for at least one of the first dynamic representation or the second dynamic representation is received from a graphical user interface of a user equipment.

20. The non-transitory computer-readable storage medium of claim 13, wherein the operation further comprises: An encoded image dataset is generated by encoding second image data comprising at least one of the following: the first spatiotemporal resolution representation or the second spatiotemporal resolution representation; Use the coded image dataset and at least one of the following to generate a coded dataset descriptor: drift correction, contrast transfer function correction, particle selection, feature segmentation, particle classification, 3D reconstruction, or artificial intelligence model; A 3D reference image is generated using the coded dataset descriptor and the coded image dataset; as well as The three-dimensional reference image, the encoded dataset descriptor, and the entropy encoder are used to generate compressed second image data.

21. A transmission electron microscope, the transmission electron microscope comprising: One or more memories, wherein the one or more memories store instructions; and One or more processors, configured to execute the instructions to cause the transmission electron microscope to perform operations, the operations including: The transmission electron microscope is configured according to a first acquisition setting, the first acquisition setting including at least one of a first dose rate or a first operating mode; During the first time period, the transmission electron microscope is operated according to the first acquisition settings to image the radiation-sensitive sample; After the first time period, the transmission electron microscope is configured according to a second acquisition setting, the second acquisition setting including at least one of a second dose rate different from the first dose rate or a second operating mode different from the first operating mode; During the second time period, the transmission electron microscope is operated according to the second acquisition settings to image the radiation-sensitive sample; and Image data is generated based on first data and second data collected during the first time period and the second time period, respectively.

22. The transmission electron microscope according to claim 21, further comprising: An encoded image dataset is generated by encoding the image data; The encoded image dataset is used to generate an encoded dataset descriptor using at least one of the following: drift correction, contrast transfer function correction, particle selection, feature segmentation, particle classification, 3D reconstruction, artificial intelligence model, downsampled image data, or denoised image data; and Compressed image data is generated using the encoded dataset descriptor and entropy encoder.

23. The transmission electron microscope of claim 22, wherein using the coded dataset descriptor to generate the compressed image data comprises: The coded dataset descriptor and the coded image dataset are used to generate a 3D reference image.

24. The transmission electron microscope of claim 23, wherein the three-dimensional reference image comprises a predetermined number of electrons at one or more points in spacetime.

25. The transmission electron microscope of claim 22, wherein the image data includes an image stack, the image stack including data representing a plurality of images.

Images