Method for Restoring Resolution of Specimen-Fitted X-Ray Microscopy

Abstract

Improved image quality can be obtained by utilizing nonlinear neural network-based feature recovery across multiple imaging modalities. Input imaging data is acquired, including first and second imaging data acquired using different imaging modalities. The first imaging data can have a lower image quality and a larger FOV than the second imaging data. The first and second imaging data can be aligned, and the aligned regions can be used to train a neural network to minimize differences between the second imaging data and output data processed from overlapping portions of the first imaging data. Once trained, the neural network can be used to generate an improved image quality output from the entirety of the first imaging data.

Description

[Technical field]

[0001] The present disclosure relates generally to x-ray imaging, and more particularly to improving image quality in x-ray microscopy. [Background technology]

[0002] X-ray microscopic imaging is a field of imaging used to obtain imaging data for many different types of specimens across many different use cases. X-ray microscopic imaging is used in biology (e.g., imaging biological materials, soft tissues, etc.), materials science (e.g., imaging the internal microstructure of materials), manufacturing (e.g., non-destructive imaging of internal components), and many other fields. Individual images (e.g., projection images) can be obtained by directing radiation from an X-ray source through the specimen to a detector. Multiple projection images can be obtained for a single specimen by rotating the direction of travel of the X-ray radiation relative to the specimen (e.g., rotating the X-ray source and detector relative to the specimen). Often, the obtained imaging data (e.g., including multiple projection images) is used to generate a three-dimensional reconstructed volume of the imaged specimen, such as through the use of computed tomography (CT).

[0003] While X-ray microscopy offers many advantages, one challenge is that of scale. Often, the resolution required to image basic structures is obtained at the expense of the field of view required to image the entire specimen. Furthermore, high-resolution tomographic acquisition tends to be extremely slow, especially inside large specimens. While high-resolution detectors exist, they are typically significantly less sensitive than low-resolution detectors, especially to high-energy X-rays. High-resolution sources are also typically significantly less powerful than low-resolution sources. When performing interior tomography in large specimens, a relatively large amount of material outside the field of view can be projected into the volume, effectively adding noise and artifacts. Non-imaged regions of the specimen can act as an X-ray filter, biasing the X-ray spectrum to higher energies, which can be particularly problematic when interior tomography is performed in large specimens. Because high-resolution detectors are less sensitive to high-energy X-rays, the resulting projection images can be noisy. Even when multiple imaging parameters and / or devices are capable of producing images at the same resolution, specific imaging parameters and / or devices may result in images of different quality, such as images with better or worse sharpness, noise, artifacts, point spread functions, etc.

[0004] Currently, when imaging of a large field of view is required, there are only two options. The first option is to rely on low image quality acquisition techniques, which can achieve a large field of view, but at the expense of image quality (e.g., image resolution, image sharpness, image noise, etc.), which may make important features indistinguishable or may otherwise be undesirable. If a high-quality image of a large field of view is required, a large-area synthetic projection image can be created and then reconstructed from two projection images offset with respect to each other. However, such acquisition modes are typically very slow and unreliable.

[0005] Techniques are needed that allow imaging with both high image quality and a large field of view. Techniques are also needed for obtaining such imaging at relatively high speeds and in a reliable manner. Summary of the Invention [Means for solving the problem]

[0006] The term "embodiments" and similar terms are intended to broadly refer to the entire subject matter of this disclosure and the claims below. Statements containing these terms should be understood not to limit the subject matter described herein or to limit the meaning or scope of the claims below. The embodiments of the disclosure covered herein are defined by the claims below and are supplemented by this summary. This summary is a high-level overview of various aspects of the disclosure and introduces some of the summaries further described in the Detailed Description section below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood with reference to any suitable portion, any or all of the drawings and claims of the entire specification of this disclosure.

[0007] An embodiment of the present disclosure includes a method including receiving first imaging data associated with a specimen. The first imaging data is acquired using a first imaging modality. The method further includes receiving second imaging data associated with the specimen. The second imaging data is acquired using a second imaging modality different from the first imaging modality. The method further includes registering the first imaging data and the second imaging data. The method further includes identifying a portion of the first imaging data that overlaps with the second imaging data. The method further includes training a neural network based at least in part on the portion of the first imaging data and the second imaging data. The method further includes generating output imaging data by applying the trained neural network to the first imaging data, the output imaging data including an area outside the area overlapping with the second imaging data. The trained neural network may now also be applied to other imaging data (e.g., third imaging data) acquired in a manner similar to the first imaging data acquisition and images of a similar specimen class.

[0008] In some cases, the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality. In some cases, the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first and second objective measurements, or (ii) the same as the second objective measurement. In some cases, the first imaging data is acquired at a first resolution and a first field of view, the second imaging data is acquired at a second resolution that is higher than the first resolution and a second field of view that is smaller than the first field of view, and the output imaging data has a resolution higher than the first resolution. In some cases, the first imaging modality and the second imaging modality share a common primary imaging modality. In some cases, the first imaging modality includes a first primary imaging modality that is x-ray microscopy, and the second imaging modality includes a second primary imaging modality that is not x-ray microscopy.

[0009] In some cases, the method further includes automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data. In some cases, the method further includes resampling at least a portion of the first imaging data onto a grid of the second imaging data. In some cases, training the neural network includes providing a portion of the first imaging data associated with the second imaging data to the neural network to generate output data, and tuning the neural network to reduce differences between the generated output data and the second imaging data. In some cases, the neural network is based on a u-net neural network. In some cases, generating the output imaging data includes applying a zoom transformation to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data, and applying the transformed first imaging data to the trained neural network to generate the output imaging data. In some cases, generating the output imaging data includes segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk, applying each of the plurality of chunks of the first imaging data to a trained neural network to generate a corresponding chunk of output imaging data, and stitching together the overlapping regions of the corresponding chunks of the output imaging data. In some cases, generating the output imaging data includes receiving a target region selection, the target region selection indicating a target region of the first imaging data, tagging the target region with metadata indicating a location of the target region, applying the first imaging data associated with the target region to the trained neural network to generate a corresponding target region of the output imaging data, tagging the corresponding target region of the output imaging data with metadata associated with the target region, and locating the corresponding target region of the output imaging data based at least in part on the metadata.

[0010] In some cases, the method further includes identifying a plurality of change regions associated with the specimen, and the steps of training the neural network and generating the output are associated with a first change region and are performed using a first subset of the first imaging data and the second imaging data associated with the first change region. In some cases, the method further includes training a second neural network based at least in part on a second subset of the first imaging data and the second imaging data associated with the second change region, generating second output imaging data associated with the second change region by applying the trained second neural network to the subset of the first imaging data associated with the second change region, and stitching the output imaging data and the second output imaging data together.

[0011] In some cases, the method further includes generating a two-dimensional image or a three-dimensional volume using the output imaging data. In some cases, the generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data, such that at least one region of the two-dimensional image or the three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or the three-dimensional volume is based on the first imaging data.

[0012] Embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause a data processing apparatus to perform any of the methods described above.

[0013] Embodiments of the present disclosure include a system comprising one or more data processors and a non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform any of the methods described above.

[0014] This specification makes reference to the accompanying drawings, in which the use of like reference numbers in different drawings is intended to describe the same or similar elements. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram illustrating an imaging data processing system in accordance with certain aspects of the present disclosure. [Diagram 2] 1 is a schematic diagram depicting imaging data used to train a neural network to obtain high-resolution output imaging data in accordance with certain aspects of the present disclosure. [Diagram 3] 1 is a flowchart illustrating a process for processing imaging data according to certain aspects of the present disclosure. [Figure 4] 1 is a schematic diagram of a graphical user interface depicting first and second imaging data in accordance with certain aspects of the present disclosure. [Diagram 5] 1 is a schematic diagram of a graphical user interface for registering first and second imaging data in accordance with certain aspects of the present disclosure. [Figure 6] 1 is a diagram of a portion of a low resolution image in accordance with certain aspects of the present disclosure. [Figure 7] 1 is a diagram of a portion of a high resolution image in accordance with certain aspects of the present disclosure. [Figure 8] 1 is a diagram of a portion of an image with improved image quality in accordance with certain aspects of the present disclosure. [Figure 9] 1 is a combined diagram illustrating a three-dimensional volume and a corresponding two-dimensional image from below for selecting regions for image quality improvement processing according to certain aspects of the present disclosure. [Figure 10] 1 is a combined diagram illustrating a three-dimensional volume and corresponding two-dimensional image from below, including a region of imaging data with improved image quality, in accordance with certain aspects of the present disclosure. [Figure 11] FIG. 1 is a block diagram illustrating an example system architecture for carrying out certain features and processes of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Certain aspects and features of the present disclosure relate to techniques for achieving high image quality (e.g., high resolution and large field of view (FOV)) X-ray microscope imaging data by leveraging nonlinear neural network-based feature recovery across multiple imaging modalities (e.g., across multiple primary imaging modalities or across multiple sub-imaging modalities within a single primary imaging modality). Input imaging data including first imaging data and second imaging data are obtained using a first imaging modality and a second imaging modality, respectively. For example, the first imaging data may be of lower resolution and have a larger FOV than the second imaging data, and the second imaging data may be of higher resolution and have a smaller FOV. The first imaging data and the second imaging data can be aligned, and the aligned regions can be used to train a neural network to minimize differences between the second imaging data and output data processed from the first imaging data. Once trained, the neural network can be used to generate a high image quality output from all of the first imaging data, resulting in a high image quality image covering the entire large FOV of imaging data. Any other data generated similarly to the first image data and utilizing similar sample classes can be improved by this trained neural network.

[0017] As used herein, the term imaging modality includes primary imaging modalities (e.g., different basic techniques for acquiring imaging data, such as X-ray microscopy, optical microscopy, electron microscopy, etc.) and lower level imaging modalities (e.g., different types of imaging techniques within a single primary imaging modality, such as different scan parameters of a single device or different devices operating under the same basic imaging technique). For example, under the primary imaging modality of X-ray imaging, some possible lower level imaging modalities can include X-ray imaging using different resolution detectors (X-ray imaging using different size scintillators, X-ray imaging using different scan recipes, etc.). Thus, imaging techniques that are of two different modalities may be imaging techniques that are of two different primary imaging modalities, or imaging techniques that are of the same primary imaging modality but of two different lower level imaging modalities.

[0018] During an X-ray microscopy procedure, radiation is emitted from one or more emitters (X-ray sources) and directed to one or more detectors. A specimen (e.g., an object to be analyzed) disposed between the emitter and the detector may affect the amount of radiation received by the detector, such as by absorbing, reflecting, or otherwise affecting the radiation incident on and / or passing through the specimen. The resulting information collected by the detector may be known as data or imaging data. As used herein, the term scan or scanning may optionally refer to the acquisition of imaging data during the movement of the specimen relative to the emitter and / or detector. As used herein, the term computed tomography (CT) is intended to include the generation of a three-dimensional reconstructed volume of the specimen using X-ray imaging data. The three-dimensional reconstructed volume may be a data set showing the three-dimensional structure or a three-dimensional image (e.g., a three-dimensional image composed of voxels) of the specimen. X-ray imaging is generally non-destructive to the specimen.

[0019] Electron microscopy (EM) is a different primary imaging modality that uses a beam of accelerated electrons directed at a specimen to image the specimen. Different lower level imaging modalities of electron microscopy include techniques such as transmission electron microscopy (TEM), scanning electron microscopy (SEM), and reflection electron microscopy (REM). EM imaging typically collects individual slices of a specimen or exposed surfaces of a specimen. In some cases, multiple slices of EM images can be combined to achieve a three-dimensional reconstructed volume. While EM imaging can achieve very high quality images in some circumstances, obtaining such images is often very time consuming and expensive. Additionally, EM imaging is often destructive to the specimen.

[0020] Optical microscopy (OM) is a different primary imaging modality that uses light (e.g., visible or non-visible light) reflected from or passing through a specimen to image the specimen. Different lower level imaging modalities of optical microscopy include techniques such as stereomicroscopy, confocal microscopy, bright field microscopy, and dark field microscopy. OM is commonly used to inspect the surface of a specimen. Image quality and the time / cost consumed to acquire an image may vary depending on the type of specimen imaged and the type of optical microscopy technique employed. Generally, optical microscopy is non-destructive to the specimen.

[0021] Certain aspects and features of the present disclosure can be used to generate improved imaging data, which can result in improved two-dimensional images, improved three-dimensional volumes (e.g., improved CT reconstructed volumes), or other improved images or volumes (e.g., laminography reconstructions) depending on the imaging data.

[0022] Deep neural networks (DNNs) can be used to generate or improve imaging data or reconstructed volumes, but must be properly trained. DNNs trained on training subjects may not perform well on different specimens, especially if the subject and specimen are different types of specimens (e.g., different materials or significantly different structural assemblies) or if the imaging geometry and / or x-ray source settings are significantly different from those used in acquiring the training data.

[0023] Certain aspects and features of the present disclosure use a modified U-net, although any other suitable neural network architecture (e.g., residual neural network, SqueezeNet neural network, etc.) can be used. U-net is a modified version of a convolutional neural network (CNN) that includes an upsampling operator to achieve an output with higher resolution than that achieved by a traditional CNN (e.g., an output with the same or higher resolution than the input resolution). In some cases, U-net can be modified to work in both 2 and 2.5 dimensions. For example, a 2.5D U-net can use anisotropic slices through data of shape AxBx5 as input to generate results of shape (A-186)x(B-186)x3 as output.

[0024] Certain aspects and features of the present disclosure can provide various image quality improvements such as super-resolution (e.g., improved resolution beyond that of the input data), improved sharpness, improved point spread function, improved signal-to-noise ratio, reduced artifacts (e.g., especially when imaging artifacts are uncorrelated between the first input imaging data and the second input imaging data). For example, detector noise that is uncorrelated between the first input imaging data and the second input imaging data can result in a trained neural network having a denoising effect, resulting in denoised output imaging data. Additional effects or imaging defects that can be further improved include limited angle or sparse angle sampling artifacts, beam hardening artifacts, and the like. In some cases, the second imaging data can be acquired in a manner (e.g., a particular lower order imaging modality or a primary imaging modality) selected to avoid or minimize certain artifacts or other problems known to occur in the first imaging data.

[0025] According to certain aspects of the present disclosure, input imaging data is acquired, including first imaging data acquired according to a first imaging modality (e.g., a first primary imaging modality and a first lower level imaging modality) and second imaging data acquired according to a second imaging modality (e.g., a second primary imaging modality different from the first primary imaging modality, or a second lower level imaging modality of the first primary imaging modality different from the first lower level imaging modality). The terms "first" and "second" with respect to the imaging data are intended to distinguish between the two sets of imaging data and do not necessarily indicate the order in which the imaging data are acquired. In some cases, the second imaging data is acquired before the first imaging data. The first imaging data may be of a larger field of view than the second imaging data and / or relate to a larger area or volume of the specimen than the second imaging data. However, the second imaging data may be of a different imaging modality capable of obtaining higher quality imaging (e.g., higher resolution, improved sharpness, improved point spread function, etc.) In some cases, both the first imaging data and the second imaging data may be of the same sized field of view, such as when low resolution imaging data is achieved by having the detector integrate the signal over a larger area to obtain low resolution, low noise, rapidly acquired data.

[0026] The first and second imaging data may be acquired from the same specimen, with the specimen supported by a single specimen holder. The specimen holder may define known registration points to facilitate registration and / or alignment of the first and second imaging data. In some cases, the specimen holder may be incorporated into the same instrument used to acquire both the first and second imaging data, which may further facilitate registration and / or alignment.

[0027] In one example, acquiring input imaging data can include acquiring low-resolution and high-resolution imaging data that are automatically registered, such as through a Scout-and-Zoom process as available in ZEISS® X-ray microscopy instruments, such as the Xradia Versa® instrument. Low-resolution, large FOV imaging can be performed first to allow the system to define a particular region of interest (e.g., as defined by a user or as automatically determined by the system using the low-resolution imaging data) and capture the corresponding high-resolution imaging data. The low-resolution and high-resolution imaging data can be aligned using the relative stage positions (e.g., XYZ stage positions) of the two tomograms, since the coordinate system of the calibrated stage positioning is known for the instrument. In some cases, sub-pixel alignment can be performed using automatic 3, 6, or 9 degree-of-freedom sub-pixel alignment, or the like.

[0028] The input imaging data, including the first imaging data, the second imaging data, or both the first imaging data and the second imaging data, may be pre-processed. Pre-processing the imaging data may include registering, aligning, resampling, and / or otherwise modifying the imaging data. The pre-processing steps may be performed in any suitable order. In some cases, registering or aligning the imaging data may include coordinate transformation, warping removal, and / or distortion removal.

[0029] Registration and / or alignment can be performed between the first and second imaging data. When possible, registration and / or alignment can be facilitated with known registration points or fiducial markers established by the use of a common specimen holder. The common specimen holder can define a known coordinate system in which the specimen is positioned and located to enable registration and / or alignment based on the known coordinate system. As long as the specimen remains held by the specimen holder, the specimen can be positioned and located in the same position and location within that coordinate system, even though the imaging data is acquired using different imaging modalities and even different equipment. In some cases, additional registration and / or alignment techniques as disclosed herein can be used in addition to the use of a common specimen holder. When the use of a common specimen holder is not possible or practical, such as when image data is acquired using multiple modalities (e.g., certain multiple primary imaging modalities, such as certain x-ray and electron microscopy imaging techniques) that preclude the use of a common specimen holder, registration and / or alignment can be performed using any other suitable technique.

[0030] In some cases, a machine learning model can be trained to facilitate the registration of imaging data acquired using one imaging modality to imaging data acquired using a different imaging modality. Such machine learning registration techniques can be trained using unsupervised training data of a plurality of different specimens acquired using different pairs of imaging modalities. Thus, the trained machine learning model can be stored for the registration and / or alignment of imaging data for different combinations of modalities (e.g., a first model for registering transmission electron microscopy imaging data to X-ray imaging data, a second model for registering scanning electron microscopy imaging data to X-ray imaging data, and a third model for registering optical microscopy imaging data to X-ray imaging data). In some cases, each trained machine learning model can be dedicated to a direction of registration, so that two separate models can be trained for a pair of imaging modalities (e.g., one model for registering imaging data from a first modality to that of a second modality, and a second model for registering imaging data from a second modality to that of the first modality). Although registration and / or alignment are described herein as being part of the pre-processing of the imaging data, this is not necessarily always the case, and registration and / or alignment may occur before or after any pre-processing of the imaging data. Registration and / or alignment may be performed before or after any other pre-processing techniques.

[0031] In some cases, one or both of the first and second imaging data can be resampled to match or better match each other. For example, the spatially registered low-resolution imaging data can be resampled (e.g., via interpolation) onto a grid of the high-resolution imaging data. Resampling the low-resolution imaging data to a higher resolution of the high-resolution imaging data, or at least to an intermediate resolution between the lower resolution of the low-resolution imaging data and the higher resolution of the high-resolution imaging data, can facilitate training of the neural network used to generate the output imaging data, as well as potentially facilitating certain pre-processing techniques (e.g., registration and / or alignment).

[0032] Furthermore, in some cases, one or both of the first imaging data and the second imaging data can be modified or otherwise adjusted to look more similar to each other. For example, when using an X-ray primary imaging modality for the first imaging data and an electron microscopy primary imaging modality for the second imaging data, the second imaging data will look different from the first imaging data. Thus, the second imaging data can be processed to look more similar to general X-ray imaging data, such as by applying grayscale, contrast, and other adjustments to the second imaging data. Thus, one or both of the first imaging data and the second imaging data can be specifically pre-processed in a manner that smoothes one of the first imaging data and the second imaging data in the training of a neural network used to generate output imaging data from the other of the first imaging data and the second imaging data.

[0033] In an example having multiple primary imaging modalities, acquiring input imaging data may include acquiring first imaging data using an X-ray microscopy instrument, and then acquiring second imaging data using an electron microscopy instrument. The first imaging data may be acquired for a large area of ​​the specimen (e.g., the entire specimen), while the second imaging data may be acquired for a specific smaller area of ​​the specimen, such as an area having a significant feature (e.g., an inclusion, an edge, a material change, etc.). Due to the nature of electron microscopy, the second imaging data will be acquired second, since it will involve cutting the specimen. The first imaging data may be X-ray imaging data acquired at a first voxel size, such as 300 nm. However, the second imaging data may be electron microscopy imaging data acquired at a second voxel size, such as 30 nm (e.g., a 30 nm slice). In some cases, the second imaging data can be pre-processed by downsampling to the first voxel size or an intermediate voxel size (e.g., a voxel size between the first and second voxel sizes, such as 200 nm), and then modifications can be applied to the downsampled imaging data designed to make the imaging data mimic the imaging modality of the first imaging data (e.g., designed to mimic typical x-ray imaging data). These modifications can be pre-programmed or dynamically determined. The second imaging data can be registered and / or aligned with the first imaging data (e.g., before, during, or after other pre-processing steps). The pre-processed and registered / aligned second imaging data, along with any overlapping first imaging data, can then be used to train a neural network to transform the first imaging data into output imaging data using the second imaging data as a target result. In this way, the first imaging data can be used as a lower image quality (e.g., low resolution) input, and the pre-processed and registered / aligned second imaging data can be used as an output where a higher image quality (e.g., high resolution) is required.Once the neural network is trained, the trained neural network can be used to process all of the first imaging data to generate output imaging data of high image quality for the entire large area of ​​the specimen from which the first imaging data was obtained (e.g., the entire specimen). However, in some cases, the trained neural network can be used to process third imaging data to generate output imaging data, where the third imaging data may be similarly acquired data of a similar specimen class as that of the first imaging data.

[0034] In an example having multiple sub-imaging modalities within the same primary imaging modality, the first imaging data can be acquired using an X-ray imager having a first scintillator, and the second imaging data can be acquired using an X-ray imager having a second scintillator different from the first scintillator. The first and second scintillators may differ in various ways, such as having different thicknesses. For example, a thinner scintillator may result in higher resolution imaging data, while a thicker scintillator may result in an image with a brighter image or a larger dynamic range (e.g., ratio between the highest and lowest intensity). A neural network may be trained using a portion of the first imaging data and the second imaging data, and then the neural network may be applied to the complete first imaging data to generate output imaging data that is improved over the other or both of the first imaging data and the second imaging data (e.g., having a higher resolution than possible with a thicker scintillator, and / or a brighter image or a larger dynamic range than possible with a thinner scintillator). Other differences in the scintillator can be used in conjunction with or in place of the scintillator thickness. In other examples, other differences in the x-ray detector and / or emitter, including user adjustable settings, can be differences between the acquisition of the first and second imaging data. In some cases, these and other differences can be made while using the same instrument (e.g., the same x-ray imager, such as an x-ray imager having multiple detectors), but that is not necessarily always the case and multiple instruments can be used.

[0035] Regardless of how the input imaging data is obtained, the neural network can be trained via any suitable technique using the input imaging data (e.g., first imaging data of low image quality and second imaging data of high image quality). Two correlated arrays (e.g., low-resolution array and high-resolution array based on low-resolution imaging data and high-resolution imaging data, respectively) can be exported as an input target pair in a format suitable for the back-end of the neural network. The neural network can now be trained to map the first imaging data (e.g., low-resolution imaging data) to the second imaging data (e.g., high-resolution imaging data). Through training, the neural network optimizes the creation of features exhibited in high image quality based on input exhibited in low image quality.

[0036] In some cases where a portion of the first imaging data is pre-processed to train a neural network (e.g., transforming the low-resolution first imaging data, such as via a zoom transformation, such that the data is interpolated such that the new pixel grid has an effective pixel size equal to the high-resolution second imaging data), the entire first imaging data may be similarly pre-processed before being applied to the trained neural network. In some cases, the first imaging data may be pre-processed in a different manner, such that it is otherwise reconstructed onto a grid having a desired pixel size for a desired output resolution that differs from the resolution of the second imaging data. For example, in some cases, significant upsampling (e.g., 4x upsampling, such as by dividing each voxel into 64 pixels) prior to application of the trained neural network may help improve (e.g., reduce or eliminate) some aliasing artifacts. However, in some cases, the first imaging data may be provided to the trained neural network without any resampling and / or other pre-processing acts.

[0037] In some cases, the entire zoomed 3D data set can be stitched together and fed through a trained neural network. The stitching can be particularly useful when computational hardware is limited. Furthermore, changes in shape between the input and output imaging data must be taken into account. In some cases, overlapping regions can be performed when stitching the data together to best maintain a continuous image, allowing overlapping regions to be stitched together.

[0038] In some cases, such as when faster processing is desired, it may be desirable to apply the trained neural network to a smaller than entire input data set. In such cases, specific regions of the input imaging data can be selected for image quality improvement (e.g., resolution restoration). These target regions can be tagged (e.g., with metadata) using region targeting software or the like to indicate the 3D location of the region. The imaging data from these target regions can be processed through the neural network to output improved image quality imaging data for those target regions. The metadata from the target regions can then be applied to the corresponding improved image quality imaging data to enable the super-resolution imaging data to be reinserted into the input imaging data at a suitable location in the 3D visualization space.

[0039] Although the trained neural network may not be able to create new information, it may be able to correlate high image quality (e.g., high resolution) structures in ways that those structures are represented in lower image quality (e.g., lower resolution). For example, edges at low resolution will reliably encode for edges at high resolution. In theory, there are many ways to recover low resolution information from high resolution information. If more than one of the many theoretical degenerate options for high resolution to low resolution transformations actually exists in the high resolution dataset and is not effectively predicted by (correlated with) other local or non-local features in the low resolution dataset, the network may not be able to learn the transformation. Such a state may be considered a "degenerate" state (as opposed to the universal "theoretical degenerate" state that is always occupied by relatively encoding two images of the same structure at different resolutions). Although the level of effective degeneracy depends on the sample and resolution, the ratio of effective degeneracy to theoretical degeneracy may always be less than or equal to 1, and is often small enough that the image quality improvement techniques disclosed herein (e.g., super-resolution techniques) provide significant benefits and image quality improvements. The ratio of effective degeneracy to theoretical degeneracy may be a technique for evaluating the effectiveness of a trained neural network.

[0040] Aspects and features of the present disclosure relate to using a neural network trained on first imaging data to generate output imaging data of higher image quality than the first imaging data. Image quality can be assessed in many ways, depending on the desired outcome and / or the particular use case. In some cases, the aspect of image quality to improve may be user selectable, dynamically determined based on available input data, or preset. Improved image quality may include measurable objective improvements (e.g., higher resolution, lower point spread function, etc.) or subjective improvements (e.g., removed or reduced artifacts, more noticeable corners and edges, etc.). In an exemplary case, the image quality may be resolution. In this case, the first imaging data may be of lower resolution than the second imaging data and the output imaging data. In another exemplary case, the image quality may be point spread function. In this case, the first imaging data may have a point spread function that is larger than that of the second imaging data and the output imaging data. Other examples of aspects of image quality that can be improved include sharpness, noise (signal-to-noise ratio), dynamic range, contrast, distortion, artifacts, and the like.

[0041] These illustrative examples are provided to introduce the reader to the general subject matter discussed herein and are not intended to limit the scope of the disclosed concepts. In the following sections, various additional features and examples are described with reference to the drawings, in which like numbers indicate like elements and directional descriptions are used to describe the exemplary embodiments, but as such should not be used to limit the present disclosure. Elements included in the description herein may not be drawn to scale.

[0042] FIG. 1 is a schematic diagram illustrating an imaging data processing system 100 according to certain aspects of the present disclosure. The imaging data processing system 100 (e.g., a control system) can include a first imaging data source 102 and a second imaging data source 103 that provide imaging data to a processing module 104. Each of the first imaging data source 102 and the second imaging data source 103 can be any suitable source of imaging data, such as an imager (e.g., an imaging machine such as an X-ray microscope, a CT scanner, an electron microscope, an optical microscope, etc.), a database of imaging data, a local memory that stores imaging data, a removable memory that stores imaging data, etc. Certain aspects and features of the present disclosure are particularly useful when the first imaging data source 102 and the second imaging data source 103 are imagers, such as an X-ray microscope. In some cases, the first imaging data source 102 and the second imaging data source 103 are the same device (e.g., a single X-ray imager that can generate both the first imaging data and the second imaging data). In some cases, the first imaging data source 102 and the second imaging data source 103 are different instruments of the same primary imaging modality (e.g., a first and second x-ray imager). In some cases, the first imaging data source 102 and the second imaging data source 103 are different instruments of different primary imaging modalities (e.g., an x-ray imager and an electron microscope).

[0043] The first imaging data source 102 and the second imaging data source 103 can provide the first imaging data and the second imaging data, respectively. The first imaging data and the second imaging data can be acquired for a single specimen 107, but using different imaging modalities, such as different primary imaging modalities (e.g., different types of imaging equipment) or different sub-imaging modalities (e.g., different settings or types of devices within the same primary imaging modality). In some cases, the specimen 107 can be held in the specimen holder 105 during acquisition of both the first imaging data and the second imaging data. In such cases, known characteristics of the specimen holder 105 can be used to facilitate registration and / or alignment of the first and second imaging data. When the first and second imaging data are acquired by the first imaging data source 102 and the second imaging data source 103, respectively, the specimen 107 can remain in the same position and arrangement relative to the specimen holder 105 during both acquisitions. In some cases, this may involve keeping the specimen holder 105 in the same instrument to collect both the first and second imaging data, while in some cases, this may involve moving the specimen holder 105 between the first and second instruments without repositioning the specimen 107 relative to the specimen holder 105.

[0044] In some cases, the first and second imaging data may be acquired for multiple specimens held by a multiple specimen holder, in which case the first and second imaging data for each specimen may be used independently of the first and second imaging data from any other specimen, although this is not necessarily always the case.

[0045] The processing module 104 can process imaging data from the first imaging data source 102 and the second imaging data source 103. In some cases, the processing module 104 can control one or both of the first imaging data source 102 and the second imaging data source 103. The processing module 104 can use the imaging data to train a neural network, such as an artificial neural network (ANN) (e.g., a deep neural network (DNN), a convolutional neural network (CNN), etc.), and / or use such a trained neural network to process the imaging data into super-resolution imaging data.

[0046] In some cases, the processing module 104 can also perform reconstruction of the imaging data (e.g., input imaging data or output imaging data, such as imaging data with improved image quality) to generate a reconstruction volume (e.g., convert a set of acquired projections into a three-dimensional reconstruction volume). In some cases, the processing module 104 can access a pre-trained neural network from the neural network storage 108, which can be applied as is or can be further trained. The pre-trained neural network can be trained based on previous scans acquired from the first imaging data source 102 and / or the second imaging data source 103, such as previous scans of the same or similar object. In some cases, the pre-trained neural network can be a neural network generated using a federated learning technique, where multiple trained neural networks can be collected and combined to generate a joint neural network that is distributed as a pre-trained neural network. In such cases, each pre-trained neural network can be associated with the same category of specimens and / or the same or similar acquisition parameters, and the pre-trained neural network accessed by the processing module 104 can be accessed based on the set of categories and / or acquisition parameters provided. When a pre-trained neural network is used, it can be further trained as disclosed herein with respect to initially training the neural network, such as using low-resolution input imaging data and high-resolution input imaging data. In some cases, the neural network storage 108 is stored on a memory that is locally or remotely accessible (e.g., accessible via a network such as a local area network, a wide area network, a cloud network, or the Internet).

[0047] The pre-trained neural network may be selected based on user input, such as by the user selecting an individual pre-trained neural network from a list of available neural networks, or by the user providing information about the specimen to be scanned or the combination of imaging modalities to be used, which can be used to suggest and / or identify one or more pre-trained neural networks suitable for the specimen. For example, specimens that share the same or similar category information (through associative learning or otherwise) used for the pre-trained neural network may be able to successfully use that pre-trained neural network. The category information may be information that can be used to identify the category to which the specimen belongs. Any suitable type of category can be used, such as "minerals", "calcium-rich specimens", "skin tissue", etc.

[0048] In some cases, the neural network trained using the processing module 104 can be stored in the neural network storage 108, optionally together with additional information related to the specimen and / or scan. Additional information related to the specimen can include identification information (e.g., a unique identifier or description), category information (e.g., an indication as to which category the specimen belongs to), imaging data or reconstructed volumes of the specimen generated using the neural network, reconstructed volumes of imaging data or standardized sets of imaging data (e.g., of general standardized specimens or of specific standardized specimens selected to be similar to the specimen, such as having the same category). Additional information related to the scan can include information related to any imaging modalities used, such as the type of primary imaging modality, the type of lower level imaging modality, imager identification information (e.g., model number of an x-ray imager, model number or type of x-ray source and / or detector, etc.), scan recipe information (e.g., information about one or more parameters used in scanning or imaging the specimen), etc. Any information stored in the neural network storage device 108 related to the pre-trained neural networks can be used to assist in selecting a pre-trained neural network to use when processing imaging data from a new specimen.

[0049] An input / output module 106 can be coupled to the processing module 104 to receive user input and / or provide output to a user. Any suitable input / output device can be implemented in the input / output module 106, such as a keyboard, a mouse, a display (e.g., a computer monitor), a touch screen, a light emitting diode (LED) or other light source, a button, etc. The processing module 104 can present input imaging data, generated output imaging data (e.g., image quality improved imaging data), a reconstructed volume, etc. via the input / output module 106. In some cases, the input / output module 106 can serve to store the imaging data, the reconstructed volume, and / or the trained neural network (e.g., on a local memory, a removable memory, or a network accessible memory). In some cases, the neural network trained using the processing module 104 can be stored in association with the generated output imaging data or the reconstructed volume.

[0050] In some cases, any of the first imaging data source 102, the second imaging data source 103, the processing module 104, the input / output module 106, and the neural network storage 108 can be incorporated in one or more housings in any suitable combination. Any combination of one or more of the first imaging data source 102, the second imaging data source 103, the processing module 104, the input / output module 106, and the neural network storage 108 can be implemented locally (e.g., on the same device as one another or on devices coupled by a bus or local area network) or remotely (e.g., via a wide area network, the Internet, or a cloud network). In one example, the processing module 104 can be implemented on a user's laptop computer, the first imaging data source 102 and / or the second imaging data source 103 can be implemented on a cloud-based health record database (e.g., one or more servers accessible via the Internet), and the neural network storage 108 can be implemented on a separate cloud-based analysis database (e.g., one or more servers accessible via the Internet). In some cases, the neural network storage 108 is not used.

[0051] In another example, the processing module 104 can be incorporated into the first imaging data source 102 and / or the second imaging data source 103, such as a computer for processing imaging data that is also used to control an X-ray microscope or other imager. In another example, the processing module 104 can be incorporated into a separate computer that accesses imaging data provided from a separate CT scanner or X-ray microscope via a network-accessible database.

[0052] 2 is a schematic diagram 200 depicting input imaging data 210 used to train a neural network to obtain high image quality output imaging data 218, in accordance with certain aspects of the present disclosure. The elements of diagram 200 can be implemented using the imaging data processing system 100 of FIG.

[0053] The input imaging data 210 may include both the first imaging data 212 and the second imaging data 214. The first imaging data 212 may have a lower image quality (e.g., lower resolution) and a larger field of view than the second imaging data 214. The terms "first" and "second" as used in reference to the first imaging data 212 and the second imaging data 214 do not necessarily indicate that one is acquired before the other, but rather that one is acquired in a different manner than the other (e.g., using a different imaging modality, equipment, and / or settings to achieve a different field of view and / or different image quality). The second imaging data 214 may overlap a portion of the first imaging data 212, such that the first imaging data 212 and the second imaging data 214 may be registered or aligned with respect to one another.

[0054] The first imaging data 212 (e.g., a portion of the first imaging data 212 that overlaps with the second imaging data 214) and the second imaging data 214 can be used to train the neural network 216. Training, which can be performed as disclosed in further detail herein, can enable the trained neural network 216 to take imaging data of lower image quality (e.g., the first imaging data 212) as input and generate output data having similar image quality as imaging data of higher image quality (e.g., the second imaging data 214). Thus, features in the imaging data of lower image quality that are associated with certain structures found in the imaging data of higher image quality can be recovered by the trained neural network to output imaging data of improved image quality from the imaging data of lower image quality.

[0055] Once the trained neural network is established (e.g., trained), the same first imaging data 212 (e.g., more than the complete first imaging data 212 or at least a portion of the first imaging data 212 used to train the neural network) may be used as an input to the trained neural network 216 to generate output imaging data 218. The output imaging data 218 may have the same field of view as the first imaging data 212. However, due to processing within the trained neural network 216, the output imaging data 218 may have improved image quality with respect to the first imaging data 212 (e.g., higher resolution than that of the first imaging data 212, such as a resolution that is the same or similar to the second imaging data 214). The output imaging data 218 may thus be considered imaging data of higher image quality (e.g., super-resolution imaging data in the context of improved resolution).

[0056] In this way, by training a neural network (e.g., trained neural network 216) using the first imaging data 212 and the second imaging data 214 of higher image quality related to the first imaging data 212 (e.g., at least partially overlapping with the first imaging data 212 and of the same specimen as the first imaging data 212), a set of output imaging data 218 of higher image quality (e.g., compared to the image quality of the first imaging data 212) can be generated for a given lower image quality set of first imaging data 212.

[0057] For illustrative purposes, the first imaging data 212 is depicted with a large rectangle sparsely filled with dots, representing a relatively low image quality and a relatively large FOV. In contrast, the second imaging data 214 is depicted with a smaller rectangle filled with dense squares, representing a relatively high image quality and a relatively small FOV. A larger dashed rectangle surrounding the smaller rectangle represents the FOV of the first imaging data 212, and thus indicates that at least a portion of the second imaging data 214 is disposed within (e.g., overlaps) at least a portion of the FOV of the first imaging data 212. The output imaging data 218 is depicted with a large rectangle that is the same size as that of the first imaging data 212, and thus has the same FOV as the first imaging data 212, but the dense squares of the second imaging data 214, and thus is of the same or nearly the same image quality as the second imaging data 214.

[0058] 3 is a flowchart illustrating a process 300 for processing imaging data according to certain aspects of the present disclosure. Process 300 can operate on any suitable imaging data processing system, such as imaging data processing system 100 of FIG.

[0059] At block 302, input imaging data is acquired. The input imaging data may be acquired from a single specimen (e.g., a single object or target). In some cases, all input imaging data may be acquired while the specimen is fixed relative to the specimen holder, regardless of movement of the specimen holder within and / or between one or more instruments. The input imaging data may be acquired using a single imager (e.g., a CT scanner), but this is not always the case, such as when multiple primary imaging modalities are used. In some cases, acquiring the input imaging data may include receiving the input imaging data from a storage device or memory. In some cases, acquiring the input imaging data may include using an imager to acquire imaging data, such as by performing one or more scans of the specimen.

[0060] Acquiring input imaging data in block 302 may include acquiring first imaging data in block 304 and acquiring second imaging data in block 306. Acquiring first imaging data in block 304 and acquiring second imaging data in block 306 may be performed using different imaging modalities, such as different primary imaging modalities (e.g., different types of imagers, such as an X-ray imager and an electron microscope), or different sub-imaging modalities (e.g., different sub-types of the same type of imager, such as an X-ray imager with low resolution and large FOV and an X-ray imager with high resolution and small FOV). Acquiring first imaging data in block 304 and acquiring second imaging data in block 306 may include using the same imager or different imagers. The step of acquiring the first imaging data in block 304 may be performed before the step of acquiring the second imaging data in block 306, such as when the first imaging data is used to determine a portion of the specimen to be imaged in the second imaging data, or when the second imaging technique is destructive, which is not necessarily always the case. The step of acquiring the first imaging data in block 304 may include acquiring imaging data of low image quality (e.g., lower image quality than the second imaging data) with a large FOV (e.g., larger FOV than the second imaging data). The step of acquiring the second imaging data in block 306 may include acquiring imaging data of high image quality (e.g., higher image quality than the first imaging data) with a small FOV (e.g., smaller FOV than the first imaging data). The second imaging data acquired in block 306 may at least partially overlap with at least a portion of the first imaging data acquired in block 304.

[0061] At block 308, the input imaging data may be pre-processed. The pre-processing at block 308 may include preparing the input imaging data for use in training a neural network. In some cases, the pre-processing at block 308 may include pre-processing regions of the first and second imaging data that overlap. In some cases, the overlapping regions include a portion of the first imaging data and all of the second imaging data, but this is not necessarily always the case.

[0062] In some cases, the pre-processing step in block 308 may include aligning and / or registering the first imaging data (e.g., all and / or overlapping portions of the first imaging data) with the second imaging data in block 310. Aligning and / or registering the first imaging data with the second imaging data in block 310 may include aligning the first imaging data with the second imaging data in any suitable manner, such as using relative specimen holder coordinates (e.g., XYZ stage position relative to the specimen stage) of the two tomograms or using other known registration points. In some cases, aligning and / or registering the first imaging data with the second imaging data may include performing a 3, 6, or 9 degree of freedom sub-pixel alignment on the first imaging data and / or the second imaging data. In some cases, aligning and / or registering the first imaging data and the second imaging data may include applying a machine learning model trained for registration to one or both of the first imaging data and the second imaging data to output registration information and / or aligned imaging data. Such a machine learning model may be trained using an unsupervised technique using acquiring the first imaging data in block 304 and training imaging data acquired using the imaging modality used to acquire the second imaging data in block 306. In some cases, aligning and / or registering the first imaging data and the second imaging data may include determining an imaging modality associated with the input imaging data and selecting a suitable machine learning model for aligning and / or registering the imaging modalities (e.g., when the first imaging data is acquired using an X-ray imager and the second imaging data is acquired using a transmission electron microscope, a particular machine learning model trained in training imaging data from the X-ray imager and the transmission electron microscope may be selected for use).Aligning and / or registering the first and second imaging data in block 310 may result in the first and second imaging data being spatially aligned.

[0063] In some cases, the pre-processing step in block 308 may include resampling the first imaging data and / or the second imaging data in block 312. Resampling the first imaging data may include resampling the first imaging data onto a grid of the second imaging data, such as via interpolation or other suitable resampling techniques. In some cases, resampling the data may include resampling the first imaging data and / or the second imaging data onto a common grid (e.g., to a common target resolution). In some cases, resampling the data in block 312 may include resampling (e.g., downsampling) the second imaging data to a target resolution that is the resolution of the first imaging data or that is between the resolution of the first imaging data and the second imaging data. In some cases, the resampling step in block 312 may be performed on spatially aligned first and second imaging data, but this is not always the case.

[0064] In some cases, the pre-processing step in block 308 may further include adjusting the first imaging data and / or the second imaging data in block 332. Adjusting the first imaging data and / or the second imaging data may include applying one or more adjustments to the first imaging data, the second imaging data, or the first imaging data and the second imaging data. These adjustments may be preset, manually selected, or automatically selected based on the imaging modalities associated with the first imaging data and the second imaging data. The adjustments may be designed to cause the first imaging data and / or the second imaging data to normalize the tolerance between the two imaging modalities. Such adjustments may be based solely on the imaging modality used, or may be further based on specimen information, such as specimen category information. In some cases, applying the adjustments in block 308 may further include applying one or more adjustments to the second imaging data to cause the second imaging data to appear more similar to how typical imaging data acquired using an imaging modality associated with the first imaging data would appear. For example, if the first imaging data is acquired using an X-ray imager and the second imaging data is acquired using a transmission electron microscope, the second imaging data may have color content, contrast, and / or other image characteristics adjusted such that the second imaging data appears as if it were imaging data from an X-ray imager.

[0065] In some cases, the pre-processing step in block 308 can include outputting the first and second imaging data (e.g., after applying spatial alignment, registration, resampling, and / or rectification) in a manner suitable for training a neural network. In some cases, the pre-processing step in block 308 can include outputting two correlated arrays (e.g., one for the first imaging data and one for the second imaging data) as input target pairs in a format suitable for the neural network. The neural network can then utilize the input target pairs during training as the neural network is trained to generate targets (or target-like outputs) from the inputs.

[0066] In some cases, the pre-processing step at block 308 may include outputting only overlapping portions of the first imaging data and the second imaging data. In other words, if the second imaging data falls entirely within the FOV of the first imaging data, the pre-processing step at block 308 may crop the first imaging data such that the first imaging data provided to the neural network for training is the same size as the second imaging data provided to the neural network for training.

[0067] At block 314, the neural network is trained using the pre-processed input imaging data. In some cases, training the neural network can include accessing an initial or pre-trained neural network and further training such neural network based on the pre-processed input imaging data. In some cases, training the neural network can include training a CNN or, more specifically, a u-net, or, more specifically, a u-net modified to work with 2D and 2.5D imaging data.

[0068] Training the neural network in block 314 may include providing the pre-processed first imaging data to the neural network to generate an output. The output may be compared to the pre-processed second imaging data. Operating parameters of the neural network may then be adjusted to minimize the difference between the output and the pre-processed second imaging data. This process may be repeated until the neural network is sufficiently trained to achieve a desired output. Training the neural network in block 314 may include training the neural network to map low image quality (e.g., low resolution) imaging data (e.g., the pre-processed first imaging data) to high image quality (e.g., high resolution) imaging data (e.g., the pre-processed second imaging data). The resulting model (e.g., the trained neural network) may optimize the creation of features when represented at a high image quality based on the input when represented at a low image quality. The result of block 314 is a neural network (eg, a model) that is trained using low image quality and high image quality imaging data of overlapping regions (eg, overlapping areas and / or volumes) of the same specimen.

[0069] In block 318, output imaging data may be generated from the first imaging data (e.g., the first imaging data acquired in block 304) and the trained neural network. In some cases, the first imaging data may be pre-processed, such as in the pre-processing step in block 308, but this is not always the case. In some cases, the step of generating output imaging data in block 308 may further use second imaging data (e.g., the second imaging data obtained in block 306), but this is not always the case.

[0070] In some cases, the first imaging data may be optionally transformed in block 316 before being applied to the trained neural network in block 318. Transforming the first imaging data in block 316 may include providing a zoom transformation to the original first imaging data such that the imaging data is interpolated such that a new pixel grid has an effective pixel size equal to the high image quality (e.g., high resolution) data set (e.g., the second imaging data).

[0071] Generating the output imaging data in block 318 may include generating the output imaging data in aggregating. In such cases, the first imaging data may be divided into chunks in block 320. In some cases, the first imaging data is divided such that at least a portion of each chunk overlaps with at least a portion of an adjacent chunk. Each of the individual chunks may then be applied to a neural network in block 322 to generate a corresponding output chunk. Each of the output chunks may then be stitched together in block 324. The overlapping regions of the chunks as divided in block 320 may facilitate stitching together the corresponding output chunks in block 324. In some cases, the stitching step may be particularly useful in enabling the process 300 to be performed on hardware that may not otherwise have sufficient computational strength to handle larger amounts of input data.

[0072] In some cases, some but not all of the first imaging data may be processed with super-resolution. In such cases, generating output imaging data at block 318 may include receiving a selection identifying a selected region at block 326. The selection may indicate a selected region to be processed with super-resolution. At block 326, the selected region may be tagged with metadata indicating a location of the selected region. At block 328, output imaging data may be generated based on the selected region (e.g., the first imaging data associated with the selected region). The output imaging data may then be positioned using the metadata from block 326. In some cases, positioning the output imaging data may include tagging the output imaging data with the metadata from block 326. In this way, metadata associated with the selected region is automatically used to tag the resulting output imaging data based on the selected region. The metadata may then be used to position the output imaging data, such as within a two-dimensional image or a three-dimensional volume.

[0073] In some cases, generating the output imaging data at block 318 may further include presenting the output imaging data, such as in the form of a two-dimensional image or a three-dimensional volume.

[0074] In some cases, some or all of the blocks of process 300 (e.g., blocks 308, 314, 318) can be repeated multiple times for a single specimen. In some cases, determinations can be made of the variation and / or uniformity within the specimen, which can be used to identify multiple change regions within the specimen. For example, transitions identified within the specimen can be used to segment the specimen into multiple change regions. Determining the change regions can be automatic (e.g., by analysis of the first imaging data and / or the second imaging data) or manual (e.g., by user input). When performed automatically, the determined change regions may, but are not always, presented to a user for approval and / or modification. The determination of the variation and / or uniformity can be performed at any suitable stage, such as during acquisition of the input imaging data in block 302, after pre-processing of the input imaging data in block 308, or at any other suitable time. The determination of the variation and / or uniformity can include identifying multiple change regions to be processed separately. Each change region can define one or more closed areas or volumes associated with portions of the input imaging data that share similar characteristics. In some cases, the change regions can be defined to include at least some overlap with adjacent change regions to facilitate subsequent stitching together of the output imaging data. A neural network can be trained for each of the change regions using the imaging data (e.g., the first imaging data and the second imaging data) associated with that particular homogeneous region. Output imaging data can then be generated for each of the homogeneous regions. This output imaging data for each of the homogeneous regions can be stitched together to take advantage of any available overlap between adjacent homogeneous regions in some cases.

[0075] In one example, a specimen including a flat, gear-shaped object of a first material centrally positioned within a cube-shaped second material can be separated into three change regions: (i) a first circular change region for the first material within the gear-shaped object, (ii) a second annular change region for the boundary region between the first and second materials, and (iii) a third change region for the remainder of the specimen made from the second material. Thus, a neural network trained on the first change region can be specifically trained to improve image quality in regions of the specimen made from the first material, a neural network trained on the third change region can be specifically trained to improve image quality in regions of the specimen made from the second material, and a neural network trained on the second change region can be specifically trained to improve image quality in regions including the boundary between the first and second materials and / or regions including a projection of the gear shape (e.g., with an increased number of edges).

[0076] In some cases, at least one of the change regions can be excluded from the creation of a new neural network, in which case an existing neural network (e.g., a specimen category specific neural network without additional specimen specific training) can be leveraged to improve the image quality of the imaging data associated with that change region, or the imaging data associated with that change region can be used without image quality improvement.

[0077] 4 is a schematic diagram of a graphical user interface (GUI) 400 depicting first and second imaging data in accordance with certain aspects of the present disclosure. The GUI 400 may be the result of an image acquisition program designed to acquire both first imaging data 412 and second imaging data 414. The GUI 400 may operate on any suitable imaging data processing system, such as the imaging data processing system 100 of FIG.

[0078] The GUI 400 may include various controls and options for acquiring imaging data, storing imaging data, and / or modifying imaging data. As depicted, the GUI 400 shows first imaging data 412 and second imaging data 414. The first imaging data 412 is relatively low resolution imaging data (e.g., low image quality imaging data). While certain structures and features may be visible to some extent in the first imaging data 412, higher resolution imaging of such structures may be desirable. In some cases, the second imaging data 414 may be acquired for an area within the FOV of the first imaging data 412. Generally, the second imaging data 414 may be acquired specifically for an area within the FOV of the first imaging data 412 that includes a structure of interest, but this is not necessarily always the case. In some cases, the boundaries used to acquire the second imaging data 414 may be defined to maximize neural network training efficiency, such as by selecting a region of the FOV of the first imaging data 412 that is likely to have the greatest variety of different structures and / or features of interest (e.g., not necessarily the greatest number of structures and / or features of interest).

[0079] In some cases, the GUI 400 may provide tools to facilitate registering and / or aligning the first imaging data 412 and the second imaging data 414.

[0080] 5 is a schematic diagram of a graphical user interface (GUI) 500 for registering first and second imaging data according to certain aspects of the present disclosure. The GUI 500 can operate on any suitable imaging data processing system, such as the imaging data processing system 100 of FIG 1. In some cases, the GUI 500 can be the GUI 400 of FIG 4 after selection of a dataset registration tool.

[0081] The dataset registration tool depicted in GUI 500 allows a user to register and / or align the first imaging data and the second imaging data. Depending on the registration option 552 selected, various registration and / or alignment techniques can be used. As depicted in FIG. 5, the registration option 552 selected by the user indicates that the registration should be performed using translation (e.g., with an initial step of 378.43 micrometers and a minimum step of 3.81 micrometers), linear interpolation, and mutual information based registration (instead of sum of squared errors based registration). Other registration / alignment techniques can be used.

[0082] 6 is a diagram of a portion of a low-resolution image 600 according to certain aspects of the present disclosure. The low-resolution image 600 may be a two-dimensional image created from the first imaging data (e.g., low image quality imaging data). The low-resolution image 600 may show features of the imaged object, but without the same clarity provided by the high-resolution image. The low-resolution image 600 may have an input pixel size of about 10 micrometers.

[0083] 7 is a portion of a high-resolution image 700 according to certain aspects of the disclosure. The high-resolution image 700 may be second imaging data (e.g., high image quality imaging data) acquired from the same region of the same object as the low-resolution image 600 of FIG. 6. The high-resolution image 700 may show more detail than the low-resolution image 600 of FIG. 6, but potentially with more noise or other artifacts. The high-resolution image 700 may have a pixel size of approximately 2.5 micrometers.

[0084] FIG. 8 is a portion of an improved image quality (e.g., super-resolution) image 800 according to certain aspects of the present disclosure. The improved image quality image 800 can be generated by passing the low-resolution image 600 of FIG. 6 through a neural network that has been trained as disclosed herein using the low-resolution image 600 of FIG. 6 and the high-resolution image 700 of FIG. 7. The improved image quality image 800 can have the same field of view as the low-resolution image 600, but with improved resolution (e.g., the same or similar resolution as that of the high-resolution image 700 of FIG. 7). In some cases, the improved image quality image 800 can further exhibit reduced artifacts, such as reduced noise, as compared to the low-resolution image 600 of FIG. 6 and / or the high-resolution image 700 of FIG. 7. The improved image quality image 800 can achieve an effective pixel size at or near that of the high-resolution image 700 of FIG. 7, such as at or about 2.5 micrometers.

[0085] 9 is a combined diagram illustrating a three-dimensional volume 900 and a corresponding two-dimensional image 901 from below to select regions for image quality improvement processing according to certain aspects of the present disclosure. The three-dimensional volume 900 and the corresponding two-dimensional image 901 may be based on first imaging data (e.g., first imaging data 212 of FIG. 2). The selection process depicted in FIG. 9 can be performed using any suitable system, such as imaging data processing system 100 of FIG. 1.

[0086] Regions 962 and 964 can be selected for image quality improvement processing. Selection of regions 962, 964 can be made using any suitable technique, such as using graphical (e.g., mouse and cursor) and / or text-based (e.g., via keyboard) techniques. Each region 962, 964 can represent a corresponding region of the first imaging data, which can be labeled with metadata indicating where such corresponding region is located within the three-dimensional volume 900 and / or the corresponding two-dimensional image 901. After selection of any desired region (e.g., regions 962, 964), the user can instruct the system to initiate image quality improvement processing accordingly.

[0087] 10 is a combination diagram depicting a three-dimensional volume 1000 and a corresponding two-dimensional image 1001 from below including a region of improved image quality imaging data in accordance with certain aspects of the present disclosure. The three-dimensional volume 1000 and the corresponding two-dimensional image 1001 may be based on a combination of first imaging data (e.g., first imaging data 212 of FIG. 2) and improved image quality imaging data (e.g., output imaging data 218 of FIG. 2). The combination image depicted in FIG. 10 may be generated and / or presented using any suitable system, such as imaging data processing system 100 of FIG. 1.

[0088] The combined image of FIG. 10 may be based on the combined image of FIG. 9 after image quality improvement processing of region 964 of FIG. 9. In FIG. 10, region 1062 is selected for future image quality improvement processing, while region 1064 has already been processed, so that region 1064 is based on imaging data with improved image quality. Thus, the three-dimensional volume 1000 may be based primarily on the first imaging data (e.g., low-resolution imaging data), except for at least region 1064, which has been replaced with imaging data with improved image quality, and thus depicted with improved image quality (e.g., higher resolution) than that of the other parts of the three-dimensional volume 1000. Similarly, the two-dimensional image 1001 may be based primarily on the first imaging data (e.g., low-resolution imaging data), except for at least region 1064, which has been replaced with imaging data with improved image quality, and thus depicted with improved image quality (e.g., higher resolution) than that of the other parts of the two-dimensional image 1001.

[0089] FIG. 11 is a block diagram of an exemplary system architecture 1100 for implementing features and processes of the present disclosure, such as those presented with reference to FIGS. 1-10. The architecture 1100 can be used to implement any suitable computing device (e.g., a server, a workstation, a tablet, a cloud, or other such device or service) for implementing various features and processes of the present disclosure. The architecture 1100 can be implemented on any electronic device that runs software applications derived from compiled instructions, including, but not limited to, personal computers, servers, smartphones, electronic tablets, game consoles, email devices, imagers (e.g., imaging machines such as X-ray microscopes and CT scanners), and the like. In some implementations, the architecture 1100 can include one or more processors 1102, one or more input devices 1104, one or more display devices 1106, one or more network interfaces 1108, and one or more computer-readable media 1110. Each of these components can be coupled by a bus 1112.

[0090] In some cases, the system architecture 1100 can be incorporated into a computing system capable of performing x-ray scans, such as a computing system used to control an x-ray imager or a CT scanner. In some cases, the system architecture 1100 can be incorporated into a workstation computer that is primarily used to view and interpret imaging data from x-ray scans. In some cases, the system architecture 1100 can be incorporated into a computer system used to train neural networks, such as a computer system optimized for training deep neural networks.

[0091] In some implementations, the system architecture 1100 may correspond to a single server in a rack of servers. Various rack configurations can be implemented. For example, a rack can include multiple chassis, and each chassis can include multiple servers. Each server in the rack can be connected by various hardware components (e.g., a backbone, a midplane, etc.).

[0092] The display device 1106 may be any known display technology, including, but not limited to, display devices using liquid crystal display (LCD) or light emitting diode (LED) technology. The processor 1102 can use any known processor technology, including, but not limited to, graphics processors and multi-core processors. The input device 1104 may be any known input device technology, including, but not limited to, a keyboard (including a virtual keyboard), a mouse, a trackball, and a touch-sensitive pad or display. The bus 1112 may be any known internal or external bus technology, including, but not limited to, ISA, EISA, PCI, PCI Express, NuBus, USB, Serial ATA, or FireWire.

[0093] The computer-readable medium 1110 may be any medium involved in providing instructions for the processor 1102 to execute, including, but not limited to, non-volatile storage media (e.g., optical disks, magnetic disks, flash drives, etc.) or volatile media (e.g., SDRAM, ROM, etc.). Computer-readable media (e.g., storage devices, media, and memories) may include, for example, cables or wireless signals containing bit streams and the like. However, when referring to non-transitory computer-readable storage media, it specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0094] The processor 1102 may include any combination of suitable processors. Certain aspects of the present disclosure may be particularly useful when implemented using one or more graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or tensor processing units (TPUs) or other artificial intelligence accelerators. In some cases, certain instructions for executing code on a GPU, FPGA, and / or TPU may be programmed using a suitable language (e.g., Compute Unified Device Architecture (CUDA) for certain GPUs, or a hardware description language for FPGAs).

[0095] The computer readable medium 1110 may include various instructions for implementing an operating system 1114 and applications 1120, such as computer programs. The operating system may be multi-user, multi-processing, multi-tasking, multi-threaded, real-time, etc. The operating system 1114 performs basic tasks including, but not limited to, recognizing input from the input device 1104, sending output to the display device 1106, managing files and directories on the computer readable medium 1110, controlling peripheral devices (e.g., disk drives, printers, x-ray microscopes, etc.) that may be controlled directly by or through an I / O controller, and managing traffic on the bus 1112. The computer readable medium 1110 may include various instructions for implementing firmware processes, such as a BIOS. The computer readable medium 1110 may include various instructions for implementing any of the processes described herein, including at least process 300 of FIG. 3.

[0096] The memory 1118 can include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR), etc. The memory 1118 (e.g., computer-readable storage devices, media, and memories) can include cables or wireless signals containing bit streams and the like. However, when referring to non-transitory computer-readable storage media, it specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves. The memory 1118 can store an operating system, such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, etc., or an embedded operating system, such as VxWorks, etc.

[0097] The system controller 1122 may be a service processor that operates independently of the processor 1102. In some implementations, the system controller 1122 may be a baseboard management controller (BMC). For example, a BMC is a dedicated service processor that monitors the physical state of a computer, network server, or other hardware device using sensors and communication with a system administrator through an independent connection. The BMC is configured on the motherboard or main circuit board of the monitored device. The sensors of the BMC can measure internal physical variables such as temperature, humidity, power supply voltage, fan speed, communication parameters, and operating system (OS) functions.

[0098] The described features may be advantageously implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to the data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform a certain activity or bring about a certain result. A computer program may be written in any form of programming language (e.g., object-oriented C, Java, CUDA), including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0099] Processors suitable for executing a program of instructions include, by way of example, both general purpose and special purpose microprocessors, and the sole processor or one of multiple processors or cores of any kind of computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data files, or is operatively coupled to communicate with one or more mass storage devices, such as magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and all forms of non-volatile memory, including CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in ASICs (Application Specific Integrated Circuits).

[0100] To enable interaction with a user, the features may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer.

[0101] Features may be implemented in a computing system that includes back-end components such as a data server, or that includes middleware components such as an application server or an Internet server, or that includes a front-end component such as a client computer having a graphical user interface or an Internet browser, or some combination thereof. The components of the system may be connected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include, for example, a LAN, a WAN, and the computers and networks forming the Internet.

[0102] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a network. The relationship of clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0103] One or more features or steps of the disclosed embodiments can be implemented using an application programming interface (API). An API can define one or more parameters that pass between a calling application and other software code (e.g., operating systems, library routines, functions) that provides a service, provides data, or performs an operation or calculation. In one example, an API can be used to facilitate accessing and / or storing a neural network.

[0104] An API may be implemented as one or more calls in program code that send or receive one or more parameters through a parameter list or other structure based on a calling convention defined in an API standard document. A parameter may be a constant, a key, a data structure, an object, an object class, a variable, a data type, a pointer, an array, a list, or another call. API calls and parameters may be implemented in any programming language. A programming language may define the vocabulary and calling conventions that a programmer employs to access functions that support the API.

[0105] In some implementations, the API calls can report to the application the capabilities of the device on which the application runs, such as input capabilities, output capabilities, processing capabilities, power capabilities, communication capabilities, and the like.

[0106] The above description of the embodiments, including the described embodiments, has been presented for illustration and description only and is not intended to be exhaustive or to limit the precise forms disclosed. Many modifications, adaptations, and uses of the above description will be apparent to those skilled in the art. Many changes to the disclosed embodiments can be made in accordance with the disclosure herein without departing from the spirit or scope of the disclosure. Thus, the breadth and scope of the disclosure should not be limited by any of the embodiments described above.

[0107] While certain aspects and features of the present disclosure have been illustrated and described with reference to one or more implementations, equivalent alterations and modifications will occur to and be known to those skilled in the art upon reading and understanding this specification and the accompanying drawings. In addition, while a particular feature is disclosed with respect to only one of some implementations, such feature may be combined with one or more other features of other embodiments as may be desirable and advantageous in any desired or particular application.

[0108] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. Furthermore, to the extent the terms "including," "includes," "having," "has," "with," or variations thereof are used in either the detailed description and / or claims, such terms are intended to be inclusive in a similar manner to the term "comprising."

[0109] As used below, any reference to a series of examples should be understood as a reference to each of those examples disjunctively (e.g., "Examples 1-4" should be understood as "Examples 1, 2, 3, or 4").

[0110] Example 1 is a system including one or more data processors and a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including receiving first imaging data associated with the specimen, the first imaging data being acquired using a first imaging modality; receiving second imaging data associated with the specimen, the second imaging data being acquired using a second imaging modality different from the first imaging modality; registering the first imaging data with the second imaging data; identifying a portion of the first imaging data that overlaps with the second imaging data; training a neural network based at least in part on the portion of the first imaging data and the second imaging data; and generating output imaging data by applying the trained neural network to the first imaging data or third imaging data acquired using the first imaging modality.

[0111] Example 2 is the system of Example 1, where the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality.

[0112] Example 3 is the system of Example 2, where the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first and second objective measurement or (ii) the same as the second objective measurement.

[0113] Example 4 is the system of Examples 1-3, where first imaging data is acquired at a first resolution and a first field of view, second imaging data is acquired at a second resolution higher than the first resolution and a second field of view smaller than the first field of view, and output imaging data has a resolution higher than the first resolution.

[0114] Example 5 is the system of Examples 1-4, where the first imaging modality and the second imaging modality share a common primary imaging modality.

[0115] Example 6 is the system of Examples 1-4, where the first imaging modality includes a first primary imaging modality that is x-ray microscopy and the second imaging modality includes a second primary imaging modality that is not x-ray microscopy.

[0116] Example 7 is the system of Examples 1-6, wherein aligning the first imaging data and the second imaging data includes automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data.

[0117] Example 8 is the system of Examples 1-7, wherein the operation further includes resampling at least a portion of the first imaging data onto a grid of the second imaging data.

[0118] Example 9 is the system of Examples 1-8, wherein training the neural network includes providing portions of the first imaging data that relate to the second imaging data to the neural network to generate output data, and adjusting the neural network to reduce a difference between the generated output data and the second imaging data.

[0119] Example 10 is the system of Examples 1-9, wherein generating the output imaging data includes applying a zoom transform to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data, and applying the transformed first imaging data to a trained neural network to generate the output imaging data.

[0120] Example 11 is the system of Examples 1-10, wherein generating the output imaging data includes segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk, applying each of the plurality of chunks of the first imaging data to a trained neural network to generate a corresponding chunk of output imaging data, and stitching together the overlapping regions of the corresponding chunks of the output imaging data.

[0121] Example 12 is the system of Examples 1-11, wherein generating the output imaging data includes receiving a target area selection, the target area selection indicating a target area of ​​the first imaging data, tagging the target area with metadata indicating a location of the target area, applying the first imaging data associated with the target area to a trained neural network to generate a corresponding target area in the output imaging data, tagging the corresponding target area in the output imaging data with metadata associated with the target area, and locating the corresponding target area in the output imaging data based at least in part on the metadata.

[0122] Example 13 is the system of Examples 1-12, wherein the operations further include identifying a plurality of change regions associated with the specimen; training the neural network and generating the output is performed using a first subset of the first imaging data and the second imaging data associated with the first change region, and the operations further include training a second neural network based at least in part on the second subset of the first imaging data and the second imaging data associated with the second change region; generating second output imaging data associated with the second change region by applying the trained second neural network to the subset of the first imaging data associated with the second change region; and stitching the output imaging data and the second output imaging data together.

[0123] Example 14 is the system of examples 1-13, wherein the operations further include generating a two-dimensional image or a three-dimensional volume using the output imaging data.

[0124] Example 15 is the system of Example 14, wherein a generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data, such that at least one region of the two-dimensional image or three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or three-dimensional volume is based on the first imaging data.

[0125] Example 16 is the system of examples 1-15, wherein generating the output imaging data includes applying the trained neural network to the first imaging data.

[0126] Example 17 is a method that includes receiving first imaging data associated with a specimen, the first imaging data being acquired using a first imaging modality; receiving second imaging data associated with the specimen, the second imaging data being acquired using a second imaging modality different from the first imaging modality; aligning the first imaging data with the second imaging data; identifying a portion of the first imaging data that overlaps with the second imaging data; training a neural network based at least in part on the portion of the first imaging data and the second imaging data; and generating output imaging data by applying the trained neural network to the first imaging data or third imaging data acquired using the first imaging modality.

[0127] Example 18 is the method of Example 17, wherein the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality.

[0128] Example 19 is the method of Example 18, wherein the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first and second objective measurement or (ii) the same as the second objective measurement.

[0129] Example 20 is the method of Examples 17-19, wherein the first imaging data is acquired at a first resolution and a first field of view, the second imaging data is acquired at a second resolution higher than the first resolution and a second field of view smaller than the first field of view, and the output imaging data has a resolution higher than the first resolution.

[0130] Example 21 is the method of examples 17-20, wherein the first imaging modality and the second imaging modality share a common primary imaging modality.

[0131] Example 22 is the method of Examples 17-20, wherein the first imaging modality includes a first primary imaging modality that is x-ray microscopy and the second imaging modality includes a second primary imaging modality that is not x-ray microscopy.

[0132] Example 23 is the method of Examples 17-22, wherein the step of aligning the first imaging data and the second imaging data includes automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data.

[0133] Example 24 is the method of Examples 17-23, further comprising resampling at least a portion of the first imaging data onto a grid of the second imaging data.

[0134] Example 25 is the method of Examples 17-24, wherein training the neural network includes providing a portion of the first imaging data that is related to the second imaging data to the neural network to generate output data; and adjusting the neural network to reduce a difference between the generated output data and the second imaging data.

[0135] Example 26 is the method of Examples 17-25, wherein generating the output imaging data includes applying a zoom transform to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data, and applying the transformed first imaging data to the trained neural network to generate the output imaging data.

[0136] Example 27 is the method of Examples 17-26, wherein the step of generating the output imaging data includes the steps of segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk, applying each of the plurality of chunks of the first imaging data to the trained neural network to generate a corresponding chunk of output imaging data, and stitching together the overlapping regions of the corresponding chunks of the output imaging data.

[0137] Example 28 is the method of Examples 17-27, wherein generating the output imaging data includes receiving a target area selection, the target area selection indicating a target area of ​​the first imaging data; tagging the target area with metadata indicating a location of the target area; applying the first imaging data associated with the target area to a trained neural network to generate a corresponding target area of ​​the output imaging data; tagging the corresponding target area of ​​the output imaging data with metadata associated with the target area; and locating the corresponding target area of ​​the output imaging data based at least in part on the metadata.

[0138] Example 29 is the method of Examples 17-28, further comprising identifying a plurality of change regions associated with the specimen, wherein the steps of training the neural network and generating the output are performed using a first subset of the first imaging data and second imaging data associated with the first change region, and the method further comprises training a second neural network based at least in part on the second subset of the first imaging data and second imaging data associated with the second change region, generating second output imaging data associated with the second change region by applying the trained second neural network to the subset of the first imaging data associated with the second change region, and stitching the output imaging data and the second output imaging data together.

[0139] Example 30 is the method of Examples 17-29, further comprising using the output imaging data to generate a two-dimensional image or a three-dimensional volume.

[0140] Example 31 is the method of Example 30, wherein a generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data, such that at least one region of the two-dimensional image or three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or three-dimensional volume is based on the first imaging data.

[0141] Example 32 is the method of Examples 17-31, wherein generating the output imaging data includes applying the trained neural network to the first imaging data.

[0142] Example 33 is a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause a data processing device to perform operations including receiving first imaging data associated with a specimen, the first imaging data being acquired using a first imaging modality; receiving second imaging data associated with the specimen, the second imaging data being acquired using a second imaging modality different from the first imaging modality; aligning the first imaging data with the second imaging data; identifying a portion of the first imaging data that overlaps with the second imaging data; training a neural network based at least in part on the portion of the first imaging data and the second imaging data; and generating output imaging data by applying the trained neural network to the first imaging data or third imaging data acquired using the first imaging modality.

[0143] Example 34 is the computer program product of Example 33, wherein the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality.

[0144] Example 35 is the computer program product of Example 34, wherein the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first objective measurement and the second objective measurement, or (ii) the same as the second objective measurement.

[0145] Example 36 is the computer program product of Examples 33-35, wherein the first imaging data is acquired at a first resolution and a first field of view, and the second imaging data is acquired at a second resolution higher than the first resolution and a second field of view smaller than the first field of view, and the output imaging data has a resolution higher than the first resolution.

[0146] Example 37 is the computer program product of Examples 33-36, wherein the first imaging modality and the second imaging modality share a common primary imaging modality.

[0147] Example 38 is the computer program product of Examples 33-36, wherein the first imaging modality includes a first primary imaging modality that is x-ray microscopy and the second imaging modality includes a second primary imaging modality that is not x-ray microscopy.

[0148] Example 39 is the computer program product of Examples 33-38, wherein aligning the first imaging data and the second imaging data includes automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data.

[0149] Example 40 is the computer program product of Examples 33-39, wherein the operations further include resampling at least a portion of the first imaging data onto a grid of the second imaging data.

[0150] Example 41 is the computer program product of Examples 33-40, wherein training the neural network includes providing portions of the first imaging data that relate to the second imaging data to the neural network to generate output data; and adjusting the neural network to reduce a difference between the generated output data and the second imaging data.

[0151] Example 42 is the computer program product of Examples 33-41, wherein generating the output imaging data includes applying a zoom transform to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data, and applying the transformed first imaging data to the trained neural network to generate the output imaging data.

[0152] Example 43 is the computer program product of Examples 33-42, wherein generating the output imaging data includes segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk; applying each of the plurality of chunks of the first imaging data to the trained neural network to generate a corresponding chunk of output imaging data; and stitching together the overlapping regions of the corresponding chunks of the output imaging data.

[0153] Example 44 is the computer program product of Examples 33-43, wherein generating the output imaging data includes receiving a target area selection, where the target area selection indicates a target area of ​​the first imaging data, tagging the target area with metadata indicative of a location of the target area, applying the first imaging data associated with the target area to a trained neural network to generate a corresponding target area of ​​the output imaging data, tagging the corresponding target area of ​​the output imaging data with metadata associated with the target area, and locating the corresponding target area of ​​the output imaging data based at least in part on the metadata.

[0154] Example 45 is the computer program product of Examples 33-44, wherein the operations further include identifying a plurality of change regions associated with the specimen; training the neural network and generating the output is performed using a first subset of the first imaging data and the second imaging data associated with the first change region, and the operations further include training a second neural network based at least in part on the second subset of the first imaging data and the second imaging data associated with the second change region; generating second output imaging data associated with the second change region by applying the trained second neural network to the subset of the first imaging data associated with the second change region; and stitching the output imaging data and the second output imaging data together.

[0155] Example 46 is the computer program product of Examples 33-45, wherein the operations further include generating a two-dimensional image or a three-dimensional volume using the output imaging data.

[0156] Example 47 is the computer program product of Example 46, wherein a generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data, such that at least one region of the two-dimensional image or three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or three-dimensional volume is based on the first imaging data.

[0157] Example 48 is the computer program product of Examples 33-47, wherein generating the output imaging data includes applying the trained neural network to the first imaging data. [Explanation of symbols]

[0158] 100 Imaging data processing system 102 First imaging data source 103 Secondary imaging data source 104 Processing Module 105 Specimen Holder 106 Input / Output Module 107 Specimen 108 Neural Network Memory Device 210 Input image data 212 First Imaging Data 214 Second Imaging Data 216 Neural Networks 218 Output imaging data 400 Graphical User Interface, GUI 412 First Imaging Data 414 Second Imaging Data 500 Graphical User Interface, GUI 552 Registration Options 600 low resolution images 700 high resolution images 800 images 900 3D volume 901 2D Images 962 areas 964 areas 1000 3D Volume 1001 2D Images 1062 areas 1064 area 1100 System Architecture 1102 Processor 1104 Input Devices 1106 Display Devices 1108 Network Interface 1110 Computer-readable medium 1112 Bus 1114 Operating Systems 1118 Memory 1120 Application 1122 System Controller

Claims

1. one or more data processors; and A non-transitory computer-readable storage medium that, when executed on the one or more data processors, causes the one or more data processors to: receiving first imaging data associated with a specimen, the first imaging data being acquired using a first imaging modality; receiving second imaging data related to the specimen, the second imaging data being acquired using a second imaging modality different from the first imaging modality; Aligning the first imaging data and the second imaging data; identifying a portion of the first imaging data that overlaps with the second imaging data; training a neural network based at least in part on the portion of the first imaging data and the second imaging data; generating output imaging data by applying the trained neural network to the first imaging data or to third imaging data acquired using the first imaging modality; A non-transitory computer-readable storage medium comprising instructions for performing operations including: A system comprising:

2. 2. The system of claim 1, wherein the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality.

3. 3. The system of claim 2, wherein the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first and second objective measurement or (ii) the same as the second objective measurement.

4. 2. The system of claim 1, wherein the first imaging data is acquired at a first resolution and a first field of view, and the second imaging data is acquired at a second resolution higher than the first resolution and a second field of view lower than the first field of view, and the output imaging data has a resolution higher than the first resolution.

5. The system of claim 1 , wherein the first imaging modality and the second imaging modality share a common primary imaging modality.

6. The system of claim 1 , wherein the first imaging modality comprises a first primary imaging modality that is X-ray microscopy and the second imaging modality comprises a second primary imaging modality that is not X-ray microscopy.

7. 2. The system of claim 1, wherein aligning the first imaging data and the second imaging data includes automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data.

8. The system of claim 1 , wherein the operations further comprise resampling at least a portion of the first imaging data onto a grid of the second imaging data.

9. Training the neural network providing the portion of the first imaging data that is related to the second imaging data to the neural network to generate the output data; tuning the neural network to reduce a discrepancy between the generated output data and the second imaging data; and The system of claim 1 , comprising:

10. generating the output imaging data, applying a zoom transform to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data; applying the transformed first imaging data to the trained neural network to generate the output imaging data; The system of claim 1 , comprising:

11. generating the output imaging data, Segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk; applying each of the plurality of chunks of the first imaging data to the trained neural network to generate a corresponding chunk of the output imaging data; stitching together the overlapping regions of the corresponding chunks of the output imaging data; and The system of claim 1 , comprising:

12. generating the output imaging data, receiving a region of interest selection, the region of interest selection indicating a region of interest of the first imaging data; tagging the target area with metadata indicative of a location of the target area; applying the first imaging data associated with the region of interest to the trained neural network to generate a corresponding region of interest in the output imaging data; tagging the corresponding regions of interest of the output imaging data with the metadata associated with the regions of interest; Locating the corresponding region of interest in the output imaging data based at least in part on the metadata. The system of claim 1 , comprising:

13. The operations further include identifying a plurality of change regions associated with the specimen, wherein training the neural network and generating the output is performed using a first subset of the first imaging data and the second imaging data associated with a first change region, the operations further comprising: training a second neural network based at least in part on a second subset of the first imaging data and the second imaging data associated with a second region of change; applying the trained second neural network to the subset of the first imaging data associated with the second changed region to generate second output imaging data associated with the second changed region; Stitching the output imaging data and the second output imaging data together. The system of claim 1 further comprising:

14. The system of claim 1 , wherein the operations further include generating a two-dimensional image or a three-dimensional volume using the output imaging data.

15. 15. The system of claim 14, wherein the generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data such that at least one region of the two-dimensional image or the three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or the three-dimensional volume is based on the first imaging data.

16. The system of claim 1 , wherein generating the output imaging data comprises applying the trained neural network to the first imaging data.

17. receiving first imaging data associated with a specimen, the first imaging data being acquired using a first imaging modality; receiving second imaging data related to the specimen, the second imaging data being acquired using a second imaging modality different from the first imaging modality; aligning the first imaging data with the second imaging data; identifying a portion of the first imaging data that overlaps with the second imaging data; training a neural network based at least in part on the portion of the first imaging data and the second imaging data; generating output imaging data by applying the trained neural network to the first imaging data or third imaging data acquired using the first imaging modality; A method comprising:

18. 20. The method of claim 17, wherein the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality.

19. 20. The method of claim 18, wherein the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first and second objective measurement, or (ii) the same as the second objective measurement.

20. 18. The method of claim 17, wherein the first imaging data is acquired at a first resolution and a first field of view, and the second imaging data is acquired at a second resolution higher than the first resolution and a second field of view lower than the first field of view, and the output imaging data has a resolution higher than the first resolution.

21. The method of claim 17 , wherein the first imaging modality and the second imaging modality share a common primary imaging modality.

22. 18. The method of claim 17, wherein the first imaging modality comprises a first primary imaging modality that is X-ray microscopy and the second imaging modality comprises a second primary imaging modality that is not X-ray microscopy.

23. 18. The method of claim 17, wherein aligning the first imaging data and the second imaging data comprises automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data.

24. 20. The method of claim 17, further comprising resampling at least a portion of the first imaging data onto a grid of the second imaging data.

25. training the neural network, providing the portion of the first imaging data that is related to the second imaging data to the neural network to generate the output data; tuning the neural network to reduce a discrepancy between the generated output data and the second imaging data; 20. The method of claim 17, comprising:

26. generating the output imaging data, applying a zoom transformation to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data; applying the transformed first imaging data to the trained neural network to generate the output imaging data; 20. The method of claim 17, comprising:

27. generating the output imaging data, Segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk; applying each of the plurality of chunks of the first imaging data to the trained neural network to generate a corresponding chunk of the output imaging data; stitching together the overlapping regions of the corresponding chunks of the output imaging data; 20. The method of claim 17, comprising:

28. generating the output imaging data, receiving a region of interest selection, the region of interest selection indicating a region of interest in the first imaging data; tagging the target area with metadata indicative of a location of the target area; applying the first imaging data associated with the region of interest to the trained neural network to generate a corresponding region of interest in the output imaging data; tagging the corresponding regions of interest of the output imaging data with the metadata associated with the regions of interest; locating the corresponding region of interest in the output imaging data based at least in part on the metadata; 20. The method of claim 17, comprising:

29. The method further comprises identifying a plurality of change regions associated with the specimen, wherein the steps of training the neural network and generating the output are associated with a first change region and are performed using a first subset of the first imaging data and the second imaging data associated with the first change region, the method comprising: training a second neural network based at least in part on a second subset of the first imaging data and the second imaging data associated with a second changed region; applying the trained second neural network to the subset of the first imaging data associated with the second changed region to generate second output imaging data associated with the second changed region; stitching the output imaging data and the second output imaging data together; 20. The method of claim 17, further comprising:

30. The method of claim 17 , further comprising using the output imaging data to generate a two-dimensional image or a three-dimensional volume.

31. 31. The method of claim 30, wherein the generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data such that at least one region of the two-dimensional image or the three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or the three-dimensional volume is based on the first imaging data.

32. 20. The method of claim 17, wherein generating the output imaging data comprises applying the trained neural network to the first imaging data.

33. A data processing device, receiving first imaging data associated with a specimen, the first imaging data being acquired using a first imaging modality; receiving second imaging data related to the specimen, the second imaging data being acquired using a second imaging modality different from the first imaging modality; Aligning the first imaging data and the second imaging data; identifying a portion of the first imaging data that overlaps with the second imaging data; training a neural network based at least in part on the portion of the first imaging data and the second imaging data; generating output imaging data by applying the trained neural network to the first imaging data or to third imaging data acquired using the first imaging modality; A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause the computer to perform operations including:

34. 34. The computer program product of claim 33, wherein the first imaging data is associated with a first image quality, the second imaging data is associated with a second image quality, and the output imaging data is associated with a third image quality that is improved relative to the first image quality.

35. 35. The computer program product of claim 34, wherein the first image quality is associated with a first objective measurement, the second image quality is associated with a second objective measurement, and the third image quality is associated with a third objective measurement that is (i) between the first and second objective measurement or (ii) the same as the second objective measurement.

36. 34. The computer program product of claim 33, wherein the first imaging data is acquired at a first resolution and a first field of view, and the second imaging data is acquired at a second resolution higher than the first resolution and a second field of view lower than the first field of view, and the output imaging data has a resolution higher than the first resolution.

37. 34. The computer program product of claim 33, wherein the first imaging modality and the second imaging modality share a common primary imaging modality.

38. 34. The computer program product of claim 33, wherein the first imaging modality comprises a first primary imaging modality that is x-ray microscopy and the second imaging modality comprises a second primary imaging modality that is not x-ray microscopy.

39. 34. The computer program product of claim 33, wherein aligning the first imaging data and the second imaging data includes automatically applying a 3, 6, or 9 degree of freedom sub-pixel alignment to at least one of the first imaging data and the second imaging data to align the first imaging data and the second imaging data.

40. 34. The computer program product of claim 33, wherein the operations further comprise resampling at least a portion of the first imaging data onto a grid of the second imaging data.

41. Training the neural network providing the portion of the first imaging data that is related to the second imaging data to the neural network to generate the output data; tuning the neural network to reduce a discrepancy between the generated output data and the second imaging data; and 34. The computer program product of claim 33, comprising:

42. generating the output imaging data, applying a zoom transform to the first imaging data to achieve an effective pixel size equal to a pixel size of the second imaging data; applying the transformed first imaging data to the trained neural network to generate the output imaging data; 34. The computer program product of claim 33, comprising:

43. generating the output imaging data, Segmenting the first imaging data into a plurality of chunks, each chunk including at least one overlapping region with an adjacent chunk; applying each of the plurality of chunks of the first imaging data to the trained neural network to generate a corresponding chunk of the output imaging data; stitching together the overlapping regions of the corresponding chunks of the output imaging data; and 34. The computer program product of claim 33, comprising:

44. generating the output imaging data, receiving a region of interest selection, the region of interest selection indicating a region of interest of the first imaging data; tagging the target area with metadata indicative of a location of the target area; applying the first imaging data associated with the region of interest to the trained neural network to generate a corresponding region of interest in the output imaging data; tagging the corresponding regions of interest of the output imaging data with the metadata associated with the regions of interest; Locating the corresponding region of interest in the output imaging data based at least in part on the metadata.

34. The computer program product of claim 33, comprising:

45. The operations further include identifying a plurality of change regions associated with the specimen, wherein training the neural network and generating the output is performed using a first subset of the first imaging data and the second imaging data associated with a first change region, the operations further comprising: training a second neural network based at least in part on a second subset of the first imaging data and the second imaging data associated with a second region of change; applying the trained second neural network to the subset of the first imaging data associated with the second changed region to generate second output imaging data associated with the second changed region; Stitching the output imaging data and the second output imaging data together.

34. The computer program product of claim 33, further comprising:

46. 34. The computer program product of claim 33, wherein the operations further comprise generating a two-dimensional image or a three-dimensional volume using the output imaging data.

47. 47. The computer program product of claim 46, wherein the generated two-dimensional image or three-dimensional volume is generated using the output imaging data and the first imaging data such that at least one region of the two-dimensional image or the three-dimensional volume is based on the output imaging data and at least one different region of the two-dimensional image or the three-dimensional volume is based on the first imaging data.

48. 34. The computer program product of claim 33, wherein generating the output imaging data comprises applying the trained neural network to the first imaging data.

Images