Fast scanning and image processing in a line scanning system

Abstract

Image scanning can be accelerated using a method for generating upsampled images. Such a method may comprise establishing relative motion between a field of view of a camera and a sample supported by a scanning stage in a first dimension at a compressing scanning speed, capturing an input image, and generating an upsampled image based on applying an upsampling algorithm to the input image. In some cases, the compressing scanning speed may be a speed that is greater than a synchronized scanning speed, wherein the synchronized scanning speed is synchronized with a line rate of the camera.

Description

FAST SCANNING AND IMAGE PROCESSING IN A LINE SCANNING SYSTEMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This is related to, and claims the benefit of, provisional patent application 63 / 549,216 for “Fast Scanning and Image Processing in a Line Scanning System,” filed in the U.S. patent office on February 2, 2024. That application is hereby incorporated by reference in its entirety.BACKGROUNDFIELD

[0002] The embodiments described herein are generally directed to operation of a line scanning system, and, more particularly, to improving scanning speed using image upsampling in a line scanning system.RELATED ART

[0003] Scan speed in line scanning based imaging is a function of camera pixel size, image magnification, and camera line rate. With a fixed camera pixel size and a fixed magnification, the higher the camera line rate is, the faster the scanning of the image will be.SUMMARY

[0004] In a line scanning system, the higher the camera line rate is, the higher its cost will be. In addition, upgrading a scanning device with a faster camera usually requires system hardware and system software redesign resulting in significant development cost and development and deployment time. Accordingly, there is a need for improvements in technology for increasing the scanning speed of line scanning systems. Some aspects of the disclosed technology can be implemented to provide such improvements.

[0005] The disclosed technology may be implemented in a variety of manners. For example, in some aspects there may be provided a method for generating upsampled images. Sucha method may comprise establishing relative motion between a field of view of a camera and a sample supported by a scanning stage in a first dimension at a compressing scanning speed, capturing an input image, and generating an upsampled image based on applying an upsampling algorithm to the input image. In some cases, the compressing scanning speed may be a speed that is greater than a synchronized scanning speed, wherein the synchronized scanning speed is synchronized with a line rate of the camera.

[0006] Other types of implementations, including in the form of systems and computer readable media for performing such methods, are also possible and will be immediately apparent to those of skill in the art in light of this disclosure. Accordingly, the example method provided in this summary should be understood as being illustrative only, and should not be treated as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The details of aspects of the disclosed technology, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0008] FIG. 1 A illustrates an example processor-enabled device that may be used in connection with the various embodiments described herein;

[0009] FIG. IB illustrates an example line-scan camera having a single linear array, according to an embodiment;

[0010] FIG. ID illustrates an example line-scan camera having a plurality of linear arrays, according to an embodiment;

[0011] FIG. 1C illustrates an example line-scan camera having three linear arrays, according to an embodiment;

[0012] FIG. 2 illustrates an example of a process which may be used for generating upsampled images using a processor-enabled slide-scanning system;

[0013] FIG. 3 shows how an input image captured at the synchronized scanning speed may compare with an input image captured at a compressing scanning speed equal to twice the synchronized scanning speed;

[0014] FIG. 4 shows how groups of interpolation pixels can be added to an input image;

[0015] FIG. 5 illustrates a method which could be used to train a neural network such as could be used in generating an upsampled image; and

[0016] FIG. 6 illustrates an application of the disclosed technology to an array of 20 micron diameter circles with 5 micron dots in their centers.DETAILED DESCRIPTION

[0017] Disclosed herein is technology which can be used to increase the scan speed of a line scanning system while maintaining high resolution digital image creation of the scanned subject. After reading this description, it will become apparent to one skilled in the art how to implement the invention in various alternative embodiments and alternative applications. However, although various embodiments of the present invention will be described herein, it is understood that these embodiments are presented by way of example and illustration only, and not limitation. As such, this detailed description of various embodiments should not be construed to limit the scope or breadth of the present invention as set forth in the appended claims.

[0018] 1. Example Scanning System

[0019] FIG. 1 A is a block diagram illustrating an example processor-enabled slide-scanning system 100 that may be used in connection with various embodiments described herein. Alternative forms of scanning system 100 may also be used as will be understood by the skilled artisan. In the illustrated embodiment, scanning system 100 is presented as a digital imaging device that comprises one or more processors 104, one or more memories 106, one or more motion controllers 108, one or more interface systems 110, one or more movable stages 112 that each support one or more glass slides 114 with one or more samples 116, one or more illumination systems 118 that illuminate sample 116,an imaging system 101 that comprises imaging optics 103 such as one or more objective lenses 120 that each define an optical path 122 that travels along an optical axis, one or more objective lens positioners 124, one or more optional epi-illumination systems 126 (e.g., included in a fluorescence-scanning embodiment), one or more focusing optics 128, and one or more line-scan cameras 130, and / or one or more area-scan cameras 132, each of which define a separate field of view 134 on sample 116 and / or glass slide 114. The various elements of scanning system 100 are communicatively coupled via one or more communication busses 102. Although there may be a plurality of each of the various elements of scanning system 100, for simplicity in the description that follows, these elements will be described in the singular, except when needed to be described in the plural to convey the appropriate information.

[0020] Processor 104 may include, for example, a central processing unit (CPU) and a separate graphics processing unit (GPU) capable of processing instructions in parallel, or a multicore processor capable of processing instructions in parallel. Additional separate processors may also be provided to control particular components or perform particular iunctions, such as image processing. For example, additional processors may include an auxiliary processor to manage data input, an auxiliary processor to perform floatingpoint mathematical operations, a special-purpose processor having an architecture suitable for fast execution of signal-processing algorithms (e.g., digital-signal processor), a slave processor subordinate to the main processor (e.g., back-end processor), an additional processor for controlling line-scan camera 130, stage 112, objective lens 120, and / or a display (e.g., a console comprising a touch panel display integral to scanning system 100). Such additional processors may be separate discrete processors or may be integrated into a single processor.

[0021] Memory 106 provides storage of data and instructions for programs that can be executed by processor 104. Memory 106 may include one or more volatile and / or non-volatile computer-readable storage mediums that store the data and instructions. These mediums may include, for example, random-access memory (RAM), read-only memory (ROM), a hard disk drive, a removable storage drive (e.g., comprising flash memory), and / or thelike. Processor 104 is configured to execute instructions that are stored in memory 106, and communicate via communication bus 102 with the various elements of scanning system 100 to carry out the overall function of scanning system 100.

[0022] Communication bus 102 may be configured to convey analog electrical signals and / or digital data. Accordingly, communications from processor 104, motion controller 108, and / or interface system 110, via communication bus 102, may include both electrical signals and digital data. Processor 104, motion controller 108, and / or interface system 110 may also be configured to communicate with one or more of the various elements of scanning system 100 via a wireless communication link.

[0023] Motion control system 108 is configured to precisely control and coordinate X, Y, and / or Z movement of stage 112 (e.g., within an X-Y plane), X, Y, and / or Z movement of objective lens 120 (e.g., along a Z axis orthogonal to the X-Y plane, via objective lens positioner 124), rotational movement of a carousel described elsewhere herein, lateral movement of a push / pull assembly described elsewhere herein, and / or any other moving component of scanning system 100. For example, in a fluorescence-scanning embodiment comprising epi-illumination system 126, motion control system 108 may be configured to coordinate movement of optical filters and / or the like in epi- illumination system 126.

[0024] Interface system 110 allows scanning system 100 to interface with other systems and human operators. For example, interface system 110 may include a console (e.g., atouch panel display) to provide information directly to an operator via a graphical user interface and / or allow direct input from an operator via a touch sensor. Interface system 110 may also be configured to facilitate communication and data transfer between scanning system 100 and one or more external devices that are directly connected to scanning system 100 (e.g., a printer, removable storage medium, etc.), and / or one or more external devices that are indirectly connected to scanning system 100, for example, via one or more networks (e.g., an image storage system, a Scanner Administration Manager (SAM) server and / or other administrative server, an operator station, a user station, etc.).

[0025] Illumination system 118 is configured to illuminate at least a portion of sample 116. Illumination system 118 may include, for example, one or more light sources and illumination optics. The light source(s) could comprise a variable intensity halogen light source with a concave reflective mirror to maximize light output and a KG-1 filter to suppress heat. The light source(s) could comprise any type of arc-lamp, laser, or other source of light. In an embodiment, illumination system 118 illuminates sample 116 in transmission mode, such that line-scan camera 130 and / or area-scan camera 132 sense optical energy that is transmitted through sample 116. Alternatively or additionally, illumination system 118 may be configured to illuminate sample 116 in reflection mode, such that line-scan camera 130 and / or area-scan camera 132 sense optical energy that is reflected from sample 116. Illumination system 118 may be configured to be suitable for interrogation of sample 116 in any known mode of optical microscopy.

[0026] In an embodiment, scanning system 100 includes an epi-illumination system 126 to optimize scanning system 100 for fluorescence scanning. It should be understood that, if fluorescence scanning is not supported by scanning system 100, epi-illumination system 126 may be omitted. Fluorescence scanning is the scanning of samples 116 that include fluorescence molecules, which are photon-sensitive molecules that can absorb light at a specific wavelength (i.e., excitation). These photon-sensitive molecules also emit light at a higher wavelength (i.e., emission). Because the efficiency of this photoluminescence phenomenon is very low, the amount of emitted light is often very low. This low amount of emitted light typically frustrates conventional techniques for scanning and digitizing sample 116 (e.g., transmission-mode microscopy).

[0027] Advantageously, in an embodiment of scanning system 100 that utilizes fluorescence scanning, use of a line-scan camera 130 that includes multiple linear sensor arrays (e.g., a time-delay-integration (TDI) line-scan camera) increases the sensitivity to light of line-scan camera 130 by exposing the same area of sample 116 to each of the plurality of linear sensor arrays of line-scan camera 130. This is particularly useful when scanning faint fluorescence samples with low levels of emitted light. Accordingly, in a fluorescence-scanning embodiment, line-scan camera 130 is preferably a monochromeTDI line-scan camera. Monochrome images are ideal in fluorescence microscopy because they provide a more accurate representation of the actual signals from the various channels present on sample 116. As will be understood by those skilled in the art, a fluorescence sample can be labeled with multiple florescence dyes that emit light at different wavelengths, which are also referred to as “channels.”

[0028] Furthermore, because the low-end and high-end signal levels of various fluorescence samples present a wide spectrum of wavelengths for line-scan camera 130 to sense, it is desirable for the low-end and high-end signal levels that line-scan camera 130 can sense to be similarly wide. Accordingly, in a fluorescence-scanning embodiment, line-scan camera 130 may comprise a monochrome 10-bit 64-linear-array TDI line-scan camera. It should be noted that a variety of bit depths for line-scan camera 130 can be employed for use with such an embodiment.

[0029] Movable stage 112 is configured for precise X-Y movement under control of processor 104 or motion controller 108. Movable stage 112 may also be configured for Z movement under control of processor 104 or motion controller 108. Movable stage 112 is configured to position sample 116 in a desired location during image data capture by line-scan camera 130 and / or area-scan camera 132. Movable stage 112 is also configured to accelerate sample 116 in a scanning direction to a substantially constant velocity, and then maintain the substantially constant velocity during image data capture by line-scan camera 130. In an embodiment, scanning system 100 may employ a high- precision and tightly coordinated X-Y grid to aid in the location of sample 116 on movable stage 112. In an embodiment, movable stage 112 is a linear-motor-based X-Y stage with high-precision encoders employed on both the X and the Y axes. For example, very precise nanometer encoders can be used on the axis in the scanning direction and on the axis that is in the direction perpendicular to the scanning direction and on the same plane as the scanning direction. Stage 112 is also configured to support glass slide 114 upon which sample 116 is disposed.

[0030] Sample 116 can be anything that may be interrogated by optical microscopy. For example, glass microscope slide 114 is frequently used as a viewing substrate forspecimens that include tissues and cells, chromosomes, deoxyribonucleic acid (DNA), protein, blood, bone marrow, urine, bacteria, beads, biopsy materials, or any other type of biological material or substance that is either dead or alive, stained or unstained, labeled or unlabeled. Sample 116 may also be an array of any type of DNA or DNA- related material, such as complementary DNA (cDNA) or ribonucleic acid (RNA), or protein that is deposited on any type of slide or other substrate, including any and all samples commonly known as microarrays. Sample 116 may be a microtiter plate (e.g., a 96-well plate). Other examples of sample 116 include integrated circuit boards, electrophoresis records, petri dishes, film, semiconductor materials, forensic materials, and machined parts.

[0031] Objective lens 120 is mounted on objective positioner 124, which, in an embodiment, employs a very precise linear motor to move objective lens 120 along the optical axis defined by objective lens 120. For example, the linear motor of objective lens positioner 124 may include a fifty-nanometer encoder. The relative positions of stage 112 and objective lens 120 in X, Y, and / or Z axes are coordinated and controlled in a closed- loop manner using motion controller 108 under the control of processor 104 that employs memory 106 for storing information and instructions, including the computerexecutable programmed steps for overall operation of scanning system 100.

[0032] In an embodiment, objective lens 120 is apian apochromatic (“APO”) infinity-corrected objective lens which is suitable for transmission-mode illumination microscopy, reflection-mode illumination microscopy, and / or epi-illumination-mode fluorescence microscopy (e.g., an Olympus 40X, 0.75NA or 20X, 0.75 NA). Advantageously, objective lens 120 is capable of correcting for chromatic and spherical aberrations. Because objective lens 120 is infinity-corrected, focusing optics 128 can be placed in optical path 122 above objective lens 120 where the light beam passing through objective lens 120 becomes a collimated light beam. Focusing optics 128 focus the optical signal captured by objective lens 120 onto the light-responsive elements of linescan camera 130 and / or area-scan camera 132, and may include optical components such as filters, magnification changer lenses, and / or the like. Objective lens 120,combined with focusing optics 128, provides the total magnification for scanning system 100. In an embodiment, focusing optics 128 may contain a tube lens and an optional 2X magnification changer. Advantageously, the 2X magnification changer allows a native 20X objective lens 120 to scan sample 116 at 40X magnification.

[0033] Line-scan camera 130 comprises at least one linear array of picture elements 142 (“pixels”). Line-scan camera 130 may be monochrome or color. Color line-scan cameras typically have at least three linear arrays, while monochrome line-scan cameras may have a single linear array or plural linear arrays. Any type of singular or plural linear array, whether packaged as part of a camera or custom-integrated into an imaging electronic module, can also be used. For example, a three linear array (“red-green-blue” or “RGB”) color line-scan camera or a ninety-six linear array monochrome TDI may also be used. TDI line-scan cameras typically provide a substantially better signal-to- noise ratio (“SNR”) in the output signal by summing intensity data from previously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of integration stages. TDI line-scan cameras comprise multiple linear arrays. For example, TDI line-scan cameras are available with 24, 32, 48, 64, 96, or even more linear arrays. Scanning system 100 also supports linear arrays that are manufactured in a variety of formats including some with 512 pixels, some with 1,024 pixels, and others having as many as 4,096 pixels. Similarly, linear arrays with a variety of pixel sizes can also be used in scanning system 100. The salient requirement for the selection of any type of line-scan camera 130 is that the motion of stage 112 can be synchronized with the line rate of line-scan camera 130, so that stage 112 can be in motion with respect to line-scan camera 130 during the digital image capture of sample 116.

[0034] In an embodiment, the image data generated by line-scan camera 130 is stored in a portion of memory 106 and processed by processor 104 to generate a contiguous digital image of at least a portion of sample 116. The contiguous digital image can be further processed by processor 104, and the processed contiguous digital image can also be stored in memory 106.

[0035] In an embodiment with two or more line-scan cameras 130, at least one of the line-scan cameras 130 can be configured to function as a focusing sensor that operates in combination with at least one of the other line-scan cameras 130 that is configured to function as an imaging sensor. The focusing sensor can be logically positioned on the same optical axis as the imaging sensor or the focusing sensor may be logically positioned before or after the imaging sensor with respect to the scanning direction of scanning system 100. In such an embodiment with at least one line-scan camera 130 functioning as a focusing sensor, the image data generated by the focusing sensor may be stored in a portion of memory 106 and processed by processor 104 to generate focus information, to allow scanning system 100 to adjust the relative distance between sample 116 and objective lens 120 to maintain focus on sample 116 during scanning. Additionally, in an embodiment, the at least one line-scan camera 130 functioning as a focusing sensor may be oriented such that each of a plurality of individual pixels 142 of the focusing sensor is positioned at a different logical height along the optical path 122.

[0036] In operation, the various components of scanning system 100 and the programmed modules stored in memory 106 enable automatic scanning and digitizing of sample 116, which is disposed on glass slide 114. Glass slide 114 is securely placed on movable stage 112 of scanning system 100 for scanning sample 116. Under control of processor 104, movable stage 112 accelerates sample 116 to a substantially constant velocity for sensing by line-scan camera 130. After scanning a segment of image data, movable stage 112 decelerates and brings sample 116 to a substantially complete stop. Movable stage 112 then moves orthogonal to the scanning direction to position sample 116 for scanning of a subsequent segment of image data (e.g., an adjacent segment). Additional segments are subsequently scanned until an entire portion of sample 116 or the entire sample 116 is scanned.

[0037] In an embodiment, computer-executable instructions (e.g., programmed modules and software) are stored in memory 106 and, when executed, enable scanning system 100 to perform the various functions (e.g., display the graphical user interface, execute the disclosed processes, control the components of scanning system 100, etc.) describedherein. In this description, the term “computer-readable storage medium” is used to refer to any media used to store and provide computer-executable instructions to scanning system 100 for execution by processor 104. Examples of these media include memory 106 and any removable or external storage medium (not shown) communicatively coupled with scanning system 100 either directly (e.g., via a universal serial bus (USB), a wireless communication protocol, etc.) or indirectly (e.g., via a wired and / or wireless network).

[0038] FIG. IB illustrates a line-scan camera 130 having a single linear array 140, which may be implemented as a charge-coupled device (“CCD”) or complimentary metal-oxide semiconductor (“CMOS”) array. Single linear array 140 comprises a plurality of individual pixels 142. In the illustrated embodiment, the single linear array 140 has 4,096 pixels 142. In alternative embodiments, linear array 140 may have more or fewer pixels. For example, common formats of linear arrays include 512, 1,024, and 4,096 pixels. Pixels 142 are arranged in a linear fashion to define a field of view 134 for linear array 140. The size of field of view 134 varies in accordance with the magnification of scanning system 100.

[0039] FIG. 1C illustrates a line-scan camera 130 having three linear arrays 140, each of which may be implemented as a CCD array. The three linear arrays 140 combine to form a color array 150. In an embodiment, each individual linear array in color array 150 detects a different color intensity, including, for example, red, green, or blue. The color image data from each individual linear array 140 in color array 150 is combined to form a single field of view 134 of color image data. Other types of color line scan cameras, such as 4-line RGB / mono or RGB / NIR, Bayer mask, prism-based 3 R / G / B sensors may also be used in embodiments in which the disclosed technology is used in a color imaging context.

[0040] FIG. ID illustrates a line-scan camera 130 having a plurality of linear arrays 140, each of which may be implemented as a CCD array. The plurality of linear arrays 140 combine to form a TDI array 160. Advantageously, a TDI line-scan camera may provide a substantially better SNR in its output signal by summing intensity data frompreviously imaged regions of a specimen, yielding an increase in the SNR that is in proportion to the square-root of the number of linear arrays 140 (also referred to as integration stages). A TDI line-scan camera may comprise a larger variety of numbers of linear arrays 140. For example, common formats of TDI line-scan cameras include 24, 32, 48, 64, 96, 120, and even more linear arrays 140.

[0041] 2. Method for Generating Upsampled Images

[0042] An example of a process which may be used for generating upsampled images using a processor-enabled slide-scanning system such as described in the context of FIGS. 1 A- 1D is illustrated in FIG. 2. As shown in that figure, a method for generating multidimensional images may include establishing 201 relative motion between a field of view of a camera (e.g., line scan camera 130) and a sample supported by a scanning stage (e.g., stage 112) at a compressing scanning speed. In this case, the compressing scanning speed may be a speed which is greater than a scanning speed which is synchronized with a line rate of the camera, pixel size of the camera, and the magnification of the imaging system (the “synchronized scanning speed”). The method may then continue with capturing 202 based on the relative motion having been established at the compressing scanning speed which is greater than the synchronized scanning speed. To illustrate what this may entail, consider FIG. 3, discussed below, which shows how an input image captured at the synchronized scanning speed may compare with an input image captured at a compressing scanning speed equal to twice the synchronized scanning speed.

[0043] Turning now to FIG. 3, that figure provides a simplified illustration in which a 4x1 pixel array 301 is used to image a 4x4 sample 302 at both a compressing scanning speed and a synchronized scanning speed. In this illustration, FIG. 3 shows both the input image as it would be captured at the synchronized scanning speed 303 and the input image as it would be captured at a compressing scanning speed which is twice as fast 304. At the synchronized scanning speed, the stage would be moved at a speed such that, when the 4x1 array of pixels 301 had captured data from a single 4x1 group of regions of the sample 302, the next group of regions would be moved into the field of view of the 4x1array of pixels 301. For example, if the pixels were each 5x5 pm, 5 ps were required to capture data for the 4x1 array of pixels 301 (i.e., the line rate is 200 kHz), and the sample was imaged at 40x magnification, the synchronized scanning speed could be calculated as .025 m / s (i.e., (pixel size / magnification) / line period = (5 pm / 40) / 5 pm = 0.125 pm / 5 ps = 0.025 m / s). Thus, where each time step is 5 ps, it would take four time steps (shown as time steps T1-T4 in FIG. 3) to create the input image of the 4x4 sample 302 at the synchronized scanning speed 303.

[0044] To contrast with operation at the synchronized scanning speed, when operating at the compressing scanning speed, only half as much time would be used to create the input image. For example, using a 2:1 ratio of compressing scanning speed to synchronized scanning speed, the scanning stage could be moved at a speed of 0.05 m / s in a scenario having the same parameters noted above for the illustration of the synchronized scanning speed. As a result, during each time step, each pixel in the 4x1 pixel array 301 would capture data from two regions of the 4x4 sample 302 (e.g., during time step Ti, pixel Pi would capture data from regions Si,i and 82,1). This would allow all data from the 4x4 sample 302 to be captured and included in the input image at the compressing scanning speed 304 after only have as many time steps as are necessary for the same data to be included in the input image captured at the synchronized scanning speed 303. However, the input image captured at the compressing scanning speed 304 would only have half as many pixels in the scanning direction, with each of the pixels in the input image captured at the compressing scanning speed 304 combining data from two regions of the 4x4 sample 302, rather than having the same 1:1 region to pixel ratio as in the input image captured at the synchronized scanning speed 303.

[0045] It should be understood that, while FIG. 3 illustrated how operation at a compressing scanning speed rather than synchronized scanning speed may impact the resulting images, the figure should be understood as only an example, and variations on the approach illustrated in that figure could be used when implementing the disclosed technology. For example, while FIG. 3 illustrates a 2:1 compressing scanning speed to synchronized scanning speed ratio, other ratios (e.g., 3:1, 4:1) may be used, with thescanning time varying inversely and the number of regions represented in each pixel varying directly with the ratio of the compressing scanning speed to the synchronized scanning speed. Other variations are also possible, and will be immediately apparent to those of skill in the art in light of this disclosure. Accordingly, FIG. 3, and the associated disclosure should be understood as being illustrative only, and should not be treated as limiting.

[0046] Returning now to FIG. 2, after the input image has been captured 202, an upsampled image may be generated 203 based on applying an upsampling algorithm to the input image. This may include, for example, adding 204 groups of interpolation pixels to the input image. For example, as shown in FIG. 4, this can be done by duplicating groups of pixels 401 402 by adding neighboring group of interpolation pixels 403 404 in the same dimension as the scanning direction. As shown in FIG. 4, this would result in each pixel from each group of pixels in the input image having a corresponding interpolation pixel with the same value in the group of interpolation pixels which corresponds to that pixel’s group of pixels (e.g., the topmost pixel in the first group of pixels 401 has a corresponding pixel with the same value as the topmost pixel in the corresponding group of interpolated pixels 403). Once the interpolation pixels had been added 204, an interpolation algorithm could be applied 205 to the input image, which at that point may include both the original groups of pixels and the pixels from the corresponding groups of interpolation pixels. This interpolation algorithm may be, for example, nearest neighbor interpolation, linear interpolation, bilinear interpolation, cubic spline interpolation, bicubic spline interpolation, generalized linear interpolation, or generalized bicubic interpolation. It should be noted that, because the capture 202 of the input image at the compressing scanning speed results only in compression in the dimension of the scanning direction, the applied 205 interpolation algorithm may be a one dimensional interpolation algorithm which interpolates in the dimension of the scanning direction.

[0047] Of course, other approaches to generating 203 an upsampled image are possible, and may be implemented in various embodiments. For example, in some cases, rather thanapplying 205 an interpolation algorithm like bicubic interpolation as described above, an input image may be provided 206 as input to a machine learning model which had been trained to generate upsampled images. Such a learning model may be, for example, a convolutional neural network (CNN) comprising one or more convolution layers which could be used to identify features in the input image, and one or more transpose convolution layers which could be used to increase the dimensions of the input image (e.g., if the dimensions were reduced by the one or more convolution layers and / or if the input image was provided 206 without first adding 204 groups of interpolation pixels). The output of the machine learning model could then be treated as the upsampled image generated through application of the method of FIG. 2. In this way, images at the full resolution of a slide-scanning system such as illustrated in FIG. 1A can be obtained in a fraction of the time that would be necessary if that system were operated at the synchronized scanning speed by operating the system at the (faster) compressing scanning speed, and then upsampling the resulting input image.

[0048] 3. Training a Neural Network to Generate Upsampled Images

[0049] Turning now to FIG. 5, that figure illustrates a method which could be used to train a neural network such as could be used in generating 203 an upsampled image in the method of FIG. 2. As shown in FIG. 5, such a training method may include obtaining 501 a set of full resolution images (e.g., by capturing using a slide scanning system such as shown in FIG. 1 A when it is operating at its synchronized scanning speed). A set of downsampled images may also be obtained 502. This may be done, for example, by, for each of the full resolution images, organizing its pixels into groups of pixels and then combining those pixels into groups of downsampled pixels in the same manner that capturing an input image at a compressing scanning speed would generate an input image where each pixel combined data from multiple regions of the underlying sample. Once the downsampled images had been obtained 502, a corresponding upsampled image could be obtained 503 from a downsampled by providing the downsampled image to the neural network being trained.

[0050] Once upsampled and full resolution images corresponding to a downsampled image were available, the upsampled and full resolution images could then be used to determine 504 a loss. This may include, for example, determining 505 a content loss based on the full resolution and upsampled images, such as by calculating the Euclidian distance between those images, or between feature maps of those images. As another example, in some cases determining 504 the loss may include providing 506 the upsampled and full resolution images to a second neural network. This second neural network may be a neural network which was trained to distinguish between upsampled images (i.e., images created by the neural network based on downsampled images) and original full resolution images, and the loss can include a loss reflecting if the second neural network was able to accurately identify the upsampled image as having been created by the network being trained (e.g., inverse of a probability given by the second neural network that the upsampled image was created by the neural network being trained). Combined losses, in which the neural network is trained to both minimize the content loss and maximize the probability that a second neural network will be unable to distinguish upsampled and full resolution images, are also possible and may be used in some cases.

[0051] However it takes place, once the loss has been determined 504, it can be used to train 507 the neural network, such as by backpropagating the loss through the neural network’s nodes and updating the weights of the connections between those nodes based on their respective contributions to the loss. Additionally, after one downsampled image had been processed, a determination 508 could be made as to whether the training should be continued with the next downsampled image. This may be done by, for example, determining if there were any further downsampled images that had not been processed and, if there were, moving to the next downsampled image and repeating the processing with that image. Other approaches are also possible. For example, in a case where the set of downsampled images was training data from a fold being used for K- fold cross validation, the determination 508 may be of whether any further downsampled images remained in the training data for that fold, and may conclude that the training process should not be repeated when the training data had been exhausted,even though further downsampled images may be present in the fold’s test data. Once the determination 508 was made, if the determination was to proceed with the next downsampled image, then the training could be repeated with the next downsampled image from the set of downsampled images. Otherwise, the training method of FIG. 5 could be treated as being done 509.

[0052] Of course, it should be understood that training a neural network to generate upsampled images may include acts other than those illustrated in FIG. 5, and the acts illustrated in FIG. 5 may be performed in different manners or contexts than those described above. For example, while the discussion of FIG. 5 described training the neural network with the loss function on a downsampled image by downsampled image basis, it is possible that approaches in which losses from different images are combined (e.g., mini-batch gradient descent), in which case training the neural network based on the loss for each downsampled image would be performed by combining that loss with one or more other losses and then updating the weights of the neural network based on the combined loss. Similarly, as noted above, it is possible that a neural network could be trained using K- fold cross validation, and in such a case there would be additional acts of obtaining sets of full resolution images and downsampled images for other folds, as well as for separating a fold in question into training data and testing data, though those additional acts are not illustrated in FIG. 5. Other variations on the training approach of FIG. 5 (e.g., making the determination 508 of whether to proceed with the next downsampled image conditional on whether the neural network appeared to be overfitting) are also possible and will be immediately apparent to those of skill in the art in light of this disclosure. Accordingly, the method of FIG. 5, and the discussion of training set forth in the context of that method, should be understood as being illustrative only, and should not be treated as limiting.

[0053] 4. Additional Variations

[0054] It should be understood that, while the above disclosure provided various examples of alternative approaches which may be taken in implementing the disclosed technology, those alternative approaches are intended to be illustrative only, and are not intended tobe an exhaustive description of all potential variations. For example, another variation which may be implemented in some cases relates to training a neural network where determining the loss involves providing both an upsampled image and a full resolution image to a second neural network. While the above description focused on the training for the first neural network (i.e., the neural network which would generate the upsampled image), it is possible in some cases that the second neural network (i.e., the neural network which would distinguish full resolution from upsampled images) may also be trained simultaneously with the first neural network, e.g., using a generative adversarial network approach. As another example of a variation on an approach which could be used in training neural networks to generate upsampled images, in some cases the neural networks may be trained using downsampled images which had had one or more groups of interpolation pixels added (e.g., as described above in the context of FIG. 4), rather than being trained directly on downsampled images (in which case the input images provided to the neural networks when performing a method such as shown in FIG. 2 would include the added groups of interpolation pixels, rather than simply be the input images as captured when operating at the compressing scanning speed). As another example, in some cases multiple neural networks may be trained for different sets of parameters which may be used on a scanning apparatus. For example, one neural network may be trained for using a compressing scanning speed which is twice the synchronized scanning speed with 20x magnification, and another might be trained for using a compressing scanning speed which is three times the synchronized scanning speed with 40x magnification. In such a case, when upsampled images were subsequently created using a method such as shown in FIG. 2, those images could be created using a neural network selected based on the parameters used in capturing the input images in that instance.

[0055] Additional variations are also possible in aspects other than how a neural network may be trained to create upsampled images. For example, in some cases, creating an upsampled image may be based on a whole slide image, where you would first capture the whole slide image (e.g., at a compressing scanning) and then apply an upsampling algorithm to the whole slide image. However, in other cases, an input image may be animage of a segment of a sample scanned while the camera is in motion at a constant velocity, and that input image may be upsampled and then added to upsampled images of other segments (e.g., segments captured after the field of view of the camera has been moved orthogonally to the scanning direction relative to the sample on the stage) to create a whole slide upsampled image. As another example, while the discussion above often focused on cases where the compressing scanning speed was twice the synchronized scanning speed, it is also possible that there may be a different relationship. For example, the compressing scanning speed may be three times the synchronized scanning speed, in which case there would be two groups of interpolation pixels added for each group of pixels in the input image, rather than one group as shown in the example of FIG. 4. There could also be variations in terms of the data that was actually used in upsampling. For example, in some cases, an image may be upsampled in only one dimension, but the upsampling may be based on data from multiple dimensions (e.g., not only on pixels which were neighbors in the dimension being upsampled, but also on pixels which neighbored each other in an orthogonal dimension), while in other cases upsampling in one dimension may be based only on information in that dimension (e.g., only on pixels which were neighbors in the dimension being upsampled).

[0056] Variations are also possible in physical components which may be used in implementing the disclosed technology. For example, while in some cases all steps of a method such as shown in FIG. 2 may be performed by a processor which is incorporated into the scanning system which captures the input images, in other cases some acts may be performed using a local processor, while other steps may be performed using a processor located remotely from the scanning system. For example, after capturing input images, a processor integrated into a scanning system may send those input images to a remote location (e.g., a cloud server) where the upsampling would be performed. As another example, while the disclosed technology was generally illustrated in the context of slide scanning, it may also be applied to increase the scanning speed of line scanning based imaging in other areas, such as machine vision, satellite imaging, and medical imaging.

[0057] As another example of how implementations of the disclosed technology may vary from the examples and illustrations set forth herein, consider that, in practice, the technology is likely to be implemented in contexts which involve the handling of much more information than described in the illustrative examples. For instance, while FIG. 3 illustrated a 4x4 sample that would result in a 4x4 input image when the input image is captured at the synchronized scanning speed, and FIG. 4 illustrated how a 4x2 image captured at a compressing scanning speed could be used to generate a 4x4 image through insertion of interpolation pixels, the disclosed technology is not limited to being applied in such small scale contexts. For example, FIG. 6 illustrates an application of the disclosed technology to an array of 20 micron diameter circles with 5 micron dots in their centers. When scanned at 40x magnification by a camera where each pixel represents 0.25 microns in both directions, a full resolution image 601 with the circles in the correct shapes is provided. However, when scanned at a compressing scanning speed which is twice as fast as the synchronized scanning speed, a distorted image 602 with 20x magnification in the dimension of the scanning direction and 40x magnification in the dimension orthogonal to the scanning dimension is obtained. Normally, such an image would be discarded as distorted. However, using the disclosed technology, an upsampled image 603 can be generated and subsequently used rather than discarded. More complex examples even than FIG. 6 are also possible. For instance, an actual sample (and therefore the corresponding input image) would likely be much larger, for example, 80,000x60,000 pixels. Accordingly, the example of FIG. 6, like the more simplified examples of FIGS. 3 and 4, should be understood as being illustrative only, and should not be treated as limiting. Other variations are also possible, and could be implemented by those of ordinary skill without undue experimentation based on this disclosure. Accordingly, the above description of variations, like the discussion which preceded it, should be understood as illustrative only, and should not be treated as limiting.

[0058] 5. Additional Non-Limiting Examples

[0059] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein can be applied to other embodiments without departing from the spirit or scope of the invention. Thus, it is to be understood that the description and drawings presented herein represent a presently preferred embodiment of the invention and are therefore representative of the subject matter which is broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses other embodiments that may become obvious to those skilled in the art and that the scope of the present invention is accordingly not limited. Other variations are also possible, and will be immediately apparent to those of skill in the art in light of this disclosure. For instance, the following examples are provided as concrete (though non-limiting) illustrations of various approaches which could be taken when implementing the disclosed technology.

[0060] Example 1

[0061] A processor enabled line scanning system for generating upsampled images, the system comprising: a camera having a line rate; a scanning stage configured to support a sample; a motor, the motor configured to establish a compressing scanning speed that is a relative motion in a first dimension between a field of view of the camera and the sample supported by the scanning stage; and one or more processors configured to: capture an input image of the sample based on operating the system at the compressing scanning speed, wherein: operating the system at the compressing scanning speed comprises establishing the relative motion in the first dimension at the compressing scanning speed between the field of view of the camera and the sample supported by the scanning stage; and the compressing scanning speed is greater than a synchronized scanning speed, wherein the synchronized scanning speed is synchronized with the line rate of the camera, pixel size of the camera, and magnification of an imaging system which comprises the camera; and generate an upsampled image based on applying an upsampling algorithm to the input image.

[0062] Example 2

[0063] The system of example 1 , wherein the compressing scanning speed is an integer multiple of the synchronized scanning speed.

[0064] Example 3

[0065] The system of any of examples 1-2 wherein the upsampling algorithm only upsamples the input image along the first dimension.

[0066] Example 4

[0067] The system of any of examples 1-3, wherein: the input image comprises a set of groups of pixels, wherein each pixel from each group of pixels has a value; the upsampling algorithm comprises: for each group of pixels from the set of groups of pixels, adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels, wherein, for each pixel from that group of pixels each group of pixels from the one or more groups of interpolation pixels corresponding to that group of pixels has a pixel corresponding to that pixel; and each pixel from the one or more groups of interpolation pixels corresponding to that group of pixels which corresponds to that pixel has a value which is equal to the value of that pixel; and applying an interpolation algorithm along the first dimension.

[0068] Example 5

[0069] The system of example 4, wherein the interpolation algorithm is selected from a group consisting of: nearest neighbor interpolation, linear interpolation, bilinear interpolation, cubic spline interpolation, bicubic spline interpolation, generalized linear interpolation, and generalized bicubic interpolation.

[0070] Example 6

[0071] The system of any of examples 4-5, wherein: the compressing scanning speed is twice the synchronized scanning speed; and for each group of pixels from the set of groups ofpixels: adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels is performed by adding one group of pixels next to that group of pixels in the first dimension; the cardinality of that group of pixels and the group of interpolation pixels corresponding to that group of pixels is the same; and for each pixel in that group of pixels: that group of pixels comprises all of that pixel’s neighbors in a dimension which is orthogonal to the first dimension, and does not comprise any of that pixel’s neighbors in the first dimension; and the pixel which corresponds to that pixel in the group of interpolation pixels neighbors that pixel in the first dimension.

[0072] Example 7

[0073] The system of any of examples 1-3, wherein applying the upsampling algorithm to the input image comprises providing the input image as input to a machine learning model.

[0074] Example 8

[0075] The system of example 7 wherein the machine learning model comprises a convolutional neural network.

[0076] Example 9

[0077] The system of any of examples 1-8, wherein: one or more processors comprises: a first processor configured to capture the input image of the sample based on operating the system at the compressing scanning speed; and a second processor configured to generate the upsampled image; the first processor, the camera, the motor and the scanning stage are located at a first location; the second processor is located at a second location; and the first processor is configured to communicate with the second processor over a wide area network connection.

[0078] Example 10

[0079] A method for generating upsampled images, the method comprising: establishing relative motion between a field of view of a camera and a sample supported by ascanning stage in a first dimension at a compressing scanning speed; capturing an input image of the sample based on establishing the compressing scanning speed as a speed that is greater than a synchronized scanning speed, wherein the synchronized scanning speed is synchronized with a line rate of the camera, a pixel size of the camera, and a magnification of an imaging system comprising the camera; and generating an upsampled image based on applying an upsampling algorithm to the input image.

[0080] Example 11

[0081] The method of example 10, wherein the compressing scanning speed is an integer multiple of the synchronized scanning speed.

[0082] Example 12

[0083] The method of any of examples 10-11, wherein the upsampling algorithm only upsamples the input image along the first dimension.

[0084] Example 13

[0085] The method of any of examples 10-12, wherein: the input image comprises a set of groups of pixels, wherein each pixel from each group of pixels has a value; the upsampling algorithm comprises: for each group of pixels from the set of groups of pixels, adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels, wherein, for each pixel from that group of pixels each group of pixels from the one or more groups of interpolation pixels corresponding to that group of pixels has a pixel corresponding to that pixel; and each pixel from the one or more groups of interpolation pixels corresponding to that group of pixels which corresponds to that pixel has a value which is equal to the value of that pixel; and applying an interpolation algorithm along the first dimension.

[0086] Example 14

[0087] The method of example 13, wherein the interpolation algorithm is selected from a group consisting of: nearest neighbor interpolation, linear interpolation, bilinear interpolation,cubic spline interpolation, bicubic spline interpolation, generalized linear interpolation, and generalized bicubic interpolation.

[0088] Example 15

[0089] The method of any of examples 13-14, wherein, for each group of pixels from the set of groups of pixels: adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels is performed by adding one group of pixels next to that group of pixels in the first dimension; the cardinality of that group of pixels and the group of interpolation pixels corresponding to that group of pixels is the same; and for each pixel in that group of pixels: that group of pixels comprises all of that pixel’s neighbors in a dimension which is orthogonal to the first dimension, and does not comprise any of that pixel’s neighbors in the first dimension; and the pixel which corresponds to that pixel in the group of interpolation pixels neighbors that pixel in the first dimension.

[0090] Example 16

[0091] The method of any of examples 10-12, wherein applying the upsampling algorithm to the input image comprises providing the input image as input to a machine learning model.

[0092] Example 17

[0093] The method of example 16 wherein the machine learning model comprises a convolutional neural network.

[0094] Example 18

[0095] A method of training a neural network to generate images which are upsampled along a first dimension, the method comprising: obtaining a set of full resolution images, wherein each full resolution image comprises a set of groups of pixels, wherein, for each group of pixels from the set of groups of pixels, for each pixel in that group of pixels: that group of pixels comprises all of that pixel’s neighbors in a first dimension; thatgroup of pixels comprises one or more other pixels at a same location as that pixel in a second dimension; obtaining a set of downsampled images based on, for each image from the set of full resolution images, creating a corresponding downsampled image wherein, for each group of pixels from the set of groups of pixels in the full resolution image, the downsampled image comprises a corresponding group of downsampled pixels wherein, for group of downsampled pixels, for each downsampled pixel in that group of downsampled pixels, that downsampled pixel has: a location in the second dimension; and a value equal to an average of values of pixels which are: in group of pixels for which that group of downsampled pixels is the corresponding group of downsampled pixels; and at a location in the second dimension which is equal to the location of that downsampled pixel in the second dimension; for each downsampled image from the set of downsampled images: obtaining a corresponding upsampled image based on providing that downsampled image to the neural network; determining a loss based on: the upsampled image corresponding to that downsampled image; and the full resolution image for which that downsampled image is the corresponding downsampled image; and training the neural network based on the loss.

[0096] Example 19

[0097] The method of example 18, wherein, for each downsampled image from the set of downsampled images, determining the loss comprises determining a content difference between: the upsampled image corresponding to that downsampled image; and the full resolution image for which that downsampled image is the corresponding downsampled image.

[0098] Example 20

[0099] The method of any of examples 18-19, wherein the neural network is a first neural network; and for each downsampled image from the set of downsampled images, determining the loss comprises providing: the upsampled image corresponding to that downsampled image; and the full resolution image for which that downsampled imageis the corresponding downsampled image to a second neural network, wherein the second neural network is trained to identify images created by the first neural network.

[0100] 6. Interpretation

[0101] None of the examples or illustrations set forth herein should be understood as implying limitations on the scope of any claims included in this document or any related document. Instead, the protection provided by this document or any related document, should be understood as being defined by the relevant document’s claims, when the terms in those claims which are explicitly defined herein are given their explicit definitions, and the terms which are not explicitly defined are given their broadest reasonable interpretation as provided by a general purpose dictionary.

[0102] Combinations, described herein, such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may contain one or more members of its constituents A, B, and / or C. For example, a combination of A and B may comprise one A and multiple B’s, multiple A’s and one B, or multiple A’s and multiple B’s.

[0103] A statement that something is “based on” something else should be understood as meaning that that thing is determined at least in part by that which it is “based on.” While the “based on” relationship includes scenarios in which one thing is completely determined by another, the “based on” relationship should not be understood as being limited to only scenarios in which one thing is completely determined by another unless the phrase used is “based exclusively on.”

Claims

CLAIMSWhat is claimed is:

1. A processor enabled line scanning system for generating upsampled images, the system comprising: a camera having a line rate; a scanning stage configured to support a sample; a motor, the motor configured to establish a compressing scanning speed that is a relative motion in a first dimension between a field of view of the camera and the sample supported by the scanning stage; and one or more processors configured to: capture an input image of the sample based on operating the system at the compressing scanning speed, wherein: operating the system at the compressing scanning speed comprises establishing the relative motion in the first dimension at the compressing scanning speed between the field of view of the camera and the sample supported by the scanning stage; and the compressing scanning speed is greater than a synchronized scanning speed, wherein the synchronized scanning speed is synchronized with the line rate of the camera, pixel size of the camera, and magnification of an imaging system which comprises the camera; and generate an upsampled image based on applying an upsampling algorithm to the input image.

2. The system of claim 1, wherein the compressing scanning speed is an integer multiple of the synchronized scanning speed.

3. The system of any of claims 1-2, wherein the upsampling algorithm only upsamples the input image along the first dimension.

4. The system of any of claims 1-3, wherein: the input image comprises a set of groups of pixels, wherein each pixel from each group of pixels has a value; the upsampling algorithm comprises: for each group of pixels from the set of groups of pixels, adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels, wherein, for each pixel from that group of pixels each group of pixels from the one or more groups of interpolation pixels corresponding to that group of pixels has a pixel corresponding to that pixel; and each pixel from the one or more groups of interpolation pixels corresponding to that group of pixels which corresponds to that pixel has a value which is equal to the value of that pixel; and applying an interpolation algorithm along the first dimension.

5. The system of claim 4, wherein the interpolation algorithm is selected from a group consisting of: nearest neighbor interpolation, linear interpolation, bilinear interpolation, cubic spline interpolation, bicubic spline interpolation, generalized linear interpolation, and generalized bicubic interpolation.

6. The system of any of claims 4-5, wherein: the compressing scanning speed is twice the synchronized scanning speed; andfor each group of pixels from the set of groups of pixels: adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels is performed by adding one group of pixels next to that group of pixels in the first dimension; the cardinality of that group of pixels and the group of interpolation pixels corresponding to that group of pixels is the same; and for each pixel in that group of pixels: that group of pixels comprises all of that pixel’s neighbors in a dimension which is orthogonal to the first dimension, and does not comprise any of that pixel’s neighbors in the first dimension; and the pixel which corresponds to that pixel in the group of interpolation pixels neighbors that pixel in the first dimension.

7. The system of any of claims 1-6, wherein applying the upsampling algorithm to the input image comprises providing the input image as input to a machine learning model.

8. The system of claim 7 wherein the machine learning model comprises a convolutional neural network.

9. The system of any of claims 1-8, wherein: one or more processors comprises: a first processor configured to capture the input image of the sample based on operating the system at the compressing scanning speed; and a second processor configured to generate the upsampled image; the first processor, the camera, the motor and the scanning stage are located at a first location; the second processor is located at a second location; and the first processor is configured to communicate with the second processor over a wide area network connection.

10. A method for generating upsampled images, the method comprising: establishing relative motion between a field of view of a camera and a sample supported by a scanning stage in a first dimension at a compressing scanning speed; capturing an input image of the sample based on establishing the compressing scanning speed as a speed that is greater than a synchronized scanning speed, wherein the synchronized scanning speed is synchronized with a line rate of the camera, a pixel size of the camera, and a magnification of an imaging system comprising the camera; and generating an upsampled image based on applying an upsampling algorithm to the input image.

11. The method of claim 10, wherein the compressing scanning speed is an integer multiple of the synchronized scanning speed.

12. The method of any of claims 10-11, wherein the upsampling algorithm only upsamples the input image along the first dimension.

13. The method of any of claims 10-12, wherein: the input image comprises a set of groups of pixels, wherein each pixel from each group of pixels has a value; the upsampling algorithm comprises: for each group of pixels from the set of groups of pixels, adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels, wherein, for each pixel from that group of pixelseach group of pixels from the one or more groups of interpolation pixels corresponding to that group of pixels has a pixel corresponding to that pixel; and each pixel from the one or more groups of interpolation pixels corresponding to that group of pixels which corresponds to that pixel has a value which is equal to the value of that pixel; and applying an interpolation algorithm along the first dimension.

14. The method of claim 13, wherein the interpolation algorithm is selected from a group consisting of: nearest neighbor interpolation, linear interpolation, bilinear interpolation, cubic spline interpolation, bicubic spline interpolation, generalized linear interpolation, and generalized bicubic interpolation.

15. The method of any of claims 13-14, wherein, for each group of pixels from the set of groups of pixels: adding, next to that group of pixels in the first dimension, one or more groups of interpolation pixels corresponding to that group of pixels is performed by adding one group of pixels next to that group of pixels in the first dimension; the cardinality of that group of pixels and the group of interpolation pixels corresponding to that group of pixels is the same; and for each pixel in that group of pixels: that group of pixels comprises all of that pixel’s neighbors in a dimension which is orthogonal to the first dimension, and does not comprise any of that pixel’s neighbors in the first dimension; andthe pixel which corresponds to that pixel in the group of interpolation pixels neighbors that pixel in the first dimension.

16. The method of any of claims 10-15, wherein applying the upsampling algorithm to the input image comprises providing the input image as input to a machine learning model.

17. The method of claim 16 wherein the machine learning model comprises a convolutional neural network.

18. A method of training a neural network to generate images which are upsampled along a first dimension, the method comprising: obtaining a set of full resolution images, wherein each full resolution image comprises a set of groups of pixels, wherein, for each group of pixels from the set of groups of pixels, for each pixel in that group of pixels: that group of pixels comprises all of that pixel’s neighbors in a first dimension; that group of pixels comprises one or more other pixels at a same location as that pixel in a second dimension; obtaining a set of downsampled images based on, for each image from the set of full resolution images, creating a corresponding downsampled image wherein, for each group of pixels from the set of groups of pixels in the full resolution image, the downsampled image comprises a corresponding group of downsampled pixels wherein, for group of downsampled pixels, for each downsampled pixel in that group of downsampled pixels, that downsampled pixel has: a location in the second dimension; and a value equal to an average of values of pixels which are: in group of pixels for which that group of downsampled pixels is the corresponding group of downsampled pixels; and at a location in the second dimension which is equal to the location of that downsampled pixel in the second dimension;for each downsampled image from the set of downsampled images: obtaining a corresponding upsampled image based on providing that downsampled image to the neural network; determining a loss based on: the upsampled image corresponding to that downsampled image; and the full resolution image for which that downsampled image is the corresponding downsampled image; and training the neural network based on the loss.

19. The method of claim 18, wherein, for each downsampled image from the set of downsampled images, determining the loss comprises determining a content difference between: the upsampled image corresponding to that downsampled image; and the full resolution image for which that downsampled image is the corresponding downsampled image.

20. The method of any of claims 18-19, wherein the neural network is a first neural network; and for each downsampled image from the set of downsampled images, determining the loss comprises providing: the upsampled image corresponding to that downsampled image; and the full resolution image for which that downsampled image is the corresponding downsampled image to a second neural network, wherein the second neural network is trained to identify images created by the first neural network.

Images