Image compression and analysis

Abstract

Efficient image analysis, processing, and manipulation based on adaptive particle representation (APR) for processing large scale color image datasets are disclosed. Analysis of large biomass or tissue samples is achieved with more than 100 multiple fast computations and improved memory and storage compression ratios. Embodiments are well suited for large 3D transparent tissue samples, large scale imaging items, such as whole brain mapping plans and human neurohistopathology. A data storage section stores adaptive sampling representations of tiles, individual tile positions, and corresponding other image data of biomass of interest. A full-resolution complete image supporting biomass is output using an image obtained by loading tiles from a data storage unit at a single time.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 366,818, filed on June 22, 2022, and U.S. Provisional Application No. 63 / 366,825, filed on June 22, 2022. The entire teachings of the above applications are incorporated herein by reference. Background Art

[0003] Biological and medical imaging is a subset of digital imaging and refers to the technical field of generating and rendering images of internal body parts and growths, including tissues, organs, bone structures, tendons, ligaments, and some blood vessels. There are a variety of medical image generation technologies and machines, such as confocal and light sheet microscopy, X-ray systems, CT (computed tomography) scanners, MRI (magnetic resonance imaging) machines, ultrasound systems, and PET (positron emission tomography) scanners, among others. The rendered images are digital visualizations that enable doctors to diagnose and treat patients, and more specifically, allow doctors to detect diseases, abnormalities, and injuries. For example, a CT scan provides a series of images of 2D slices or cross-sections of the body, allowing doctors to digitally view a tumor, its shape, size, and location, and the blood vessels that supply the tumor on a computer display.

[0004] Researchers and attending physicians rely on biological and medical images. The typical model of these imaging processes, and digital imaging systems in general, includes image acquisition (e.g., by one of the machines or technologies described above), archiving of the acquired raw digital images, retrieval, processing (analysis), transmission / distribution, and display. The image data of certain biological or medical research and their corresponding digital files can be relatively large, which can pose problems for computer memory storage (archiving) and the speed of computer retrieval, processing (analysis), and display. For example, color images of tissue samples of a whole mouse brain or large human brain sections, as well as other annotated medical images, are relatively large digital files. Various image compression algorithms exist, but they are mainly used to store files, because reading and understanding the contents of the file requires decompressing it, which loses the benefits of compression. Summary of the invention

[0005] Embodiments of the present invention address shortcomings in the areas of manipulation (processing for analysis, transmission / distribution, display, etc.) and management (storage in computer memory, archiving, retrieval, transmission, etc.) of certain medical image data, particularly large volume, multi-dimensional image data. Embodiments advantageously provide faster processing and improved ability to store image files in computer memory without requiring upgrades in processor resources or memory.

[0006] Embodiments provide computer-based systems and computer-implemented methods for improved multi-dimensional image / medical image / biomedical image manipulation (e.g., processing analysis, distribution, display) and management (e.g., storage, retrieval, transmission, etc.). In particular, embodiments enable relatively large image data storage (archiving), retrieval, analysis, processing, and display at acceptable speeds without requiring (in a free manner) additional memory or processor resources. Relatively large image data include (non-limiting examples) biological and medical images generated by any of the following: confocal and light sheet microscopy (other optical microscopy includes but is not limited to wide field, super resolution, and electron microscopy), X-ray, computed tomography (CT), positron emission tomography (PET), optical coherence tomography (OCT), MRI, ultrasound, and other digital imaging techniques.

[0007] Image data may be multi-dimensional and include, in addition to the digitized subject image: color, annotations (e.g., text, notes, and / or markings by a researcher or physician), graphics, chronological or time dimensions, etc.

[0008] The subject of relatively large image data is, by way of non-limiting example, a tissue sample, a whole mouse brain, a human brain slice, other mammalian brains or parts thereof, or other biological masses or materials of interest.

[0009] In one embodiment, for given image data of a biological mass, a computer-implemented image manipulation method obtains, accesses or otherwise receives a sequence of cross-sectional images (typically two-dimensional image units) of increasing depth along an axis in a third spatial dimension. Each two-dimensional image unit or cross section is an nxm pixel frame in a corresponding parallel plane orthogonal to the axis. The method obtains different sequences of two-dimensional image units with increasing depth along the corresponding orthogonal axes. Each two-dimensional unit of the sequence is an nxm pixel frame located in a corresponding parallel plane orthogonal to the corresponding axis of the sequence. The corresponding orthogonal axes of different sequences are located at different frame positions across the corresponding planes. As a result, the method obtains multiple sequences (so-called tiles) that represent image volumes that capture different parts of the biological mass in three spatial dimensions. Different sequences or tiles of the resulting multiple sequences represent different image volumes obtained from the given image data.

[0010] Therefore, as used herein, a "frame" is a two-dimensional (2D) nxm array of pixel data. Multiple such 2D arrays (frames) taken consecutively along the same orthogonal axis constitute a 3D data array, referred to herein as a "tile". Different tiles (consecutive frames) have different orthogonal axes along which corresponding consecutive frames are taken. To reiterate, one tile is formed by corresponding consecutive frames taken along one orthogonal axis. A second tile is formed by corresponding consecutive frames taken along another orthogonal axis. The consecutive frames of a third tile are taken along the orthogonal axis of the tile, and so on.

[0011] Next, during image acquisition, the method automatically and independently digitally processes each tile (sequence) by:

[0012] i. compressing image data of low information content tile areas and maintaining full resolution or high resolution of high information content areas, thereby generating an adaptively sampled representation of the tile;

[0013] ii. Automatically determine the transformation parameters of the block;

[0014] iii. Automatically compare the compressed data with the original data to verify that the converted data captures the required information;

[0015] iv. Calculate the maximum intensity projection of the image block in multiple spatial directions and for each edge of the image block that has a neighboring image block;

[0016] v. aligning the tile with other neighboring tiles by computing pairwise registrations between neighboring tiles in 3 spatial dimensions using the computed maximum intensity projections. After image acquisition, an embodiment globally optimizes the tile positions with respect to a reliability metric. The resulting relative tile positions enable stitching to be performed with a single load of each tile and to reconstruct the image volume; and

[0017] vi. storing the adaptively sampled representation of the tile, the determined transformation parameters of the tile, and an indication of the spatial position of the tile in a data storage in a computer memory. The step of storing is performed in a manner where the data storage stores individual tile positions and a reduced amount of given image data rather than storing raw data of a biological quality source image.

[0018] As has not been achieved so far in the prior art, embodiments provide efficient image processing (using compressed form of image data for analysis) and scalability (analyzing tiles independently), as described below. During image acquisition, for a given tile, the method: (i) applies a classifier in a manner to segment objects from a given tile, including extracting object features and object positions, and (ii) aggregates and stores the segmented objects in a data storage unit. Objects that appear two or more times on adjacent tile areas are automatically merged with the help of a matching algorithm, a non-limiting example of which is as follows: (a) find the nearest neighbor in a feature space taking into account the position (in the final sample space, after alignment), size and shape of each object; and (b) use the Lowe criterion to check whether the match is reliable (the second nearest neighbor must be farther than a predefined margin). Other matching algorithms are also applicable. Next, embodiments may subsequently (after acquiring the source image) stitch together and reconstruct the subject image using a single load of the individual tiles represented in or from the corresponding data in the data store (i.e., stored compressed image data, stored relative tile positions, compression parameters, stored segmented objects, and stored object positions). In this way, embodiments avoid the prior art problem of applying object segmentation and other analyses to the entire image file, which may become too large to manipulate or save.

[0019] Finally, the method uses a single load of tiles from the data store (i.e., compressed image data and corresponding data) (rather than loading tiles multiple times or at different times) to form an image output that supports a complete image at full resolution of biological quality. Using the stored compressed image data, stored relative tile positions, compression parameters, stored segmented objects, and stored object positions from the data store can alleviate the need for additional computer memory and / or processor resources for any of the following: storage, retrieval, transmission, distribution, processing, analysis, display, and combinations thereof of the complete image.

[0020] In an embodiment, compression of low information content tile regions includes employing an adaptive particle representation technique.

[0021] A computer-implemented image manipulation method, comprising: (A) for given image data of a biological mass, obtaining a sequence of two-dimensional image units of increasing depth along an axis in a third spatial dimension, each two-dimensional image unit being an nxm pixel frame in a corresponding parallel plane orthogonal to the axis, obtaining different sequences of two-dimensional image units of increasing depth along corresponding orthogonal axes, each two-dimensional image unit in a sequence being an nxm pixel frame in a corresponding parallel plane orthogonal to the corresponding axis of the sequence, different sequences having different orthogonal axes at different tile locations across the corresponding planes, the obtaining step producing a plurality of sequences, the plurality of sequences serving as tiles and representing image volumes capturing different portions of the biological mass in three spatial dimensions, different sequences in the plurality of sequences being different tiles representing different image volumes;

[0022] In some embodiments, during source image acquisition (and in other embodiments, at another time), method (B) automatically processes tile by tile, i.e., processes each tile independently by: (i) compressing image data of low information content tile regions and maintaining high (full) resolution of high information content regions, thereby generating an adaptively sampled representation of the tile, (ii) determining a relative tile position with respect to other tiles, (iii) segmenting the adaptively sampled representation of the tile for an object of interest, the step of segmenting including extracting object features and corresponding object locations, and (iv) storing the adaptively sampled representation of the tile, an indication of the determined tile position, the segmented object, the extracted corresponding object features and the object location in a data storage in a computer memory. The step of storing is performed in such a way that the data storage stores individual tile positions and a reduced amount of the given image data rather than storing the biological quality source image.

[0023] As a next step, the method: (C) aggregates the segmented objects stored in the data store, wherein the step of aggregating includes automatically merging segmented objects that appear two or more times on adjacent tiles to generate a working list of segmented objects stored in the data store. And (D) based on the adaptive sampling representation of tiles, tile locations, and working list of segmented objects stored in the data store, the method loads tiles once and forms a complete image of a portion or all of the biological mass at full resolution. To reiterate, the method does not require loading tiles multiple times or at different times in order to facilitate storage and performance of image analysis (e.g., object segmentation). As configured, the step of loading tiles once alleviates the need for additional computer memory and / or processor resources for any of the following: storage, retrieval, transmission, distribution, processing, analysis, display, and combinations thereof of the complete image.

[0024] The worklist may include indications of segmented objects, object locations, and corresponding features.

[0025] In an embodiment, the step of independently processing each tile further includes: (i) automatically determining compression parameters for compressing the image data of the tile; (ii) storing the determined compression parameters in the data storage unit; and (iii) automatically comparing the resulting adaptive sampling representation and the source image data in a manner to verify that the adaptive sampling representation captures the required information of the given image data.

[0026] In an embodiment, the step of determining the relative tile position of a tile comprises: (i) calculating the maximum intensity projection of the tile in multiple spatial directions and for each edge of the tile that exists in a neighboring tile; (ii) calculating tile pairwise registration parameters with the neighboring tiles that have been acquired; and (iii) storing the calculated tile pairwise registration parameters in the data storage unit. The calculation of the tile pairwise registration parameters is completed in the following manner: for each edge of the neighboring tile that exists in a corresponding manner, the calculated maximum intensity projection of the edge of the tile is associated with the calculated maximum intensity projection of the corresponding edge of the corresponding neighboring tile that is acquired. As a result, the method improves the accuracy of the relative tile position of the tile relative to the neighboring tiles.

[0027] In an embodiment, the tile registration further comprises a global optimization. Specifically, after image acquisition, the method globally optimizes all stored pairwise registration results with respect to a reliability metric such that all tile positions are accurately corrected. The increased accuracy and correction of relative tile positions further enables: (a) splicing of image data corresponding to the tiles, (b) correct reconstruction of the entire biomass; and (c) reliable extraction of high-level information about the imaged biomass, such as, but not limited to, the volume of the biomass and the number of segmented objects inside. In an embodiment, step C (aggregation of segmented objects) is performed after the global optimization.

[0028] In an embodiment, the given image data includes biological or medical images acquired and generated by any of the following items: optical microscopy, electron microscopy (including but not limited to wide field, confocal, super resolution, electron and light sheet microscopy), X-ray, computed tomography, positron emission tomography, optical coherence tomography, magnetic resonance imaging, ultrasound and other digital imaging techniques. For the subject block, steps A and B are completed during the acquisition of the biological or medical image, and step C is performed after the image corresponding to the subject block is acquired.

[0029] In embodiments, the biological mass is any of the following: tissue samples, whole mouse brain, human brain slices, other mammalian brain parts, and other biological materials of interest.

[0030] In an embodiment, one or more digital processors perform the steps of the method described herein in an automated manner. Other embodiments provide computer-based systems, computer program products, and software as a service (SaaS). A computer-based system embodying the present invention can be formed by an interface, a data storage unit in a computer memory, and a processor executable work module operatively connected between the interface and the data storage unit. The interface receives or accesses subject image data of a biological mass of interest. The work module responds to the interface and is configured (programmed) to automatically perform image compression on the image data through adaptive sampling representation (e.g., APR), tile registration, and object segmentation and aggregation. For each tile, the data storage unit saves the adaptive sampling representation, the determined relative tile position, the segmented object and the corresponding object position, and the extracted features. From the content of the data storage unit, the system generates a full-resolution complete image of part or all of the biological mass using a single load of the tile. The single load of the tile alleviates the need for additional computer memory and / or processor resources for any of the following items: manipulating or managing the storage, retrieval, transmission, distribution, processing, analysis, display, or any combination of the complete image.

[0031] In an embodiment, the stored adaptively sampled representations of the individual tiles are merged and cropped over overlapping regions so that a physical location in the biomass is uniquely described by a single particle in the adaptively sampled representations of the tiles stored in the data store. For a given overlapping region (where two or more tiles overlap), the adaptively sampled representation of the overlapping region is merged into one of the tiles and cropped from the remainder of the two or more tiles. The merged and cropped version of the adaptively sampled representation of the tiles is stored in the data store. In this way, the adaptively sampled representations of the tiles (merged and cropped over overlapping regions) provide unprecedented efficiency in image data storage and manipulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The foregoing will become apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings, in which like reference numerals refer to the same parts throughout the different views. The drawings are not necessarily drawn to scale, emphasis instead being placed upon illustrating the embodiments.

[0033] Figure 1Figure 1 is a graph of APR (Adaptive Particle Representation) stitching times evaluated for different computation ratios (CR) and compared to TeraStitcher stitching times. The computation ratio is the ratio of the total number of voxels before compression to the total number of particles after compression.

[0034] Figure 2 is a graph illustrating a merge of segmented objects on multi-tile data of a subject's medical image. The number of objects detected by the software in the medical image is plotted relative to the number of true objects. The number of objects detected on the raw image data before slicing it to create the multi-tile dataset is also reported. The calculated ratio is the ratio of the total number of voxels before compression to the total number of particles after compression.

[0035] Figure 3A is a block diagram of an implementation scheme, with the keys in the bottom row.

[0036] Figure 3B It is a flow chart of an embodiment of the present invention.

[0037] Figure 4 is a schematic diagram of a computer network environment in which embodiments may be deployed.

[0038] Figure 5 yes Figure 4 A block diagram of a computer node in a computer network.

[0039] Fig. 6A and Figure 6B It is a schematic diagram of merging and cropping image data of overlapping blocks in an embodiment of the present invention. Fig. 6A Four overlapping tiles are illustrated, overlapping adjacent tiles to the left, right, above, and below. Figure 6B Illustrated Fig. 6A The tiles in , where the adaptive sampling representation of the tiles is modified by merging and cropping overlapping areas. DETAILED DESCRIPTION

[0040] A description of example implementations follows.

[0041] Adaptive particle representation (APR) is an efficient, content-adaptive image representation technique that also provides data compression. APR adaptively represents the content of an image while maintaining image quality. By providing data compression, APR not only reduces image storage costs, but also advantageously overcomes memory and processing bottlenecks by being used directly for image processing tasks without conversion back to pixel data. See BL Cheeseman et al., "Adaptive particle representation of fluorescence microscopy images," Nature Communications 9, no. 5160, Dec. 4, 2018.

[0042] For a subject image or image volume, APR replaces pixels or voxels with particles (points in space that carry intensities) that are positioned according to the image content. Particles can be placed anywhere the image content requires, and particles can have different sizes in different parts of the image. The different sizes define the corresponding resolution at which the image is represented locally. The required resolution is represented by an implicit resolution function, which assigns high resolution to image regions where the intensity varies rapidly in space (e.g., edges) and assigns low resolution to image regions where the intensity varies less (e.g., background or uniform foreground). The implicit resolution function defines the radius of a neighborhood around each pixel. At any pixel location, the image intensity value can be reconstructed within a user-set error threshold by taking an arbitrary non-negative weighted average of the intensities of the particles contained within this neighborhood. Since lighting conditions (and therefore the intensity range) can vary greatly between different regions within a sample, the errors considered are normalized relative to the local intensity scale.

[0043] To reiterate, APR uses spatially adaptive sampling to resample an image guided by local information content, while using local gain control. The output from applying APR to an image is a representation of the image formed by a set of particles with associated intensity values.

[0044] The principles of the present invention utilize adaptive particle representation techniques for efficient image analysis and processing of relatively large image data sets. Embodiments enable analysis of large biological masses or tissue samples with more than 100 times faster computational rates and improved memory and storage compression ratios. Embodiments are well suited for large-scale medical or biomedical imaging projects, such as whole-brain mapping projects and human neurohistopathology, as non-limiting examples. Embodiments are also well suited for sequential analysis of the same sample type (e.g., mouse brain) when experiments must be repeated for statistical purposes.

[0045] Figure 3A and Figure 3BThe data flow and control flow in the method and system 1000 embodying the principles of the present invention are illustrated. Although illustrated and described as an end-to-end pipeline from a source imager to a final useful image for distribution, storage and communication, the embodiments are not limited thereto. Different embodiments or parts thereof may be used as image compression tools or devices, image storage and retrieval methods and systems, image analysis and post-processing methods and systems, etc. For the purpose of illustration and not limitation, in this case, the following discussion Figure 3A and Figure 3B , and then further describes the implementation details.

[0046] Figure 3A The keys in the bottom row include hardware indicated in blue (dark gray or standard diagonal pattern); units that act on individual tiles indicated in orange (medium gray or regular cross-hatching); advanced units (acting on tiles and other data) indicated in green (medium dark gray or dotted pattern); and external libraries that work on voxel data indicated in light gray (or pure white background). According to the key, the hardware elements of an embodiment include imaging device 101 and data storage 125. Units that act on individual tiles include: APR compressor (with automatic conversion parameters) 121, segmenter 123, and pre-stitcher (with projector and pairwise registration) 122. Advanced units include: visualizer 305, stitcher (with global optimizer) 310; matcher (intelligent merging of segmented objects) 130; and reconstructor 320. Examples of external libraries include Atlaser 350 and other algorithms that work on voxel data 355.

[0047] In the embodiment system 1000, a source imager 101 (e.g., a microscope) acquires and generates a biological or medical image 103 of a subject's biological mass (e.g., an organ, body part, tissue, or biological material of interest). Those skilled in the art will appreciate that a variety of source imagers (confocal and light sheet microscopes, OCT scanners, CAT scanners, PET scanners, ultrasound systems, MRI systems, X-ray systems, etc.) 101 may be utilized. Other optical microscopy techniques or electron microscopy techniques may be used as the source imager 101 (optical type microscopes), including but not limited to wide field microscopy, super resolution microscopy, and electron microscopy. Other digital imaging techniques are also applicable. The original biological or medical image 103 ( Figure 3B ) can be large-scale (or relatively large in volume compared to commonly seen digital images). In some embodiments, the medical or biomedical image 103 is a large human brain slice, a whole mouse brain, or a portion or the entirety of another mammalian brain (non-limiting examples).

[0048] Source 101 acquires raw biomedical image 103 and provides it to local memory, computer storage, sharable memory, archive, etc. (commonly referred to as working memory, Figure 3B ). In the working memory 105, researchers, doctors and other users can add annotations (text notes, marks, etc.), graphics and / or colors to the original biological or medical image 103. Therefore, the resulting image data 110 is formed of multi-dimensional information, that is, the original biomedical image 103 plus annotations, graphics and colors. Each pixel of the image data 110 is represented by an n-tuple. The N-tuple data representation is implemented by a linked list, array or other data structure, which stores multi-dimensional information for each pixel. The different dimensions (or tuples) of information for each pixel can include: the original source image (digital image pixel value), annotations or graphics (overlay values), color (red, green, blue factor values) and other layers or types of information, such as (non-limiting example) a second layer or more layers of text or graphics, fluorescent channel indicators, time dimensions, blood pressure or blood flow indications, other physical conditions, etc. In one embodiment, one channel or tuple is focused on, so each pixel has one value instead of multiple values.

[0049] The system and method 1000 of the present invention obtains, accesses or otherwise receives image data 110 (per pixel n-tuple data) of a subject's biological quality from a working memory 105 in near real time during or after acquisition by a source 101. In particular, the interface 120 ( Figure 3B ) receives image data 110 and accesses or otherwise obtains therefrom a sequence of cross-sectional images (typically two-dimensional image units) of increasing depth along an axis in a third spatial dimension. Each two-dimensional image unit is an nxm pixel frame located in a corresponding parallel plane orthogonal to the axis in the third dimension. A sequence of frames along a subject axis is called a tile. Interface 120 obtains different tiles, i.e., a sequence of cross-sectional images (two-dimensional image units or frames) along a corresponding axis. Each two-dimensional image unit of a sequence (tile) is an nx m pixel frame located in a corresponding parallel plane orthogonal to the corresponding axis of the sequence. Different sequences or tiles are located at different tile positions across corresponding planes.

[0050] As a result, interface 120 obtains a plurality of such sequences or tiles. Each sequence / tile in the plurality of sequences / tiles represents an image volume capturing a corresponding portion of a biological mass of a subject in image data 110 in three spatial dimensions (3D space). Different sequences / tiles represent different image volumes, and different image volumes capture different portions of a biological mass of a subject.

[0051] The tile processing modules 121, 122, 123, 124 are connected to communicate with the interface 120. During the source image acquisition, for each sequence / tile obtained at the interface 120, the tile processing modules 121, 122, 123, 124 process each tile as follows. First, the tile processing module at step 121 compresses the tile areas with low information content and maintains the full resolution of the areas with high information content. This will produce an adaptively sampled representation of the tile. In an embodiment, this image compression and adaptive sampling of the tile is accomplished using APR. Other adaptive sampling techniques are also applicable.

[0052] Next, the tile processing module 121 automatically determines the conversion parameters for compressing the image data of the tile. In an embodiment, the conversion parameters (or compression parameters in this article) are automatically determined by the algorithms and techniques detailed below. In some embodiments, the tile processing module 121 can independently use APR on different channels or dimensions (tuples) of the image data 110 of the tile. For a non-limiting example, the tile processing module 121 can independently apply APR to the fluorescence channel and automatically determine the corresponding compression parameters for the same tile. Module steps 121, 124 store the compressed tile (i.e., the adaptive sampling representation of the tile) and the corresponding determined compression parameters in the data storage unit 125.

[0053] Module 121 also verifies the compressed representation of the tile by automatically comparing the resulting adaptively sampled representation with the source image data. The comparison determines whether the adaptively sampled representation captures the required information of the source image data. Known or common techniques are used in the verification comparison at module step 121, such as peak signal-to-noise ratio, structural similarity index, etc.

[0054] Embodiments may provide improvements to the compressed image data representation of overlapping regions of adjacent tiles stored in data store 125. In embodiments, module 121 stores each tile individually in data store 125, and data store 125 contains raw compressed data (adaptively sampled representation) where each adjacent tile has an overlapping region with its neighbors. In such a configuration, a physical location in a biological mass (physical sample being imaged) is not always uniquely defined by a single particle (of the adaptive sampling representation) in data store 125, but may be defined by 2 particles (adjacent neighboring regions) or 4 particles (angular overlapping regions). Fig. 6A This phenomenon is illustrated, where 4 different tiles (represented by different colors) overlap.

[0055] Fig. 6AThe schematic diagram of is illustrative, where the numbers 1, 2 and 4 indicate the number of compressed image particles (adaptively sampled representation particles) that define a spatial position in a biomass. Specifically, a block or quadrant 600 of four tiles is shown. In practice, there may be more tiles, but the principle remains the same. The upper left tile 610 overlaps with the adjacent tile 620 on the right (i.e., the upper right tile of the block 600) along the length of its right edge. The upper left tile 610 overlaps with the adjacent tile 630 below (i.e., the lower left tile of the block 600) along its bottom edge. In a similar manner, the upper right tile 620 overlaps with the upper left tile 610 along the left edge. The upper right tile 620 overlaps with the adjacent tile 640 below (i.e., the lower right tile of the block 600) along its bottom edge. The lower left tile 630 of the quadrant (block 600) overlaps with the upper left tile 610 along its upper edge. The lower left tile 630 overlaps with the lower right tile 640 along its right edge. The lower right tile 640 overlaps with the lower left tile 630 along its left edge. The lower right tile 640 overlaps with the upper right tile 620 along its upper edge.

[0056] Each non-overlapping tile area is labeled with a number 1 to indicate that a single particle (of the adaptive sampling representation of the corresponding tile) in the data storage 125 uniquely defines a spatial location represented in the subject's biological mass (the physical sample being imaged). Each area where two tiles overlap is labeled with a number 2 to indicate that two particles (one from each compressed image / adaptive sampling representation of adjacent neighboring tiles) in the data storage 125 define a corresponding spatial location in the subject's biological mass. For non-limiting example, the overlapping area 615 between the upper left tile 610 (along its right edge) and the upper right tile 620 (along its left edge) (excluding or subtracting their overlapping inner corner area toward the center of the tile 600) is labeled with a number 2, indicating that two particles (one from each of the two sets of particles from the corresponding adaptive sampling representations of tiles 610 and 620) in the data storage 125 define a corresponding location or spatial location in the biological mass of interest. Similarly, the overlapping area 625 (between the upper right tile 620 and the lower right tile 640), the overlapping area 635 (between the upper left tile 610 and the lower left tile 630), and the overlapping area 645 (between the lower left tile 630 and the lower right tile 640) minus the corresponding overlapping inner corner areas are marked with the number 2 to indicate that two particles in the data storage unit 125 (the corresponding sets of particles from the adaptive sampling representation of two adjacent neighboring tiles) define corresponding positions (spatial positions) in the biomass.

[0057] The overlapping areas of all four tiles 610, 620, 630, 640 (their inner corners, i.e., the lower right corner of the upper left tile 610, the lower left corner of the upper right tile 620, the upper right corner of the lower left tile 630, and the upper left corner of the lower right tile 640) are labeled with the number 4 to indicate that four particles from the corresponding sets of particles corresponding to the adaptive sampling representations of the four adjacent neighboring tiles 610, 620, 630, 640 in the data storage unit 125 define corresponding spatial locations or locations in the biomass.

[0058] Interesting refinements in other embodiments are: a) merging particle information of overlapping tile regions to store the merged result in the data store 125, and b) cropping the tiles (representations thereof) so that a spatial location or point location in the sample (biological mass) of interest is uniquely defined by a particle in one stored tile, i.e., uniquely defined by a single particle in the data store 125. Fig. 6A For example, Figure 6B The improved embodiment shown in merges 690 particle information (from adaptively sampled or other compressed image data representations) on overlapping tile areas by (i) retaining the smallest particle (representing the highest resolution) in the union of particle sets corresponding to overlapping tile areas of the subject. Next is the merging method 690: (ii) for each particle set involved in the overlap, calculate the average intensity value of the particles in the set (the average value for each tile area), (iii) across the overlapping tile areas of the subject, select the maximum value of the calculated average intensity values, and (iv) assign the result of (iii) as the intensity value to the smallest particle retained in step (i). Finally, method 690 crops the affected tiles (their representations) so that any location in the subject's biomass is uniquely defined by a single particle in the tile representation stored in the data storage unit 125.

[0059] exist Figure 6BIn the example of , method 690 retains or merges the overlapping regions 615 and 635 of the upper left tile 610, and removes the associated overlapping regions of the upper right adjacent tile 620 and the lower left adjacent tile 630. That is, method 690 applies the above steps (i)-(iv) to the overlapping regions 615 and 635 and the inner corners of tile 610, and defines particles in the lightly shaded regions shown along the right edge and the bottom edge of tile 610. Method 690 crops out the corresponding overlapping regions along the left edge of the upper right tile 620 and crops out the corresponding overlapping regions along the upper edge of the lower left tile 630. Module 121 implementing method 690 stores the resulting compressed image data / adaptive sampling representation of tile 610 in data storage 125. Similarly, method 690 applies steps (i)-(iv) to the overlapping region 625, and defines particles in the lightly shaded region shown along the bottom edge of the upper right tile 620. Method 690 merges the particles and light shaded regions so defined into the upper right tile 620 and crops the corresponding overlapping regions from the upper edge of the lower right tile 640. Method 690 (module 121) stores the resulting compressed image data / adaptive sampling representation of the upper right tile 620 in data storage 125 and stores the cropped adaptive sampling representation of the lower right tile 640 in data storage 125. Method 690 applies steps (i)-(iv) to the overlapping region 645 and defines particles in the light shaded region shown along the right edge of the lower left tile 630. Method 690 merges the particles and light shaded regions so defined into the lower left tile 630 and crops the corresponding overlapping regions from the left edge of the lower right tile 640. Method 690 (module 121) stores the resulting compressed image data / adaptive sampling representation of the lower left tile 630 and the compressed image data / adaptive sampling representation of the lower right tile 640 in data storage 125.

[0060] According to the above improved implementation, except for the tiles along the acquired rightmost edge and the tiles along the acquired bottom edge, all tiles (their adaptive sampling representations) ultimately stored in the data storage unit 125 by module 121 and method 690 have the same size. Figure 6B The lighter shades of illustrate where the merging and cropping is done (by module 121 applying method 690). In particular, the size of the upper left tile 610 (including the shaded areas 615, 635 merged with the tiles) remains unchanged. The upper right tile 620 (including the shaded area 625 merged with it) is cropped to a narrower size. The length of the lower left tile 630 (including the shaded area 645 merged with it) is cropped. The lower right tile 640 is cropped in width and length.

[0061] Back to Figure 3B, in step 122, the tile processing module determines the relative tile position relative to other tiles. To achieve this, for a given tile, step 122 calculates the maximum intensity projection of the tile in multiple spatial directions and for each edge with a neighboring tile. Next, the tile processing module step 122 uses the already acquired neighboring tiles to calculate the tile pairwise registration parameters. Specifically, for each edge of a given tile for which there is a corresponding acquired neighboring tile, step 122 associates the calculated maximum intensity projection of the tile edge with the calculated maximum intensity projection of the corresponding edge of the corresponding acquired neighboring tile. By associating the corresponding calculated maximum intensity projections of the relevant edges of a given tile and its neighboring tiles, the embodiment provides a higher accuracy of the relative tile position of the tile relative to its neighboring tiles. Module steps 122, 124 store the calculated paired registration parameters in the data storage unit 125. In an embodiment, the tile processing module 122 registers the tiles with other tiles by: (a) obtaining pairwise registration between adjacent tiles using the calculated maximum intensity projection during source image acquisition, and (b) performing a global optimization on the reliability metric after source image acquisition to calculate the optimal position of each tile using a single load of each tile. More details on image stitching are discussed below.

[0062] In an embodiment, tile-by-tile processing continues during source image acquisition, and the tile processing module step 123 segments the object of interest from the adaptively sampled representation of the tile (output from step 121). Module 123 applies a classifier in such a way that the object is segmented from a given tile, including extracting object features and object locations. Object features may include (non-limiting examples) the diameter, shape, volume, etc. of the detected object. In some embodiments, module 123 further uses the tile positions output from step 122 and stored in the data storage unit 125 to calculate the position of the segmented object. In an embodiment, the classifier can be formed by an image filtering operation followed by thresholding. In other embodiments, the classifier is formed by manually annotating a subset of the tiles after compressing the tile area, and training a machine learning classifier to perform segmentation directly on the given tile at step 123. Other classifiers are also suitable for object segmentation, as will be further discussed below.

[0063] In a preferred embodiment, step 123 stores in data store 125: (i) the output of the classifier (typically object segmentation), which is a binary mask computed directly in APR (hence very efficiently), and (ii) an indication of the physical aspects and characteristics of the segmented object, i.e., position and features, computed from the binary mask. Storing the binary mask together with the compressed image data (adaptively sampled representation of the tiles) in data store 125 is very space efficient compared to twice the memory space required to store the segmented object (image) and the uncompressed subject image.

[0064] For each processed tile, the tile processing module step 124 stores in a data store 125 in a computer memory: (a) an adaptively sampled representation of the tile output from step 121, (b) compression parameters for the tile determined from step 121, (c) an indication of the location of the tile in 3D space, i.e., the relative tile location of step 122, and (d) the segmented objects of interest output from step 123. In one embodiment, the data store 125 stores a list of segmented objects, extracted object features, and object location statistics across tiles. In this way, the data store 125 stores individual tile locations and reduced amounts of image data 110, rather than storing raw data of the source image 103 of the subject's biological mass.

[0065] like Figure 3A As shown, in some embodiments, the effect of the tile processing modules 121, 122, 123, 124 configures the data store 125 as part of an image compression device or tool or service 310, 130, 320. In embodiments, specific portions of the image data 110 can also be selectively read and displayed 305 from the compressed form stored in the data store 125 by a user. See below for more details on the visualization 305. In embodiments, the interface 120 and tile processing modules 121, 122, 123, 124 are completed during image acquisition at the source machine 101, while the remaining processing of the system / method 1000 is completed after acquisition of the source image 103, as will be made clear below.

[0066] continue Figure 3B, the aggregation module 130 is communicatively connected to the data storage unit 125. The aggregation module 130 aggregates the segmented objects in the data storage unit 125 and forms a working list that replaces the initial statistical list. For objects that appear two or more times on adjacent tile areas, the aggregation module 130 automatically merges the objects together with the help of a matching algorithm. For non-limiting examples, an embodiment (aggregation module 130) adopts a nearest neighbor type matching algorithm, in which each object is embedded in a feature space with its position, size and shape. The ratio between the nearest neighbor and the second nearest neighbor can be used to reliably verify the match; if the ratio is above a predefined threshold (usually 0.7), the match is considered reliable and the matched objects are merged.

[0067] Other matching algorithms or means for merging objects at module or step 130 are suitable.

[0068] Back to Figure 3B , the system / method 1000 employs the merger 140 (also referred to as the reconstructor 320) of the response aggregation module 130, Figure 3A ). The merger (module or step) 140, 320 merges the tiles of the adaptively sampled representation using a single load of the tiles (i.e., as represented by the data storage unit 125 data) and thereby forms an output component (file, stream, etc.) or output image (generally speaking) 150 of a full-resolution complete image of the subject's biological mass. The merging or reconstruction 320 performed by the merger 140 includes stitching the tiles together. The merging method supports conversion from the merged tiles back to voxel data of the image volume in the working memory 105. The output image 150 is formed by the method of loading the tiles in a single time without the need for additional computer memory and / or processor resources to perform any of the following operations: storage, retrieval, transmission, distribution, processing, analysis, display of the complete image 150, and combinations thereof. Depending on the needs of certain applications, the merger 140 / reconstructor 320 may also obtain the complete image 150 at a lower resolution. Other embodiments alternatively directly merge the adaptively sampled representation, i.e., APR files. See below for more details on image merging.

[0069] As non-limiting examples, embodiments may interface with a registration pipeline, working algorithms, other systems, etc. at 350, 355. The registration pipeline aligns images of the same class of biological subjects or masses. With such registration, the output data file (full image) 150 is aligned with similar subject images in a common coordinate frame, and the segmented objects (from the aggregation module 130) are mapped to corresponding objects in the registration pipeline. The alignment and mapping advantageously enables counting of cells in a subject's body part, etc. Other uses and advantages of the embodiments are within the knowledge of those skilled in the art, based on the present disclosure.

[0070] The following describes different embodiments of the present invention. Figure 3A and Figure 3B Further implementation details of additions or variations of the modules, steps and elements.

[0071] Automatic image conversion

[0072] The adaptive sampling of APR is based on the calculation of the gradient in the original image signal relative to the local intensity scale used to normalize the error. Both the gradient and the local intensity scale must be estimated numerically from the acquired image. This is very sensitive to noise, causing high-frequency fluctuations to appear in areas where the underlying (true) signal is completely flat. The current implementation uses a smooth B-spline approximation of the signal to reduce the effects of noise on the gradient and local intensity scale.

[0073] The conversion from voxel images to APRs depends on many parameters. For a given optical system or experiment, most of them can be reliably kept fixed (e.g., voxel size, background intensity level, and smoothing in the B-spline calculation). However, smoothing is not sufficient to completely eliminate the effects of noise in the signal. Therefore, in order to avoid oversampling due to normalization of small gradients, the gradients and the local intensity scale are thresholded from below. This introduces two additional threshold parameters, one for each quantity. Compared to the other parameters, the optimal values ​​of these thresholds are highly dependent on the local signal content. Therefore, automatic estimation of these parameters is necessary for large-scale experiments where manual tuning is not possible.

[0074] The optimal threshold is the threshold that best separates the distribution of values ​​(gradients and local intensity scales) from actual signal and fluctuations caused by noise. Lower thresholds lead to adaptation to noise and suboptimal compression, while higher values ​​lead to undersampling and information loss. In principle, any automatic thresholding algorithm (local or global) can be used to estimate the optimal value. However, since the distribution is highly dependent on both the optical setup and the local characteristics of the sample, algorithms that rely on strict assumptions about the form of the distribution are not suitable. The implementation adopts the minimum cross entropy thresholding algorithm of Li et al.

[10] because of its proven robustness, simplicity and computational efficiency.

[0075] Image stitching 122, 310

[0076] The samples are scanned and acquired using a raster scan pattern or any pattern that allows covering the entire desired area, where the tiles are acquired sequentially. Our pipeline supports sparse tiling as long as each tile is connected to other tiles by at least one of its edges, and is adapted for APR by taking inspiration from [1]. A neighbor graph (i.e., edge neighbors of each tile) is first computed for each tile. Then, for each pair of neighboring tiles, a maximum intensity projection (I x ,Iy ,I z ) to calculate the displacement (x, y, z) of these patches. Depending on the characteristics of the biological or medical image, other alignment methods can also be used to improve the performance. Such alignment methods include but are not limited to feature (e.g., SIFT) matching (e.g., RANSAC), and machine learning methods.

[0077] If artifacts are present in the volume (e.g., bubbles embedded around the sample), a masked version of phase cross-correlation is used [3].

[0078] The maximum intensity projections are pre-computed (only on the expected overlapping areas to eliminate noise in the phase cross-correlation) to allow stitching to be performed with a single loading of the individual tiles. In doing so, a total of 6 displacements are calculated for each pair of adjacent tiles (2 in the x direction, 2 in the y direction, and 2 in the z direction). The most reliable one is retained, where reliability is calculated as the minimum square difference between the registered maximum projections. Reliability can also be calculated using other metrics, such as but not limited to the ratio between the maximum value of the difference between the registered and original maximum intensity projections and twice the maximum value of the original image, which the applicant has found to be very effective for sparse samples.

[0079] The displacements are stored in three graphs (one for each spatial dimension), where each vertex corresponds to a given tile and each edge corresponds to the displacement between two tiles. A corresponding reliability graph is constructed for each spatial dimension. Finally, the splicer 310 performs a global optimization of each displacement graph using a maximum spanning tree on the reliability graph to satisfy the internal problem constraint that the sum of each cycle in the graph should be 0.

[0080] For stitching, the individual steps are done entirely with APR, except for computing the phase cross-correlation. For this step, the maximum intensity projection is computed on the APR and then the 2D pixel image is reconstructed. In this case, regenerating the pixel data does not cause a disadvantage, since it is only 2D, so it takes up very little memory.

[0081] Finally, as mentioned above Figure 3A and Figure 3B As mentioned in , the tile positions are stored in the data store 125 and may be used later to calculate the position of the segmented object or for visualization at 305 .

[0082] Object Segmentation

[0083] To segment objects such as labeled cells, a particle classification strategy is used in one embodiment. A random forest with 100 estimators is trained on a small set of data that was previously manually sparsely annotated [4].

[0084] The proposed pipeline does not merge tiles before performing segmentation, since segmentation is done directly after fully acquiring one tile while acquiring the next one. Object features and locations are extracted, and a merging strategy inspired by [5] is developed to merge high-level information without actually merging APR data. For each neighboring tile, the two nearest neighbors (NN) of each object in the overlapping area of ​​the first tile are calculated from the set of objects in the overlapping area of ​​the second tile. If the ratio between the second NN and the first NN is greater than a threshold (usually 0.7) and if the object center is closer than a threshold distance (usually one-quarter of the object size), the objects are considered identical and automatically merged. Objects that touch the edge of the overlapping area are automatically discarded to avoid duplicate counting. The strategy has been validated on synthetic datasets such as Figure 2 shown.

[0085] Figure 2 is a diagram illustrating the merging of segmented objects on multi-patch data of a subject medical or biomedical image. Synthetic multi-patch data sets are randomly created with varying numbers of objects and random shifts. The individual patches are segmented individually and then the objects are merged on the overlapping regions (after calculating the correct registration as described above). The number of detected objects is plotted against the true number of objects. Figure 2 The chart also shows the number of detected objects on the original data (before being sliced ​​to create a multi-tile dataset) for reference.

[0086] Image Merging

[0087] Certain applications, such as registering samples to an atlas 350, etc. 355 require merged volumes. Fortunately, these applications 350, 355 typically require a lower sample resolution, avoiding the need to regenerate the space occupied by the original image data. Simply choose the right APR level to perform efficient voxel reconstruction of the individual tiles at a lower resolution, and interpolate later if the exact voxel size is required. The tiles are then merged using any merging strategy, such as taking the average or maximum of overlapping areas. Preprocessing (such as histogram equalization) can be performed before conversion and merging to benefit from APR computation acceleration.

[0088] Registered to atlas 350

[0089] The merged mouse brain can be registered to the Allen Mouse Brain Atlas [6] using the AMAP pipeline [7] via the Brainreg front-end [8].

[0090] Visualization 305

[0091] In one embodiment, Napari [9] is used as a front end for visualization 305. It requires an object with a slice attribute to allow lazy loading, thereby avoiding the need to regenerate a full sample in voxel space. The so-called "lazy loading" is to read the compressed image data (output from step 121 and stored in the data storage 125) to display it.

[0092] Stitching benchmark

[0093] The stitching speed is compared with TeraStitcher [1], steps 1-5 (corresponding to finding stitching parameters without merging the data). Multiple instances of TeraStitcher can be run simultaneously using openmpi, and the best performance is achieved using 4 cores. The applicant created a synthetic dataset consisting of 4x4 tiles of 2048x512x512 unit16 voxels (about 1Gb / tile) with different computation ratios (by varying the number of objects). The computation ratio is the ratio of the total number of voxels before compression to the total number of particles after compression. The voxel data was converted to APR using the automatic parameter determination described above (see the Automatic Image Conversion section).

[0094] Reference Figure 1 , the APR stitching time was evaluated for different computation ratios (CR) and compared with that of TeraStitcher. Figure 1 The graph of FIG. 1 shows that the stitching time of the embodiments is 20 to 1000 times faster than that of TeraStitcher.

[0095] In FIG. 6 of the priority application, which is incorporated herein by reference in its entirety, the accompanying drawings further illustrate the APR conversion, stitching, segmentation, and atlas registration in the embodiments. An end-to-end pipeline for analyzing large 3D cleared tissue samples of whole mouse brain or large human brain slices is shown. The disclosed system and method achieves more than 100 times faster computation.

[0096] The stitching step or module in the above end-to-end pipeline performs stitching directly on the subject image representation generated by APR. The stitching module supports arbitrary tiling patterns. Advantageously, the stitching module is very efficient and each tile only needs to be read once.

[0097] The segmentation module of the above end-to-end pipeline operates directly on the subject image representation generated by APR and segments (extracts) objects, object features, and object locations from the subject image. Users can efficiently create manual annotations directly on the representation generated by APR.

[0098] Finally, atlas registration 350 is shown in the example end-to-end pipeline. Lower resolution pixel data can be reconstructed and merged directly from the multi-tile APR representation of the subject image. This enables an atlas registration pipeline (e.g., AMAP via Brainreg (as a front end)) to be used and mapped back to APR data.

[0099] As a result, methods and systems embodying the present invention provide an efficient APR-based image analysis pipeline for processing large 3D data sets with 20 to 1000 times faster speeds (increased processing rates) and reduced storage (memory) requirements relative to existing solutions. Such implementations are particularly suitable for large imaging projects.

[0100] The Appendix to the priority application (incorporated herein by reference in its entirety) provides further details of embodiments of the present invention.

[0101] Computer Support

[0102] Figure 4 It illustrates a computer network or similar digital processing environment in which embodiments of the present invention may be implemented.

[0103] The client computers / devices 50 and server computers 60 provide processing, storage, and input / output devices for executing application programs, kernel programs, etc. The client computers / devices 50 may include imaging sources, such as the medical imaging machine 101 discussed above. The client computers / devices 50 may also be linked to other computing devices, including other client devices / processors 50 and server computers 60, via a communication network 70. The communication network 70 may be part of a remote access network, a global network (e.g., the Internet), a cloud computing server or service, a global collection of computers, a local area network or a wide area network, and gateways that currently communicate with each other using corresponding protocols (TCP / IP, Bluetooth, etc.). Other electronic device / computer network architectures are also applicable.

[0104] Figure 5 yes Figure 4 1 and 1 . A diagram of the internal structure of a computer (e.g., a client processor / device 50 or a server computer 60) in a computer system. Each computer 50, 60 includes a system bus 79, where a bus is a set of hardware lines used to transfer data between components of a computer or processing system. The bus 79 is essentially a shared pipe that connects the different components of the computer system (e.g., processor, disk storage device, memory, input / output ports, network ports, etc.) to enable information to be transferred between the components. Attached to the system bus 79 is an I / O device interface 82 for connecting various input and output devices (e.g., keyboard, mouse, display, printer, speakers, etc.) to the computer 50, 60. The network interface 86 allows the computer to connect to a computer attached to a network (e.g., Figure 4 The memory 90 provides volatile storage for computer software instructions 92 and data 94 used to implement the embodiments of the present invention (e.g., as described above in Figure 3A , Figure 3B , Fig. 6A and Figure 6B The disk storage device 95 provides non-volatile storage for computer software instructions 92 and data 94 used to implement embodiments of the present invention. The central processing unit 84 is also attached to the system bus 79 and provides execution of computer instructions.

[0105] In one embodiment, the processor routine 92 and data 94 are a computer program product (generally referred to as 92) including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROMs, CD-ROMs, floppy disks, magnetic tapes, etc.) that provides at least a portion of the software instructions for the system of the present invention. The computer program product 92 can be installed by any suitable software installation process, as is known in the art. In another embodiment, at least a portion of the software instructions can also be downloaded via a cable, communication and / or wireless connection. In other embodiments, the program of the present invention is a computer program propagation signal product 107 embodied on a propagation medium (e.g., radio waves, infrared waves, laser waves, sound waves, or electric waves propagated through a global network such as the Internet or other networks). Such a carrier medium or signal provides at least a portion of the software instructions of the routine / program 92 of the present invention.

[0106] In alternative embodiments, the propagation signal is an analog carrier wave or a digital signal carried on a propagation medium. For example, the propagation signal may be a digitized signal propagated through a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagation signal is a signal transmitted through a propagation medium over a period of time, such as instructions of a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of the computer program product 92 is a propagation medium that can be received and read by the computer system 50, such as by receiving the propagation medium and identifying the propagation signal embodied in the propagation medium, as described above for the computer program propagation signal product.

[0107] In general, the term "carrier medium" or transient carrier encompasses the aforementioned transient signal, propagation signal, propagation medium, storage medium, etc.

[0108] In other embodiments, program product 92 may be implemented as so-called software as a service (SaaS), or to support other installations or communications with end users.

[0109] In view of the above description and details, embodiments of the present invention provide many advantages over prior art methods and approaches. For example, embodiments segment objects of interest from image data during source image acquisition rather than after acquisition. This novel approach achieves scalability and generates readily accessible results (image objects) at the end of image acquisition.

[0110] Embodiments advantageously utilize a single load of individual tiles (i.e., stored compressed image data, stored relative tile positions, compression parameters, stored segmented objects, and stored object positions) to stitch and reconstruct a subject image. In contrast, prior art methods require loading tile data twice, i.e., once for evaluating alignment parameters and then at the end stage to merge all tiles to create an image file. Prior art methods may apply object segmentation and other analyses to the image file. However, merging all tiles is not necessarily a scalable approach because at some point the image file may become too large to manipulate or save. Applicants address these shortcomings of the prior art and keep individual tiles separate. Instead, Applicants' method merges segmented object information when needed (e.g., in Figure 3A Applicants' method scales well because individual tiles can be processed independently (or together with their neighbors) as long as tile processing keeps up with the data throughput of source machine 101, which is achievable when using an adaptive sampling representation of tiles (e.g., APR).

[0111] The teachings of all patents, published applications, and references cited herein are incorporated by reference in their entirety.

[0112] References

[0113] [1]Bria,A.&Iannello,G.TeraStitcher-a toolfor fast automatic 3d-stitching of teravoxel-sized microscopy images.BMC Bioinformatics 13(2012).https: / / doi.org / 10.1186 / 1471-2105-13-316.

[0114] [2] Guizar-Sicairos, M., Thurman, S. T. & Fienup, J. R. Efficient subpixel image registration algorithms. Optics Letters 33, 156 (2008). https: / / doi.org / 10.1364 / ol.33.000156.

[0115] [3] Padfield, D. Masked object registration in the fourier domain. IEEE Transactions on Image Processing 21, 2706{2718 (2012). https: / / doi.org / 10.1109 / tip.2011.2181402.

[0116] [4] Fernández-Delgado, M., Cernadas, E., Barro, S. & Amorim, D. Do we need hundreds of classifiers to solve realworld classification problems?Journal of Machine Learning Research 15, 3133{3181 (2014). http: / / jmlr.org / papers / v15 / delgado14a.html.

[0117] [5] Lowe, D. G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91{110 (2004). https: / / doi.org / 10.1023 / b:visi.0000029664.99615.94.

[0118] [6] Lein, E. S. et al. Genome-wide atlas of gene expression in the adult mouse brain. Nature 445, 168{176 (2006). https: / / doi.org / 10.1038 / nature05453.

[0119] [7] Niedworok, CJ et al. aMAP is a validated pipeline for registration and segmentation of high-resolution mouse brain data. Nature Communications 7 (2016). https: / / doi.org / 10.1038 / ncomms11879.

[0120] [8] Tyson, AL et al. Tools for accurate post hoc determination of markerlocation within whole-brain microscopy images (2021). https: / / doi.org / 10.1101 / 2021.05.21.445133.

[0121] [9] Sofroniew, N. et al. napari / napari:0.4.9rc0 (2021). URL https: / / zenodo.org / record / 3555620.

[0122]

[10] Li, CH and Tam, PKS, 1998. Aniterative algorithm for minimum crossentropy thresholding. Patternrecognitionletters, 19(8), pp.771-776.

[0123] While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments as encompassed by the appended claims.

Claims

1. A computer-implemented image manipulation method, the image manipulation method comprising the following steps: A. for given image data of a biological mass, obtaining a sequence of two-dimensional image units of increasing depth along an axis in a third spatial dimension, each two-dimensional image unit being a frame of nxm pixels in a corresponding parallel plane orthogonal to said axis, obtaining different sequences of two-dimensional image units of increasing depth along corresponding orthogonal axes, each two-dimensional image unit in a sequence being a frame of nxm pixels in a corresponding parallel plane orthogonal to the corresponding axis of the sequence, different sequences having different orthogonal axes at different tile positions across the corresponding planes, the obtaining step producing a plurality of sequences, the plurality of sequences serving as tiles and representing image volumes capturing different portions of the biological mass in three spatial dimensions, different sequences in the plurality of sequences being different tiles representing different image volumes; B. During source image acquisition, each tile is automatically and independently processed by a computer processor in the following manner: i) compressing image data of low information content tile regions and maintaining high resolution of high information content regions, thereby producing an adaptively sampled representation of the tile, ii) determine the relative tile position with respect to other tiles, iii) segmenting the adaptively sampled representation of the image block for an object of interest, the segmentation step comprising extracting object features and corresponding object locations, iv) storing said adaptively sampled representation of the tile, the determined tile positions, the segmented objects, the extracted object features and the object positions in a data storage in a computer memory, the step of storing being performed in such a way that said data storage stores individual tile positions and a reduced amount of said given image data instead of storing said biological quality source image; C. aggregating the segmented objects stored in the data storage unit, wherein the step of aggregating includes automatically merging segmented objects that appear two or more times on adjacent tiles, thereby generating a working list of segmented objects stored in the data storage unit; as well as D. Based on the adaptive sampling representation of tiles, tile positions and working list of segmented objects stored in the data storage unit, a single loading of tiles and thereby forming a full-resolution complete image of part or all of the biological mass, the step of loading tiles in a single manner alleviates the need for additional computer memory and / or processor resources for any of the following items: storage, retrieval, transmission, distribution, processing, analysis, display and combinations thereof of the complete image.

2. The method according to claim 1, wherein: The steps to process each tile independently also include: automatically determining compression parameters for compressing image data of the image tile; storing the determined compression parameters in the data storage unit; and The resulting adaptively sampled representation is automatically compared to the source image data in a manner that verifies that the adaptively sampled representation captures desired information for the given image data.

3. The method according to claim 1, wherein: The steps for determining the relative tile position of a tile include: computing a maximum intensity projection of the adaptively sampled representation of the tile in a plurality of spatial directions and for each edge of the tile that has a neighboring tile; calculating a pairwise registration parameter of a tile with an already acquired and adaptively sampled neighboring tile by: for each edge of a correspondingly acquired neighboring tile of the tile, associating a calculated maximum intensity projection of the edge of the tile with a calculated maximum intensity projection of a corresponding edge of the correspondingly acquired neighboring tile, thereby improving the accuracy of the relative tile position of the tile with respect to the neighboring tile; and The calculated image patch pairwise registration parameters are stored in the data storage unit.

4. The method according to claim 3, further comprising: After image acquisition, all stored pairwise registration results are globally optimized with respect to a reliability metric such that all tile positions are accurately corrected to enable: (a) stitching of image data corresponding to the tiles, (b) correct reconstruction of the entire biological mass; and (c) reliable extraction of high-level information about the imaged biological mass, such as, but not limited to, the volume of the biological mass and the number of segmented objects inside; and Among them, step C is performed after global optimization.

5. The method according to claim 4, wherein: Stitching is performed using a single load of each tile.

6. The method according to claim 1, wherein: The given image data includes biological or medical images acquired and generated by any of the following: optical microscopy, electron microscopy, X-ray, computed tomography, positron emission tomography, optical coherence tomography, magnetic resonance imaging, ultrasound and other digital imaging techniques; and Wherein, for the subject image block, step A and step B are completed during acquisition of the biological or medical image, and step C is performed after acquiring the image corresponding to the subject image block.

7. The method according to claim 1, wherein: The given image data also includes any one or a combination of the following items: color, annotation, text, graphics, timing characteristics, etc.

8. The method according to claim 1, wherein: The biological mass is any of the following: tissue samples, whole mouse brain, human brain slices, other mammalian brain parts, and other biological materials of interest.

9. The method according to claim 1, further comprising: In addition to compressing tile regions, different fluorescence channels of the given image data are also compressed independently.

10. The method according to claim 1, further comprising: In addition to compressing tile regions, the given image data is also compressed across different time periods.

11. The method according to claim 1, wherein: The step of compressing the low information content tile regions includes using an adaptive particle representation (APR) technique.

12. The method according to claim 1, wherein: The step of segmenting applies a classifier to the adaptively sampled representation of the tile, and the classifier is formed by one of: (i) an image filtering operation followed by thresholding, and (ii) Machine learning algorithms trained on similar data, such as but not limited to random forests, support vector machines, and neural networks.

13. The method according to claim 1, wherein: The step of aggregating the segmented objects automatically merges the objects through a matching algorithm. 14 . The method of claim 1 , further comprising calculating the position of a segmented object using the tile positions stored in the data store.

15. The method of claim 1 , further comprising displaying any one or a combination of: (a) an image of the biological mass based on the stored adaptive sampling representation of the tile and the tile position stored in the data storage unit, and (b) an image of the segmented object based on the data of the segmented object stored in the data storage unit.

16. The method of claim 1, further comprising interfacing with a registration pipeline that aligns volumes of biological masses in a common coordinate frame, and mapping the segmented object in the common coordinate frame.

17. The method according to claim 1, wherein: The full image is alternatively or optionally displayed at a lower resolution.

18. The method according to claim 1, further comprising: A specific portion of the given image data is read from the adaptively sampled representation and displayed.

19. The method according to claim 1, wherein: The stored adaptively sampled representations of the individual tiles are merged and cropped over overlapping tile areas such that a physical location in the biomass is uniquely described by a single particle in the adaptively sampled representation of the tile stored in the data store.

20. A computer-based biomedical image manipulation system, the biomedical image manipulation system comprising: A. An interface for receiving or accessing image data of a subject of a biological mass, the interface obtaining a sequence of two-dimensional image units of increasing depth along an axis in a third spatial dimension, each two-dimensional image unit being an nxm pixel frame in a respective parallel plane orthogonal to the axis, the obtaining comprising obtaining different sequences of two-dimensional image units of increasing depth along respective orthogonal axes, each two-dimensional image unit in a sequence being an nxm pixel frame in a respective parallel plane orthogonal to the respective axis of the sequence, different sequences having different orthogonal axes at different tile locations across the respective planes, the obtaining producing a plurality of sequences, the plurality of sequences acting as tiles and representing image volumes capturing different portions of the biological mass in three spatial dimensions, different sequences in the plurality of sequences being different tiles representing different image volumes; B. A tile processing module, which is communicatively coupled to the interface and which is automatically executed by a digital processor during image acquisition and processes each tile independently by: i) compressing image data of low information content tile regions and maintaining full resolution of high information content regions, thereby producing an adaptively sampled representation of the tile, ii) determine the relative tile position with respect to other tiles, iii) segmenting the adaptively sampled representation of the image block for an object of interest, the segmentation comprising extracting object features and corresponding object locations, and iv) storing the adaptively sampled representation of the tile, the determined tile positions, the segmented objects, the extracted object features and the object positions in a data store in a computer memory, the storing being performed in such a manner that the data store stores individual tile positions and a reduced amount of the subject image data rather than storing the source image of the biological quality; C. an aggregation module responsive to the tile processing module, the aggregation module aggregating the segmented objects stored in the data storage unit, wherein aggregating includes automatically merging segmented objects that appear two or more times on adjacent tiles, thereby generating a working list of segmented objects, object locations, and features stored in the data storage unit; as well as D. An output component having the adaptive sampling representation of tiles stored in the data storage unit, tile positions, and the contents of a work list of segmented objects, and the output component is configured so that a single loading of tiles thereby forms a complete image of part or all of the biological mass at full resolution, the single loading of tiles alleviating the need for additional computer memory and / or processor resources for any of the following items: storage, retrieval, transmission, distribution, processing, analysis, display, and combinations thereof of the complete image.

21. The computer-based biomedical image manipulation system of claim 20, wherein: The interface receives or accesses the subject image data from a source machine, a computer memory, an archive, or any combination thereof.

22. The computer-based biomedical image manipulation system of claim 20, wherein: The subject image data includes biological or medical images acquired and generated by a source machine using any of the following: X-ray, computed tomography, positron emission tomography, optical microscopy, electron microscopy, optical coherence tomography, magnetic resonance imaging, ultrasound, and other digital imaging techniques; and The tile processing module is configured to process the acquisition of the biological or medical image in near real time.

23. The computer-based biomedical image manipulation system of claim 20, wherein: The subject image data also includes any one or a combination of the following items: color, annotations, text, graphics, timing features, etc.

24. The computer-based biomedical image manipulation system of claim 20, wherein: The biological mass is any of the following: tissue samples, whole mouse brain, human brain slices, other mammalian brain parts, and other biological materials of interest.

25. The computer-based biomedical image manipulation system of claim 20, wherein: The tile processing module independently applies adaptive sampling compression to different fluorescence channels of the subject image data.

26. The computer-based biomedical image manipulation system of claim 20, wherein: The tile processing module applies adaptive sampling compression to the subject image data across different time periods.

27. The computer-based biomedical image manipulation system of claim 20, wherein: The tile processing module uses adaptive particle representation (APR) technology to compress image data of tile areas with low information content.

28. The computer-based biomedical image manipulation system of claim 20, wherein: The system also allows for reading and displaying specific portions of the subject image data from the adaptively sampled representation stored in the data store.

29. The computer-based biomedical image manipulation system of claim 20, wherein: The complete image from the output component may optionally or alternatively be displayed at a lower resolution.

30. The computer-based biomedical image manipulation system of claim 20, further comprising a registration interface that enables: (i) interfacing with a registration pipeline that aligns volumes of various biological masses in a common coordinate system, and (ii) mapping the segmented object in the common coordinate system.

31. The computer-based biomedical image manipulation system of claim 20, wherein: The tile processing module segments the object by applying a classifier to the adaptively sampled representation, the classifier being formed by one of: (i) an image filtering operation followed by thresholding, and (ii) Machine learning algorithms trained on similar data, such as but not limited to random forests, support vector machines, and neural networks.

32. The computer-based biomedical image manipulation system of claim 20, wherein: The aggregation module that aggregates the segmented objects automatically merges the objects through a matching algorithm.

33. The computer-based biomedical image manipulation system of claim 20, wherein: The tile processing module also uses the tile positions stored in the data storage to calculate the positions of the segmented objects.

34. The computer-based biomedical image manipulation system of claim 20, further comprising a display module coupled to display an image of the biological mass based on the stored adaptive sampling representation of tiles, tile locations stored in the data storage unit, and data of the segmented object stored in the data storage unit.

35. The computer-based biomedical image manipulation system of claim 20, wherein: The tile processing module stores in the data store the adaptive sampling representations of individual tiles merged and cropped over overlapping tile regions such that a physical location in the biomass is uniquely described by a single particle in the adaptive sampling representations of tiles in the data store.

36. A computer-implemented image manipulation method, the image manipulation method comprising the steps of: A. for given image data of a biological mass, obtaining a sequence of two-dimensional image units of increasing depth along an axis in a third spatial dimension, each two-dimensional image unit being an nxm pixel frame in a respective parallel plane orthogonal to said axis, obtaining different sequences of two-dimensional image units of increasing depth along respective orthogonal axes, each two-dimensional image unit in a sequence being an nx m pixel frame in a respective parallel plane orthogonal to the respective axis of the sequence, different sequences having different orthogonal axes at different tile positions across the respective planes, the obtaining step producing a plurality of sequences, the plurality of sequences serving as tiles and representing image volumes capturing different portions of the biological mass in three spatial dimensions, different sequences in the plurality of sequences being different tiles representing different image volumes; B. Each tile is processed independently and automatically by a computer processor in the following manner: i) compressing image data of low information content tile regions and maintaining high resolution of high information content regions, thereby producing an adaptively sampled representation of the tile, ii) determine the relative tile position with respect to other tiles, iii) segmenting the adaptively sampled representation of the image block for an object of interest, the segmentation step comprising extracting object features and corresponding object locations, iv) storing said adaptively sampled representation of the tile, the determined tile positions, the segmented objects, the extracted object features and the object positions in a data storage in a computer memory, the step of storing being performed in such a way that said data storage stores individual tile positions and a reduced amount of said given image data instead of storing said biological quality source image; C. aggregating the segmented objects stored in the data storage unit, wherein the step of aggregating includes automatically merging segmented objects that appear two or more times on adjacent tiles, thereby generating a working list of segmented objects stored in the data storage unit; as well as D. Based on the adaptive sampling representation of tiles, tile positions and working list of segmented objects stored in the data storage unit, a single loading of tiles and thereby forming a full-resolution complete image of part or all of the biological mass, the step of loading tiles in a single manner alleviates the need for additional computer memory and / or processor resources for any of the following items: storage, retrieval, transmission, distribution, processing, analysis, display and combinations thereof of the complete image.

37. The method of claim 36, wherein: For a subject patch, steps A and B are performed during source image acquisition, and step C is performed after image acquisition of an image corresponding to the subject patch.

38. A computer-based biomedical image manipulation system, the biomedical image manipulation system comprising: A. An interface for receiving or accessing image data of a subject of a biological mass, the interface obtaining a sequence of two-dimensional image units of increasing depth along an axis in a third spatial dimension, each two-dimensional image unit being an nxm pixel frame in a respective parallel plane orthogonal to the axis, the obtaining comprising obtaining different sequences of two-dimensional image units of increasing depth along respective orthogonal axes, each two-dimensional image unit in a sequence being an nxm pixel frame in a respective parallel plane orthogonal to the respective axis of the sequence, different sequences having different orthogonal axes at different tile locations across the respective planes, the obtaining producing a plurality of sequences, the plurality of sequences acting as tiles and representing image volumes capturing different portions of the biological mass in three spatial dimensions, different sequences in the plurality of sequences being different tiles representing different image volumes; B. A tile processing module, which is communicatively coupled to the interface, and which is automatically executed by a digital processor and processes each tile independently by: i) compressing image data of low information content tile regions and maintaining full resolution of high information content regions, thereby producing an adaptively sampled representation of the tile, ii) determine the relative tile position with respect to other tiles, iii) segmenting the adaptively sampled representation of the image block for an object of interest, the segmentation comprising extracting object features and corresponding object locations, and iv) storing the adaptively sampled representation of the tile, the determined tile positions, the segmented objects, the extracted object features and the object positions in a data store in a computer memory, the storing being performed in such a manner that the data store stores individual tile positions and a reduced amount of the subject image data rather than storing the source image of the biological quality; C. an aggregation module responsive to the tile processing module, the aggregation module aggregating the segmented objects stored in the data storage unit, wherein aggregating includes automatically merging segmented objects that appear two or more times on adjacent tiles, thereby generating a working list of segmented objects, object locations, and features stored in the data storage unit; as well as D. An output component having the adaptive sampling representation of tiles stored in the data storage unit, tile positions, and the contents of a work list of segmented objects, and the output component is configured so that a single loading of tiles thereby forms a complete image of part or all of the biological mass at full resolution, the single loading of tiles alleviating the need for additional computer memory and / or processor resources for any of the following items: storage, retrieval, transmission, distribution, processing, analysis, display, and combinations thereof of the complete image.

39. The computer-based biomedical image manipulation system of claim 38, wherein: The subject image data includes a biological or medical image acquired and generated by a source machine; and the tile processing module is configured to process the acquisition of the biological or medical image in near real time.

Images