Image Compression and Analysis

Abstract

Provided is the processing of a large-scale colored image dataset by efficiently performing image analysis, processing, and manipulation based on adaptive particle representation (APR). 【Solution means】The analysis of large-scale biological objects and tissue samples is performed with a calculation speed 100 times or more faster and an excellent memory storage compression ratio. The embodiment is extremely suitable for large-scale imaging projects of large three-dimensional transparent tissue samples such as whole-brain mapping initiatives and human nerve tissue pathology. The data store holds an adaptive sampling representation for tiles, individual tile positions, and other corresponding image data for the biological object of interest. By reading out a tile from the data store only once, a complete image of the maximum resolution of the biological object is output as an image.

Description

Related Applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 366,818, filed on June 22, 2022, and U.S. Provisional Patent Application No. 63 / 366,825, filed on June 22, 2022. The entire disclosure content of these applications is hereby incorporated herein by reference. BACKGROUND OF THE INVENTION

[0002] Biological imaging and medical imaging are a type of digital imaging, and refer to the technical field of generating and depicting images of the growth of internal parts, tissues, organs, skeletal structures, tendons, ligaments, and some blood vessels in the body. To give some examples, there are various techniques and devices for generating medical images, such as confocal light sheet microscopy, X-ray systems, CT (computed tomography) scanners, MRI (magnetic resonance imaging) devices, ultrasonic systems, PET (positron emission tomography) scanners, and the like. The images depicted are digital visualizations, by which medical staff can perform patient diagnosis and treatment, or more specifically, detect diseases, abnormalities, and injuries. For example, a CT scan provides a series of two-dimensional slices of the body, that is, cross-sectional images, by which medical staff can digitally view a tumor, its shape, size and location, as well as the blood vessels nourishing the tumor, on a computer display screen.

[0003] Researchers, attending physicians, etc. rely on biological images and medical images. Typical modalities of these imaging methods (more generally, digital imaging systems) include imaging, retrieving, processing (analyzing), communicating / distributing, and displaying raw digital images (such as by any of the above-mentioned devices and techniques). SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0004] In some biological and medical research, the volume of image data and its corresponding digital files can be relatively large, which may cause problems in the speed of storing (archiving) in computer memory, retrieving, processing (analyzing), and displaying by a computer. For example, tissue samples of the whole mouse brain, color images of wide cross-sections of the human brain, and other medical images with annotations are relatively large digital files. Although there are various image compression algorithms, the main purpose is to store the files. This is because the content of the files has to be decompressed to be read and understood, eliminating the advantage of compression.

Means for Solving the Problem

[0005] Embodiments of the present invention solve the disadvantages in the technical field regarding the operation (processing for analysis, communication / delivery, display, etc.) and management (storage in computer memory, archiving, retrieval, communication, etc.) of specific medical image data (specifically, large-volume multi-dimensional image data). Advantageously, the embodiments bring about an increase in processing speed and an improvement in the storage capacity of image files in computer memory without scaling up the resources and memory of the processor.

[0006] Embodiments provide a computer-based system and a method implemented on a computer for better operation (e.g., processing and analysis, distribution, display, etc.) and management (e.g., storage, retrieval, communication, etc.) of multi-dimensional images / medical images / biomedical images. Specifically described, embodiments achieve storage (archiving), retrieval, analysis, processing, and display of relatively large-scale image data at an acceptable speed without adding resources such as memory and processors (even without such resources). Examples of relatively large-scale image data include confocal / light-sheet microscopes (and other optical microscopes such as wide-field microscopes, super-resolution microscopes, electron microscopes, etc., but not limited thereto), X-rays, computed tomography (CT), positron emission tomography (PET), optical coherence tomography (OCT), MRI, ultrasound, and other biomedical images and medical images generated by any of other digital imaging techniques, but not limited thereto.

[0007] The image data can be multi-dimensional and, in addition to the digital image of the object, can include color, annotations (e.g., text and / or notes and / or marks of researchers or medical staff), graphics, a temporal dimension or time dimension, etc.

[0008] Examples of the objects of the relatively large-scale image data include tissue samples, whole mouse brains, cross-sections of human brains, other mammalian brains or parts, and other biological objects or substances of the object, but not limited thereto.

[0009] In one embodiment, in an image processing method implemented on a computer, for given image data of a living body, a series of tomographic images (broadly, two-dimensional image units) is acquired or accessed along the axis of the third spatial dimension in the depth direction. Each two-dimensional image unit (i.e., tomogram) is a frame of n×m pixels in each parallel plane orthogonal to the axis. In the same method, each series of two-dimensional image units along each orthogonal axis is acquired in the depth direction. Each two-dimensional unit of a series is a frame of n×m pixels in each parallel plane orthogonal to the corresponding axis of the series. The orthogonal axes of different series intersect the respective planes at different frame positions. As a result, in the same method, a plurality of series (so-called tiles) representing an image volume obtained by imaging various parts of the living body in three spatial dimensions are obtained. Each of the plurality of obtained series (i.e., tiles) represents a separate image volume obtained from the given image data.

[0010] That is, the "frame" as used in this specification refers to a 2D (two-dimensional) n×m pixel array of data. A plurality of such two-dimensional arrays (frames) are continuous along the same orthogonal axis, thereby constituting a three-dimensional data array called a "tile" in this specification. The orthogonal axes of separate tiles (continuations of frames) are separate orthogonal axes along which the respective frames are continuous. In other words, one tile is a continuous one of corresponding frames along one orthogonal axis. The second tile is a continuous one of corresponding frames along another orthogonal axis, the third tile is a continuous one of frames along the axis of the same tile, and so on.

[0011] Next, in the same method, at the time of image acquisition, i) Compress the image data in the tile region with less information amount and maintain the maximum resolution or high resolution in the region with large information amount, thereby obtaining an adaptive sampling representation for the tile, ii) Automatically obtain the conversion parameters of the tile, iii) automatically compare the compressed data and the raw data to verify that the data after conversion captures the necessary information, iv) calculate the maximum intensity projection images in a plurality of spatial directions of the tile for each edge where adjacent tiles exist, v) perform registration of the tile with adjacent tiles other than the tile itself by calculating the registration for each pair of three spatial dimensions between adjacent tiles using the calculated maximum intensity projection images (in an embodiment, after image acquisition, the tile positions are globally optimized with respect to a confidence measure. The relative tile positions thus obtained enable stitching to be performed with only one readout of each tile, and the image volume can be reconstructed by this stitching), vi) store in a data store in the computer memory the adaptive sampling representation for the tile, the conversion parameters obtained for the tile, and an index for the spatial position of the tile (this storage is performed such that the raw data of the source image of the biological object is not stored in the same data store, but rather the given image data is stored with reduced amounts together with the individual tile positions), automatically process each tile (series) individually and digitally.

[0012] In an embodiment, high-efficiency image processing (using compressed-form image data for analysis) and scalability (by individual analysis of tiles), which have not been achieved by the prior art so far, are provided as follows. In the method, at the time of image capture, for a given tile, (i) a classifier is applied to extract object feature amounts and object positions from the given tile to perform segmentation of the object, and (ii) the segmented objects are aggregated and stored in the data store. Objects that appear two or more times in adjacent tile regions are automatically merged by a matching algorithm. Examples of the matching algorithm include, but are not limited to, (a) finding the nearest neighbor in a feature space considering each object position, size, and shape in the final specimen space after registration, and (b) using Lowe's condition (the condition that the second nearest neighbor must be separated by a predetermined margin) to confirm that it is a reliable match. Other matching algorithms are also suitable. Then, in the embodiment, after that (after imaging the source image), each tile represented in the data store, or each tile from the corresponding data in the data store (that is, the stored compressed image data, the stored relative tile position, the compression parameter, the stored segmented object, and the stored object position) is read only once, so that the target image can be joined and reconstructed. In this way, in the embodiment, the problem that may occur in the prior art of applying analysis such as object segmentation to the entire image file, where the image file may be too large to process and store, is prevented.

[0013] Finally, in the method, a complete image of the maximum resolution of the living object is output as an image by reading tiles only once from the data store (i.e., compressed image data and corresponding data), rather than reading them multiple times or at separate times. By using the stored compressed image data, stored relative tile positions, compression parameters, stored segmented objects, and stored object positions from the data store, the need for additional computer memory and / or processor resources is alleviated with respect to storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof of the complete image.

[0014] In an embodiment, a tile region with low information amount is compressed using Adaptive Particle Representation technology.

[0015] An image processing method implemented on a computer is (A) For given image data of a living object, along the axis of a third spatial dimension, obtaining, in the depth direction, a series of two-dimensional image units that are frames of n×m pixels of each parallel plane orthogonal to the axis, so that for each orthogonal axis at each tile position on each plane, each series of two-dimensional image units along each orthogonal axis is obtained in the depth direction, and each two-dimensional image unit of the series is a frame of n×m pixels of each parallel plane orthogonal to the corresponding axis of the same series. By the same acquisition, a plurality of series that function as tiles are obtained, representing an image volume of various parts of the living object imaged in three spatial dimensions, and each of the plurality of series is a separate tile representing a separate image volume.

[0016] In some embodiments, the method comprises, at the time of imaging the source image (or at another time point in other embodiments), (B)(i) compressing the image data of tile regions with low information content and maintaining the high (maximum) resolution of regions with high information content, thereby obtaining an adaptive sampling representation for the tiles, (ii) determining the relative tile positions for other tiles, (iii) performing segmentation of the adaptive sampling representation for the tiles for the target object by extracting object feature amounts and corresponding object positions, and (iv) automatically performing per-tile processing by storing, in a data store in a computer memory, the adaptive sampling representation for the tiles, an index of the determined tile positions, the object after segmentation, the extracted corresponding object feature amounts, and the object positions, i.e., automatically processing each tile individually. The storage is performed such that the given image data is stored with its amount reduced, together with the individual tile positions, rather than storing the source image of the living body object in the data store.

[0017] As a next step, the method includes: (C) aggregating the segmented objects stored in the data store, where the aggregation automatically combines objects that appear two or more times between adjacent tiles among the segmented objects, to obtain a work list for the segmented objects stored in the data store; and (D) based on the adaptive sampling representation for the tiles, the tile positions, and the work list for the segmented objects stored in the data store, reading the tiles only once to form a complete maximum-resolution image of part or all of the living thing. In other words, in the method, there is no need to read the tiles multiple times or at different times to facilitate storage or perform image analysis (such as object segmentation). As configured, reading the tiles only once alleviates the need for additional computer memory and / or processor resources with respect to storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof of the complete image.

[0018] The work list may include indicators for the segmented objects, object positions, and respective feature amounts.

[0019] In an embodiment, the individual processing of each tile further includes: (i) automatically determining compression parameters used to compress the image data of the tile; (ii) storing the determined compression parameters in the data store; and (iii) automatically comparing the obtained adaptive sampling representation and the source image data to verify that the adaptive sampling representation captures the necessary information of the given image data.

[0020] In an embodiment, obtaining the relative tile position of a tile includes (i) calculating, for each edge of the tile where an adjacent tile exists among the plurality of spatial direction maximum projection images of the tile, (ii) calculating the registration parameter for each tile pair with the already acquired adjacent tile, and (iii) storing the calculated registration parameter for each tile pair in the data store. The registration parameter for each tile pair is calculated by correlating the maximum projection image calculated for the same edge of the tile with the maximum projection image calculated for the corresponding edge of the already acquired corresponding adjacent tile for each edge of the tile where the already acquired corresponding adjacent tile exists. As a result, in the method, the accuracy of the relative tile position of the tile with respect to the adjacent tile is improved.

[0021] In an embodiment, the registration further includes global optimization. Specifically, after image acquisition, the method globally optimizes all the registration results for each pair stored with respect to the confidence measure so that all tile positions are accurately corrected. By improving the accuracy and correcting the relative tile position, (a) stitching together the image data corresponding to the tiles, (b) appropriately reconstructing the entire biological object, and (c) reliably extracting higher-order information such as (but not limited to) the volume of the biological object and the number of segmented objects inside the imaged biological object are also possible. In an embodiment, process C (aggregation of segmented objects) is performed after global optimization.

[0022] In an embodiment, the given image data includes a biological image or a medical image captured / generated by any one of optical microscopy or an optical microscope (such as a wide-field microscope, a confocal microscope, a super-resolution microscope, an electron microscope, a light-sheet microscope, etc., but not limited thereto), X-ray, computed tomography, positron emission tomography, optical coherence tomography, magnetic resonance imaging, ultrasonic waves, and other digital imaging technologies. For a target tile, processes A and B are performed during the imaging of the biological image or the medical image, and process C is performed after the imaging of the image corresponding to the target tile.

[0023] In an embodiment, the biological object is any one of a tissue sample, a whole mouse brain, a cross-section of a human brain, other parts of a mammalian brain, and other biological substances of a target.

[0024] In an embodiment, one or more digital processors automatically execute the processes of the methods described herein. Another embodiment provides a computer-based system, a computer program product, and software as a service (SaaS). A computer-based system embodying the present invention may be composed of an interface, a data store in a computer memory, and a work module operably connected between these interface and data store and executable by a processor. The interface receives or accesses image data of a target for a target living body. The work module is configured (programmed) to automatically execute, in response to the interface, image compression on the image data by an adaptive sampling representation (e.g., APR, etc.), tile registration, and segmentation and aggregation of objects. The data store holds, for each tile, the adaptive sampling representation, the obtained relative tile position, the segmented object, the corresponding object position, and the extracted feature amount. The system generates a complete image of the whole or a part of the living body at the maximum resolution by reading out the tiles only once from the contents of the data store. By reading out the tiles only once, the need for additional computer memory and / or processor resources is alleviated with respect to any of storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof in the operation and management of the complete image.

[0025] In an embodiment, for each tile, the adaptive sampling representation to be stored is subjected to merging and cropping with overlapping regions such that a physical position of the living body is uniquely described by a particle of the adaptive sampling representation stored in the data store for the tile. In a given overlapping region where two or more tiles overlap, the adaptive sampling representation of the overlapping region merges with one of the same tiles and is cropped from the rest of these two or more tiles. The adaptive sampling representation of the merging and cropping version for the tile is stored in the data store. In this way, the adaptive sampling representation (subjected to merging and cropping with overlapping regions) for the tile brings about an efficiency that has not been achieved so far for storing and manipulating image data.

[0026] The foregoing will become apparent from the following detailed description of the exemplary embodiments shown in the accompanying drawings. Throughout the different figures, the same reference numerals refer to the same components. The drawings are not necessarily to scale, rather, emphasis is placed on illustrating the embodiments.

Brief Description of the Drawings

[0027]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6A

Figure 6B

Embodiments for Carrying Out the Invention

[0028] Hereinafter, exemplary embodiments will be described.

[0029] Adaptive Particle Representation (APR) is a content-adaptive and highly efficient image representation technology, and data compression is also performed. APR adaptively represents the content of an image while maintaining the image quality. In APR, the image storage cost is reduced by compressing the data. However, if it can be directly used for image processing tasks without re-converting it back to pixel data, advantageously, the memory and processing bottlenecks are solved. See B. L. Cheeseman, et al, "Adaptive particle representation of fluorescence microscopy images," Nature Communications 9, no. 5160, 4 December 2018.

[0030] In APR, pixels or voxels for a target image or image volume are replaced with particles (points in space that convey intensity) placed according to the content of the image. The particles can be placed anywhere required by the content of the image. If the parts of the image are different, the sizes of the particles can also be different. With the change in their sizes, each resolution for locally representing the image is determined. The required resolution is indicated by an Implied Resolution Function that assigns a high resolution to an image region where the intensity in space changes rapidly (e.g., edges, etc.) and a low resolution to an image region where the intensity change is small (e.g., background, uniform foreground, etc.). The Implied Resolution Function determines the radius of the neighborhood around each pixel. For any pixel position, the intensity value of the image can be reconstructed within the error threshold range set by the user by calculating the non - negatively weighted average of the intensities of the particles included in its neighborhood. Since the luminance conditions, i.e., the intensity range, can vary greatly between different regions within the sample, the error in question is normalized based on the local intensity scale.

[0031] In other words, APR is to resample an image relying on local information content while controlling local gain by spatially adaptive sampling. When APR is applied to an image, an image representation composed of a group of particles with intensity values is obtained as the output.

[0032] The principle of the present invention is to efficiently perform image analysis and processing of a relatively large - scale image dataset using the technology of adaptive particle representation. In an embodiment, the analysis of a large biological object or tissue sample is performed at a calculation speed 100 times or more faster and with an excellent memory storage compression ratio. The embodiment is extremely suitable for large - scale medical or biomedical imaging projects such as (but not limited to) whole - brain mapping initiatives and human nerve tissue pathology. The embodiment is also extremely suitable for continuous analysis of the same sample type (e.g., mouse brain, etc.) when experiments must be repeated for statistical purposes.

[0033] Figures 3A and 3B illustrate the flow of data and control in a method and system 1000 embodying the principles of the present invention. Although illustrated and described in an end-to-end pipeline from a source imaging device to a final useful image for distribution, storage, and communication, embodiments are not limited thereto. Other embodiments or portions thereof may function as, for example, image compression tools or devices, image storage / retrieval methods or systems, image analysis / post-processing methods or systems, and the like. Although merely non-limiting examples, Figures 3A and 3B will be described next in such a context, followed by a detailed description of implementation details.

[0034] In the supplementary explanation at the bottom of Figure 3A, hardware is shown in blue (darkest gray or common diagonal pattern), components acting on individual tiles are shown in orange (medium gray or common skew-parallel pattern), higher-order components (acting on tiles and other data) are shown in green (medium-dark gray or dotted pattern), and external libraries acting on voxel data are shown in light gray (or white-filled background). According to this capture explanation, the hardware components of the embodiment would include the imaging device 101 and the data store 125. Components acting on individual tiles would include the APR compression unit 121 (having automatic conversion parameters), the segmentation unit 123, and the preliminary stitching unit 122 (having a projection unit and pair-wise registration). Higher-order components would include the visualization unit 305, the stitching unit 310 (having a global optimization unit), the matching unit 130 (smart merger of segmented objects), and the reconstruction unit 320. Examples of external libraries include the atlasization unit 350 and other algorithms 355 acting on voxel data.

[0035] In the system 1000 of one embodiment, a source imaging device 101 such as a microscope captures and generates a biological image or a medical image 103 of a target biological object such as an organ, a part of the body, a tissue, or a target biological substance. Those skilled in the art will understand that various source imaging devices (confocal / light-sheet microscopes, OCT scanners, CAT scanners, PET scanners, ultrasonic systems, MRI systems, X-ray systems, etc.) 101 may be used. It will also be understood that other optical microscopy techniques and electron microscopy techniques (but not limited to these), such as wide-field microscopes, super-resolution microscopes, electron microscopes, etc., may function as the source imaging device 101 (an optical microscope). Other digital imaging techniques besides these are also suitable. The raw biological image or medical image 103 (FIG. 3B) can be large-scale (or relatively large volume compared to a general digital image). In some embodiments, the medical image or biological medical image 103 is, for example, a wide cross-section of a human brain, a whole mouse brain, a part or the whole of other parts of a mammalian brain, etc., but is not limited thereto.

[0036] Source 101 captures raw biomedical image 103 and transfers it to 105 such as local memory, computer storage, shareable memory, archive, etc. (collectively referred to as working memory; Fig. 3B). In working memory 105, users such as researchers and medical staff can add annotations (text notes, marks, etc.) and / or graphics and / or colors to the raw biological image or medical image 103. That is, the resulting image data 110 is composed of multiple information dimensions, namely, not only the raw biomedical image 103 but also annotations, graphics, and colors. Each pixel of the image data 110 is in the form of an N-tuple representation. The N-tuple data representation is realized by a data structure such as a linked list or an array that stores multiple information dimensions for each pixel. Each information dimension (i.e., tuple) of each pixel may include the raw source image (digital image pixel value), annotations, graphics (overlay value), colors (red, green, and blue component values), and in addition, a second or more additional text layer or graphic layer, fluorescence channel marks, time dimension, indicators of blood pressure and blood flow, other physical states, etc. (but not limited to these), information layers or information types. In one embodiment, only one channel or tuple is considered, that is, there is only one numerical value for each pixel instead of multiple numerical values.

[0037] The system and method 1000 of the present invention acquires, accesses, or receives image data 110 (N-tuple data for each pixel) of a target living object from the working memory 105 in near real-time during or after imaging by the source 101. Specifically, the interface 120 (FIG. 3B) receives the image data 110 and accesses a series of depth-direction tomographic images (broadly, two-dimensional image units) along the axis of the third spatial dimension or obtains such a series from the image data 110. Each two-dimensional image unit is a frame of n×m pixels of each parallel plane orthogonal to the axis of the third dimension. A series of frames along the target axis is referred to as a tile. The interface 120 obtains various tiles, that is, separate series of tomographic images (two-dimensional image units or frames) along each axis. Each two-dimensional image unit of a series (tile) is a frame of n×m pixels of each parallel plane orthogonal to the corresponding axis of the same series. Different series (i.e., tiles) are located at different tile positions on different planes.

[0038] As a result, the interface 120 obtains a plurality of such series (i.e., tiles). Each of the plurality of series / titles represents an image volume obtained by imaging a corresponding part of the target living object in the image data 110 in three spatial dimensions (three-dimensional space). Separate series / titles represent separate image volumes, and the separate image volumes are those obtained by imaging separate parts of the target living object.

[0039] The tile processing modules 121, 122, 123, 124 are connected to communicate with the interface 120. At the time of imaging the source image, for each series / tile acquired by the interface 120, the tile processing modules 121, 122, 123, 124 process each tile as follows. First, in step 121, the tile processing module compresses the tile area with less information amount and maintains the maximum resolution of the area with a large information amount. Thereby, an adaptive sampling representation for the tile is obtained. In an embodiment, such image compression and adaptive sampling for the tile are performed using APR. Other adaptive sampling techniques are also suitable.

[0040] Next, the tile processing module 121 automatically obtains the conversion parameters used to compress the image data for the tile. In an embodiment, the conversion parameters (i.e., the compression parameters in this specification) are automatically obtained by algorithms and techniques described in detail later. In some embodiments, the tile processing module 121 may apply APR individually to each channel or each dimension (tuple) of the image data 110 for the tile. As a non-limiting example, the tile processing module 121 may apply APR individually to each fluorescence channel for the same tile and automatically obtain the corresponding compression parameters. In steps 121 and 124 of the module, the compressed tile (i.e., the adaptive sampling representation for the tile) and the obtained corresponding compression parameters are stored in the data store 125.

[0041] The module 121 further verifies the compressed representation for the tile by automatically comparing the obtained adaptive sampling representation with the source image data. In this comparison, it is determined whether the adaptive sampling representation captures the necessary information of the source image data. In this verification comparison in step 121 of the module, known or common techniques such as the peak signal-to-noise ratio and the structural similarity index are used.

[0042] In an embodiment, improvement of the compressed image data representation for the overlapping regions between adjacent tiles stored in the data store 125 can be performed. In the embodiment, the module 121 stores each tile individually in the data store 125. However, in the original compressed data (adaptive sampling representation) stored in the data store 125, there are overlapping regions between each pair of adjacent tiles. In such a state, a physical position of a biological object (a physical sample to be imaged) is not always uniquely determined by one particle (in the adaptive sampling representation) in the data store 125, and may be determined by two particles (adjacent regions between adjacent ones) or four particles (corner overlapping regions). FIG. 6A shows, as the same phenomenon, a state where four different tiles (represented in different colors) overlap each other.

[0043] The schematic diagram of FIG. 6A represents the number of compressed image particles (adaptive sampling representation particles) that define the spatial position in the biological object by the numbers 1, 2, and 4. Specifically, a block (i.e., quadrant) 600 of four tiles is illustrated. In reality, there are more tiles than this, but the principle remains the same. The upper left tile 610 overlaps the adjacent tile 620 on the right, i.e., the upper right tile in the block 600, along the length of its right edge. The upper left tile 610 overlaps the adjacent tile 630 below, i.e., the lower left tile in the block 600, along its lower edge. Similarly, the upper right tile 620 overlaps the upper left tile 610 along its left edge. The upper right tile 620 overlaps the adjacent tile 640 below, i.e., the lower right tile in the block 600, along its lower edge. The lower left tile 630 in the quadrant (block 600) overlaps the upper left tile 610 along its upper edge. The lower left tile 630 overlaps the lower right tile 640 along its right edge. The lower right tile 640 overlaps the lower left tile 630 along its left edge. The lower right tile 640 overlaps the upper right tile 620 along its upper edge.

[0044] The non-overlapping tile regions are each marked with the number 1 to indicate that the spatial position to be represented in the target living organism (physical sample to be imaged) is uniquely defined by a single particle in the data store 125 (adaptive sampling representation of each tile). The region where two tiles overlap is marked with the number 2 for each to indicate that the corresponding spatial position in the target living organism is defined by two particles (one from each of the compressed images / adaptive sampling representations for adjacent neighboring tiles) in the data store 125. As a non-limiting example, in the overlapping region 615 along the right edge of the upper left tile 610 and the left edge of the upper right tile 620, the part excluding or removing the overlapping part of the middle corner closer to the center of the block 600 is marked with the number 2 to indicate that the corresponding position or spatial position in the target living organism is defined by two particles (one from each of the corresponding two sets of particle groups of the adaptive sampling representations for tiles 610, 620) in the data store 125. Similarly, in the overlapping region 625 (between the upper right tile 620 and the lower right tile 640), the overlapping region 635 (between the upper left tile 610 and the lower left tile 630), and the overlapping region 645 (between the lower left tile 630 and the lower right tile 640), the parts excluding the corresponding middle corner parts are marked with the number 2 to indicate that the corresponding position (spatial position) in the living organism is defined by two particles (one from each of the particle groups of the respective adaptive sampling representations for two adjacent neighboring tiles) in the data store 125.

[0045] In the region where all four tiles 610, 620, 630, and 640 overlap (their inner corners, i.e., the lower right corner of the upper left tile 610, the lower left corner of the upper right tile 620, the upper right corner of the lower left tile 630, and the upper left corner of the lower right tile 640), the number 4 is marked to indicate that the corresponding spatial arrangement or position in the biological object is defined by four particles, one from each of the respective particle groups corresponding to the adaptive sampling representation in the data store 125 for the four adjacent neighboring tiles 610, 620, 630, 640.

[0046] In a different embodiment, interesting improvements are as follows: a) Combine the particle information of the overlapping tile regions and store the combined result in the data store 125. b) Cut out the tiles (tile representations) such that a spatial position or a point position in the target sample (biological object) is uniquely determined by a single particle within a stored tile, i.e., uniquely determined by a single particle in the data store 125. Continuing with the example of FIG. 6A, in the improved version of the embodiment shown in FIG. 6B: (i) Maintain the smallest particle (representing the highest resolution) among the union of the particle groups corresponding to the overlapping tile regions of interest, and combine the particle information (of the adaptive sampling or compressed image data representation) of the overlapping tile regions (690). Next, in the combining method 690: (ii) For each particle group involved in the overlap, average the intensity values of the particles within the group (average for each tile region). (iii) Select the maximum value of the calculated average intensity values among the overlapping tile regions of interest. (iv) Assign the result of (iii) as the intensity value to the smallest particle maintained in step (i). Finally, in method 690, cut out the target tiles (tile representations) such that all positions in the target biological object are uniquely determined by a single particle contained in the tile representation in the data store 125.

[0047] In the example of FIG. 6B, method 690 maintains or combines the overlapping regions 615, 635 as those of the upper left tile 610, and removes the corresponding overlapping regions from the upper right adjacent tile 620 and the lower left adjacent tile 630. That is, in method 690, the above steps (i) to (iv) are applied to the overlapping regions 615, 635 and the inner corners of tile 610, and particles within the lightly shaded portions drawn along the right edge and the lower edge of tile 610 are determined. In method 690, the corresponding overlapping region is cut out along the left edge of the upper right tile 620, and the corresponding overlapping region is cut out along the upper edge of the lower left tile 630. The compressed image data / adaptive sampling representation for tile 610 obtained by module 121 implementing method 690 is stored in data store 125. Similarly, in method 690, steps (i) to (iv) are also applied to overlapping region 625 to determine particles within the lightly shaded portion drawn along the lower edge of the upper right tile 620. In method 690, the particles and the lightly shaded portion thus determined are combined with the upper right tile 620, and the corresponding overlapping region is cut out from the upper edge of the lower right tile 640. In method 690 (module 121), the resulting compressed image data / adaptive sampling representation for the upper right tile 620 is stored in data store 125, and the post-cut adaptive sampling representation for the lower right tile 640 is stored in data store 125. In method 690, steps (i) to (iv) are applied to overlapping region 645 to determine particles within the lightly shaded portion drawn along the right edge of the lower right tile 630. In method 690, the particles and the lightly shaded portion thus determined are combined with the lower left tile 630, and the corresponding overlapping region is cut out from the left edge of the lower right tile 640. In method 690 (module 121), the resulting compressed image data / adaptive sampling representation for the lower left tile 630 and the compressed image data / adaptive sampling representation for the lower right tile 640 are stored in data store 125.

[0048] In the above-described improved embodiment, as a result of the module 121 and method 690, all tiles (adaptive sampling representations of tiles) stored in the data store 125 are the same size, except for the tiles along the rightmost edge of the acquisition and the tiles along the lower edge of the acquisition. The lightly shaded portions in FIG. 6B indicate the locations where merging and cropping were performed (by application of the method 690 by the module 121). Specifically, the upper left tile 610 (including the shaded portions 615, 635 merged with the tile) has not changed in size. The upper right tile 620 (where the shaded portion 625 merged) has been cropped and is smaller in the width direction. The lower left tile 630 (where the shaded portion 645 merged) has had its length cropped. The lower right tile 640 has had both its width and length cropped.

[0049] Returning to FIG. 3B, in step 122, the tile processing module determines the relative tile position with respect to other tiles. In step 122, to do this, for a given tile, the maximum projection images in a plurality of spatial directions of the tile are calculated for each edge where adjacent tiles exist. Next, in step 122 of the tile processing module, the registration parameters for each tile pair with the already acquired adjacent tiles are calculated. Specifically, in step 122, for each edge of the given tile where the corresponding acquired adjacent tile exists, the maximum projection image calculated for the same edge of the tile is correlated with the maximum projection image calculated for the corresponding edge of the acquired corresponding adjacent tile. In an embodiment, by correlating the corresponding maximum projection images calculated for the related edges between a given tile and its adjacent tile, the accuracy of the relative tile position of the tile with respect to the adjacent tile is improved. In steps 122 and 124 of the module, the calculated registration parameters for each pair are stored in the data store 125. In an embodiment, the tile processing module 122 performs the registration between a tile and other tiles by: (a) at the time of imaging the source image, acquiring the registration for each pair of adjacent tiles using the calculated maximum projection image; (b) after imaging the source image, performing global optimization with respect to the confidence measure; By executing in this way, even if each tile is read only once, the position of each tile is optimally calculated. The stitching of the images will be described in more detail later.

[0050] In an embodiment, following the per-tile processing during imaging of the source image, at step 123 of the tile processing module, segmentation of the target object from the adaptive sampling representation (output from step 121) for the tile is performed. In module 123, a classifier is applied so as to perform segmentation of the object by extracting object feature amounts and object positions from a given tile. Examples of object feature amounts may include, but are not limited to, the diameter, shape, volume, etc. of the object to be detected. In some embodiments, module 123 further calculates the position of the segmented object using the tile position output from step 122 and stored in data store 125. In an embodiment, the classifier may be formed by image filtering processing and subsequent threshold processing. In another embodiment, after compression of the tile region, some tiles are manually annotated and learned by a machine learning classifier, and the classifier is formed by directly executing segmentation for a given tile at step 123. As will be described in detail later regarding segmentation of the object, other classifiers are also suitable.

[0051] In a preferred embodiment, at step 123: (i) the output of the classifier, which is a binary mask calculated directly (very efficiently) by APR (including, comprehensively, segmentation of the object); and (ii) indicators regarding the physical aspects and characteristics, i.e., the position and feature amounts, of the segmented object calculated from the binary mask; are stored in data store 125. The configuration of storing the binary mask together with the compressed image data (adaptive sampling representation for the tile) in data store 125 is extremely space-efficient compared to the case where twice the memory space is required to store the segmented object (image) and the uncompressed target image.

[0052] In step 124 of the tile processing module, for each processed tile, (a) the adaptive sampling representation for the tile output from step 121, (b) the compression parameters obtained for the tile from step 121, (c) the position of the tile in the three-dimensional space, i.e., the index for the relative tile position in step 122, and (d) the segmented object output from step 123 are stored in the data store 125 in the computer memory. In one embodiment, the data store 125 holds a list for the segmented object, the extracted object feature quantities, and the object positions aggregated across tiles. Thus, the data store 125 does not store the raw data of the source image 103 of the target living thing, but rather the image data 110 is stored in a reduced amount together with the individual tile positions.

[0053] In some embodiments, as shown in FIG. 3A, the data store 125 is configured to function as part of the image compression apparatuses, tools or services 310, 130, 320 by the actions of the tile processing modules 121, 122, 123, 124. In the embodiment, further, a specific part of the image data 110 can be selectively read out and displayed (305) by the user in the compressed form stored in the data store 125. For the visualization unit 305, refer to further details described later. As will be made clear later, in the embodiment, the interface 120 and the tile processing modules 121, 122, 123, 124 are executed during image capture by the source device 101, and the other processes of the system / method 1000 are executed after the capture of the source image 103.

[0054] As a continuation of FIG. 3B, an aggregation module 130 is communicatively connected to the data store 125. The aggregation module 130 aggregates the segmented objects in the data store 125 to form a work list and replaces the initial aggregation list. The aggregation module 130 automatically combines objects with each other by a matching algorithm for objects that appear two or more times between adjacent tile regions. In an embodiment (aggregation module 130), for example, a nearest neighbor type matching algorithm is applied, but not limited thereto, within a feature space in which each object is embedded together with its position, size, and shape. The match can be reliably verified by using the ratio of the nearest neighbor candidate to the next nearest neighbor candidate. If this ratio exceeds a certain predetermined threshold (usually 0.7), it is regarded as a reliable match, and the matched objects are combined with each other.

[0055] As the matching algorithm or means for combining objects with each other in module or step 130, matching algorithms and means other than the above are also suitable.

[0056] Returning to FIG. 3B, in system / method 1000, a combining unit 140 (also referred to as a reconstruction unit 320; FIG. 3A) responsive to the aggregation module 130 is used. The combining unit (module or step) 140, 320 reads each tile (i.e., a tile represented by the data in data store 125) only once, combines the tiles in an adaptive sampling representation, and from there forms an output assembly (file, stream, etc.) of a complete image at the maximum resolution of the target biological object or (collectively) the output image 150. In the combination or reconstruction 320 by the combining unit 140, the tiles are joined together. The combination is performed to support the re-conversion of the tiles to be combined into the voxel data of the image volume in the work memory 105. The approach of reading each tile only once to form the output image 150 alleviates the need for additional computer memory and / or processor resources with respect to the storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof of the complete image 150. Also, the combining unit 140 / reconstruction unit 320 may make the complete image 150 have a low resolution as required by the application. In another embodiment, instead, the adaptive sampling representations (i.e., APR files) are directly combined. For more details regarding the combination of images, refer to the further detailed description later.

[0057] Embodiments may involve interface connections (350, 355) with, for example, but not limited to, a registration pipeline, a working algorithm, other systems, etc. In the registration pipeline, alignment of images of the same category of biological objects or living bodies is performed. Through such registration, an output data file (complete image) 150 is aligned with a similar target image within a common coordinate framework, and the segmented objects (from the aggregation module 130) are mapped to corresponding objects within the registration pipeline. Advantageously, such alignment and mapping enable counting cells in a target body part, etc. Based on the present disclosure, other uses and advantages of the embodiments will also be within the scope of those skilled in the art.

[0058] Next, in addition to the modules, steps, and components of FIGS. 3A and 3B, or as modifications thereof, further implementation details are introduced as another embodiment of the present invention.

[0059] (Automatic Image Conversion)

[0060] The adaptive sampling of APR is calculated based on the gradient of the raw image signal with reference to a local intensity scale for error normalization. Both the gradient and the local intensity scale need to be numerically estimated from the captured image. This is extremely susceptible to noise, and high-frequency fluctuations will be added even in regions where the underlying (true) signal is in a perfect flat state. In this implementation, the influence of noise on the gradient and the local intensity scale is reduced by using a smoothed B-spline approximation of the signal.

[0061] The conversion of the voxel image to APR depends on multiple parameters. Most of these (e.g., voxel size, background intensity level, smoothing degree of B-spline operation, etc.) can surely be kept constant for a given optical system and experiment. However, smoothing is not sufficient to completely escape the influence of noise in the signal. Therefore, in order to avoid oversampling caused by normalizing even small gradients, the gradients and local intensity scales are thresholded from below. This adds two additional threshold parameters, one for each quantity. The optimal values of these thresholds, in contrast to the other parameters, are greatly influenced by the local signal amount. Therefore, in large-scale experiments where manual adjustment is not feasible, it is necessary to automatically estimate those parameters.

[0062] The optimal threshold is a value that best distinguishes the distribution of numerical values (gradients and local intensity scales) derived from the actual signal from the fluctuations caused by noise. If the threshold is low, it adapts to the noise and results in non-optimal compression. On the other hand, if the value is high, it leads to undersampling and loss of information. In principle, any automatic (local or global) thresholding algorithm may be used for estimating the optimal value. However, since the distribution greatly depends on the optical configuration and the local characteristics of the sample, an algorithm based on strict assumptions about the form of the distribution is not appropriate. In the embodiment, the minimum cross-entropy thresholding algorithm of Li et al. (Reference 10) is used because of its proven robustness, simplicity, and computational efficiency.

[0063] (Stitching of images (122, 310))

[0064] Scanning and imaging of the sample are performed by a pattern such as a raster scan pattern that can cover the entire desired region so that each tile is acquired in order. The applicant's pipeline of the present application supports sparse tiling where it is sufficient that the tiles are connected at least on one side, which is an application to APR by obtaining a hint from Reference Document 1. First, an adjacency map for each tile (i.e., the side of each tile) is calculated. Next, for each pair of adjacent tiles, the displacement amounts (x, y, z) for aligning these tiles are calculated using phase cross-correlation (Reference Document 2) of the maximum intensity projection images (I x , I y , I z ). Depending on the characteristics of the biological image and medical image, another alignment method may be used for performance improvement. Examples of such alignment methods include, but are not limited to, feature quantity (e.g., SIFT, etc.) matching (e.g., RANSAC, etc.) and machine learning approaches.

[0065] When artifacts (e.g., bubbles that have entered around the sample, etc.) exist in the volume, a masked version of phase cross-correlation is used (Reference Document 3).

[0066] The maximum intensity projection image is calculated in advance (only for the overlapping region expected to eliminate the noise of the phase cross-correlation) so that the connection can be made with only one readout of each tile. By doing this, for each pair of adjacent tiles, a total of six displacement amounts (two for x, two for y, and two for z) are calculated. The reliability is calculated as the least squares difference between the maximum intensity projection images after registration, and the displacement amount with the highest reliability is maintained. The reliability may be calculated by other metrics, for example, the ratio of the maximum value of the difference between the maximum intensity projection image after registration and the original maximum intensity projection image to twice the maximum value of the original image (the applicant of the present application has found that this is extremely effective for sparse samples), etc., but is not limited thereto.

[0067] These displacement amounts are stored in three graphs (one for each spatial dimension) where each vertex corresponds to a given tile and each edge corresponds to the displacement amount between two tiles. For each spatial dimension, a corresponding confidence graph is constructed. Finally, the joiner 310 globally optimizes each displacement graph using the maximum spanning tree of the confidence graph to satisfy the internal problem constraint that each loop in the graph must be zero.

[0068] For the joining, all steps except calculating the phase cross-correlation are performed entirely using APR. In this step, after the maximum value projection image is calculated by APR, a two-dimensional pixel image is reconstructed. Here, regenerating the pixel data is not a disadvantage because it is simply two-dimensional and has a small memory footprint.

[0069] Finally, as previously described in FIGS. 3A and 3B, the tile positions are stored in the data store 125 and can be used later for calculating and visualizing (305) the positions of the segmented objects.

[0070] (Segmentation of Objects)

[0071] In one embodiment, a particle classification strategy is used to segment objects such as labeled cells. A random forest with 100 estimators is trained with a small amount of data that has been manually sparsely annotated in advance (see reference 4).

[0072] The pipeline of this proposal directly performs segmentation while acquiring the next tile after completely acquiring a certain tile, so it does not combine tiles before performing segmentation. After the feature amounts and positions of objects are extracted, high-order information is combined without actually combining the APR data according to the combination strategy developed with reference to Reference 5. For each adjacent tile, two nearest neighbor candidates (NNs) for each object in the overlapping region of the first tile are calculated within the group of objects in the overlapping region of the second tile. If the ratio of the second NN to the first NN exceeds a threshold (typically 0.7) and the centers of the objects are closer than a distance threshold (typically 1 / 4 of the size of the object), the objects are regarded as the same and automatically combined. Objects touching the edge of the overlapping region are automatically ignored so as not to be counted twice. As shown in Figure 2, this strategy has been confirmed to be effective even in the synthetic dataset.

[0073] Figure 2 is a graph showing the combination of objects after segmentation in multi-tile data for a target medical image or biomedical image. A multi-tile synthetic dataset was randomly generated with various numbers of objects and random shifts. After segmenting each tile individually, the objects are combined in the overlapping region (after calculating the exact registration as described above). The number of detected objects is plotted against the actual number of objects. The graph in Figure 2 also depicts, for reference, the number of objects detected in the original data (before slicing and forming the multi-tile dataset).

[0074] (Combination of images)

[0075] For some applications such as registering the sample to Atlas 350 or other 355, a merged volume is required. Fortunately, these applications 350, 355 generally allow for a low resolution of the sample and do not require reconstructing the footprint of the original image data. By simply selecting an appropriate APR level, each tile is efficiently voxel-reconstructed at a low resolution and interpolation is performed later if a precise voxel size is required. Then, the tiles are merged by any merging strategy such as calculating the average or maximum value of the overlapping regions. Before transformation or merging, preprocessing (e.g., histogram equalization, etc.) can be performed to gain the advantage of improved APR calculation speed.

[0076] (Registration to Atlas (350))

[0077] From the Brainreg front-end (Reference 8), the merged mouse brain can be registered to the Allen brain mouse atlas (the mouse brain atlas of the Allen Institute) (Reference 6) by the AMAP pipeline (Reference 7).

[0078] (Visualization (305))

[0079] In one embodiment, Napari (Reference 9) is used as the front-end for visualization 305. Napari requires an object with slicing attributes to enable lazy loading, so there is no need to reconstruct the complete sample within the voxel space. So-called "lazy loading" means reading and displaying the compressed image data (output from Step 121 and stored in the data store 125).

[0080] (Linking of Benchmarks)

[0081] Steps 1-5 (equivalent to identifying stitching parameters by stitching without merging data) were used to compare the stitching speed with TeraStitcher (Reference 1). TeraStitcher can run multiple instances simultaneously with openmpi. Four cores were used to obtain the best performance. The applicant of the present application generated synthetic datasets consisting of 16 voxels (about 1 Gb per tile) of 4×4 tiles of 2048×512×512 units at various operation ratios (by varying the number of objects). The operation ratio is the ratio of the total number of voxels before compression to the total number of particles after compression. The conversion from voxel data to APR was performed by automatically determining the parameters as described above (see the section on automatic image conversion).

[0082] Referring to FIG. 1, the APR stitching time was examined at various operation ratios (CR) and compared with the stitching time of TeraStitcher. From the graph in FIG. 1, it can be seen that the embodiment is 20 to 1000 times faster in stitching time than TeraStitcher.

[0083] FIG. 6 of the priority application, which incorporates the entire content herein by reference, also describes the APR conversion, stitching, segmentation, and registration to atlas of one embodiment. The figure describes an end-to-end pipeline for analyzing large 3D transparent tissue samples of the entire mouse brain or wide cross-sections of the human brain. The systems and methods of the present disclosure achieve operations that are over 100 times faster.

[0084] In the stitching process or module in the above end-to-end pipeline, stitching is performed directly on the representation generated by APR for the target image. The stitching module is compatible with any tiling pattern. Advantageously, the stitching module reads each tile only once, making it extremely efficient.

[0085] The above-described end-to-end pipeline segmentation module acts directly on the representation generated by the APR for the target image and performs segmentation (extraction) of objects, object features, and object positions from the target image. Annotations can be efficiently and directly attached manually by the user to the representation generated by the APR.

[0086] Finally, the end-to-end pipeline of this example describes a registration 350 to an atlas. Low-resolution pixel data can be directly reconstructed and combined from the multi-tile APR representation of the target image. This enables the use of an atlas registration pipeline such as AMAP from Brainreg (as a front end) or remapping to the APR data.

[0087] As a result, the method and system embodying the present invention provide a highly efficient image analysis pipeline based on APR, reducing the storage (memory) requirements while processing large-scale 3D datasets at a speed 20 to 1000 times faster (improved processing speed) than existing solutions. Such embodiments are extremely suitable for large-scale imaging projects.

[0088] Further details of embodiments of the present invention are provided by the attached kind of the priority application (which is incorporated herein by reference in its entirety).

[0089] (Computer assistance)

[0090] Figure 4 shows a digital processing environment such as a computer network in which embodiments of the present invention can be realized.

[0091] One or more client computers / devices 50 and one or more server computers 60 serve as processing devices, storage devices, and input / output devices that execute application programs, core programs, and the like. One or more client computers / devices 50 can consist of imaging sources such as the aforementioned medical imaging device 101. Further, one or more client computers / devices 50 can be connected via a communication network 70 to additional client devices / processes 50 and other computing devices such as one or more server computers 60. The communication network 70 can be a remote access network, a global network (e.g., the Internet, etc.), a cloud computing server or service, a collection of computers around the world, a local area network, a wide area network, or a part of a gateway that currently communicate with each other using their respective protocols (TCP / IP, Bluetooth (registered trademark), etc.). Other electronic device / computer network architectures are also suitable.

[0092] FIG. 5 is a diagram of the internal structure of a computer (e.g., client processor / device 50, server computer 60, etc.) of the computer system of FIG. 4. Each computer 50, 60 includes a system bus 79 which is a series of hardware lines used to transfer data between components of the computer and processing systems. Substantially, the bus 79 is a shared conduit that connects each component of the computer system (e.g., processor, disk storage, memory, input / output ports, network ports, etc.) and enables the transfer of information between these components. An input / output device interface 82 for connecting various input / output devices (e.g., keyboard, mouse, display, printer, speaker, etc.) to the computers 50, 60 is attached to the system bus 79. The network interface 86 enables the computer to connect to various other devices attached to a network (e.g., network 70 of FIG. 4, etc.). The memory 90 serves as a volatile storage unit for computer software instructions 92 and data 94 used to implement an embodiment of the present invention (e.g., the image processing method, system, technology, data store / database, program code, etc. described in detail in FIGS. 3A, 3B, 6A, and 6B). The disk storage 95 serves as a non-volatile storage unit for computer software instructions 92 and data 94 used to implement an embodiment of the present invention. Further, a central processing unit 84 for executing computer instructions is attached to the system bus 79.

[0093] In one embodiment, the routine 92 and data 94 of the processor are computer program products (generally denoted as 92), such as computer-readable media (e.g., removable storage media such as at least one DVD-ROM, CD-ROM, diskette, tape, etc.) that provide at least a portion of the software instructions of the system of the present invention. The computer program product 92 may be installed by any suitable software installation procedure well known in the art. In other embodiments, at least a portion of the software instructions may be downloaded via wired communication and / or wireless connection. In other embodiments, the program of the present invention is a computer program propagation signal product 107 incorporated in a propagation signal in a propagation medium (e.g., an electromagnetic wave propagated by a global network such as radio waves, infrared waves, laser waves, sound waves, the Internet or other one or more networks). Such a carrier medium or signal becomes at least a portion of the software instructions for the routine / program 92 of the present invention.

[0094] In an alternative embodiment, the propagation signal is an analog carrier wave or a digital signal carried by a propagation medium. For example, the propagation signal may be a digitized signal propagated by a network such as a global network (e.g., the Internet, etc.), a telecommunication network. In one embodiment, the propagation signal is a signal transmitted by the propagation medium for a period of time, and is, for example, instructions for a software application transmitted in packets by a network for a period of several milliseconds, several seconds, several minutes or more. In other embodiments, the computer-readable medium of the computer program product 92 is a propagation medium that can be received and read by the computer system 50. For example, the computer system 50 identifies a propagation signal incorporated in the received propagation medium as in the case of the computer program propagation signal product described above.

[0095] Generally speaking, the term "carrier medium" or transient carrier includes the aforementioned transient signals, propagation signals, propagation media, storage media, and the like.

[0096] In other embodiments, the program product 92 may be implemented as so-called software as a service (SaaS), or other installations or communications that support end users.

[0097] Based on the foregoing description and details, the embodiments of the present invention bring many advantages that exceed the prior art methods and approaches. For example, in the embodiments, the object of interest is segmented from the image data at the time of imaging the source image, rather than after imaging. Such a novel approach enables scalability and makes it possible to generate a resultant (object of the image) that is available immediately after imaging the image.

[0098] Advantageously, in an embodiment, by reading each tile (i.e., the stored compressed image data, the stored relative tile position, the compression parameter, the stored segmented object, and the stored object position) only once, the target image is pieced together and reconstructed. In contrast, in the prior art method, the tile data is read twice, i.e., once for the evaluation of the registration parameter and then once at the end stage of combining all the tiles to generate an image file. In the prior art approach, analysis such as object segmentation can be applied to the image file. However, the approach of combining all the tiles may not necessarily be a scalable approach because the image file may be too large to process or store at some point. The applicant of the present application solves such drawbacks in the art and maintains individual tiles as separate entities. Rather, the applicant's approach of the present application combines the information of the segmented objects (e.g., the segmented objects of the objects that appear more than once in the overlapping or adjacent tile regions as described above in module 130 of FIG. 3A) as needed. The applicant's approach of the present application can scale well because the processing of each tile is performed individually (or together with its neighbor) as long as the processing of the tile keeps up with the data throughput of the source device 101 (moreover, this can be achieved by using an adaptive sampling representation of the tile (e.g., APR, etc.)).

[0099] All the teachings of all patents, patent application publications, and publications cited in this specification are incorporated herein by reference.

Prior Art Documents

Non-Patent Documents

[0100]

Non-Patent Document 1

[0101] Although the exemplary embodiments have been specifically illustrated and described, those skilled in the art will understand that various changes may be made to the form and details without departing from the scope of the embodiments encompassed by the appended patent claims.

Claims

1. An image processing method implemented on a computer, comprising: A. For given image data of a living object, along the axis of the third spatial dimension, a series of two-dimensional image units each being a frame of n×m pixels of each parallel plane orthogonal to the axis is acquired in the depth direction, and a series of each two-dimensional image unit along each orthogonal axis is acquired in the depth direction. Each two-dimensional image unit in a series is a frame of n×m pixels of each parallel plane orthogonal to the corresponding axis in the same series, and different series have different orthogonal axes intersecting each plane at different tile positions. Through the above acquisition, a plurality of series functioning as tiles are obtained, representing an image volume of various parts of the living object imaged in three spatial dimensions, and each of the plurality of series is a separate tile representing a separate image volume. The process of acquisition; B. At the time of imaging the source image, i) A sub-process of obtaining an adaptive sampling representation for the tile by compressing the image data in the tile region with less information amount and maintaining the high resolution in the region with large information amount; ii) A sub-process of obtaining the relative tile position with respect to other tiles; iii) A sub-process of segmenting the adaptive sampling representation for the tile with respect to the target object, the segmentation including extracting object feature amounts and corresponding object positions; and iv) Instead of storing the source image of the living object in a data store in the computer memory, the adaptive sampling representation for the tile, the obtained tile position, the segmented object, the extracted object feature amounts, and the object positions are stored in the data store such that the given image data is stored with its amount reduced, together with the individual tile positions, by a sub-process; The process of automatically processing each tile individually by a computer processor; C. A process of aggregating the segmented objects stored in the data store, the aggregation including obtaining a work list for the segmented objects stored in the data store by automatically combining the objects that appear two or more times among adjacent tiles among the segmented objects. The process; D. A process of reading out a tile only once based on the adaptive sampling representation for the tile, the tile position, and the work list for the segmented object stored in the data store, and forming a complete image of the whole or part of the biological object at the maximum resolution therefrom, wherein, by reading out the tile only once, the need for additional computer memory and / or processor resources is alleviated with respect to storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof of the complete image, A method comprising the above. **Claim 2** In the method according to claim 1, the process of processing each tile further comprises: A sub-process of automatically determining the compression parameters used to compress the image data of the tile, A sub-process of storing the determined compression parameters in the data store, and A sub-process of automatically comparing the obtained adaptive sampling representation and the source image data so as to verify that the adaptive sampling representation captures the necessary information of the given image data, A method comprising the above. **Claim 3** In the method according to claim 1, the sub-process of determining the relative tile position of the tile comprises: Calculating the maximum value projection image of the adaptive sampling representation of each edge of the tile in a plurality of spatial directions and the tile where adjacent tiles exist, For each edge of the tile where acquired corresponding adjacent tiles exist, correlating the calculated maximum value projection image of the edge of the tile with the calculated maximum value projection image of the corresponding edge of the acquired corresponding adjacent tile, thereby improving the accuracy of the relative tile with respect to the adjacent tiles of the tile and calculating the registration parameters for each tile pair with the already acquired adjacent tiles after adaptive sampling, and Storing the calculated registration parameters for each tile pair in the data store, A method comprising the above. **Claim 4** In the method according to claim 3, further, A process of globally optimizing all the results of the registration for each of the stored pairs with respect to a reliability measure after imaging of the image, wherein by the maximizing process, all tile positions are accurately corrected to (a) stitch together image data corresponding to the tiles, (b) appropriately reconstruct the entire living body, and (c) reliably extract high-order information such as the volume of the imaged living body, the number of objects after segmentation inside, etc. (but not limited to these), the optimizing process. A method comprising that process C is performed after global optimization.

5. The method according to claim 4, wherein the stitching is performed by reading each tile only once.

6. The method according to claim 1, wherein the given image data includes a biological image or a medical image captured / generated by any one of optical microscopy, electron microscopy, X-ray, computed tomography, positron emission tomography, optical coherence tomography, magnetic resonance imaging, ultrasonic waves, and other digital imaging techniques, and for the target tile, process A and process B are performed during imaging of the biological image or the medical image, and process C is performed after imaging of the image corresponding to the target tile.

7. The method according to claim 1, wherein the given image data further includes any one or a combination of color, annotation, text, graphic, temporal feature amount, etc.

8. The method according to claim 1, wherein the living body is any one of a tissue sample, a whole mouse brain, a cross-section of a human brain, other parts of a mammalian brain, and other biological substances of the target.

9. The method according to claim 1, further comprising a process of individually performing compression for different fluorescence channels of the given image data in addition to compression of the tile region. The method comprising this.

10. The method according to claim 1, further comprising a process of performing compression of the given image data at various periods in addition to compression of the tile region. The method comprising this.

11. The method according to claim 1, wherein a tile region with a small amount of information is compressed using an adaptive particle representation (APR) technique.

12. In the method according to claim 1, by means of the segmentation, a classifier is applied to the adaptive sampling representation for the tile, and the classifier is (i) an image filtering process followed by a thresholding process, and (ii) a machine learning algorithm such as, but not limited to, a random forest, a support vector machine, a neural network, etc. trained with similar data, formed by any of the above, method.

13. In the method according to claim 1, in the process of aggregating the segmented objects, the objects are automatically merged by a matching algorithm, method.

14. In the method according to claim 1, further, a process of calculating the positions of the segmented objects using the tile positions stored in the data store, comprising, method.

15. In the method according to claim 1, further, (a) an image of the living body object based on the stored adaptive sampling representation for the tile and the tile positions stored in the data store, (b) an image of the segmented object based on the data of the segmented object stored in the data store, or a combination thereof, a process of displaying, comprising, method.

16. In the method according to claim 1, further, a process of interfacing with a registration pipeline for aligning the volumes of living body objects within a common coordinate framework, and a process of mapping the segmented objects within the common coordinate framework, comprising, method.

17. In the method according to claim 1, the complete image is instead or optionally displayed at a low resolution, method.

18. In the method according to claim 1, further, a process of reading out and displaying a specific part of the given image data from the adaptive sampling representation, comprising, method.

19. In the method according to claim 1, the adaptive sampling representation stored for each tile is subjected to merging and cropping with overlapping tile regions such that the physical position of the living body object is uniquely described by one particle of the adaptive sampling representation stored in the data store for the tile, method.

20. A computer-based biomedical image processing system, A. An interface for receiving or accessing image data of a subject with respect to a living body, wherein the interface acquires, along an axis of a third spatial dimension, a series of two-dimensional image units each of which is a frame of n×m pixels of each parallel plane orthogonal to the axis in a depth direction, and the acquisition includes acquiring each series of two-dimensional image units in the depth direction along each orthogonal axis, each two-dimensional image unit of a series is a frame of n×m pixels of each parallel plane orthogonal to the corresponding axis of the series, different series have different orthogonal axes intersecting each plane at different tile positions, and the acquisition results in obtaining a plurality of series that function as tiles and represent an image volume of various parts of the living body imaged in three spatial dimensions, and each of the plurality of series is a separate tile representing a separate image volume, and the interface; B. A tile processing module communicably connected to the interface, wherein the tile processing module is automatically executed by a digital processor during image capture, i) compresses image data of a tile region with low information amount and maintains the maximum resolution of a region with high information amount to obtain an adaptive sampling representation for the tile, ii) determines a relative tile position with respect to other tiles, iii) segments the adaptive sampling representation for the tile with respect to a target object, and the segmentation includes extracting object feature amounts and corresponding object positions, iv) stores, in a data store in a computer memory, instead of storing a source image of the living body, the adaptive sampling representation for the tile, the determined tile position, the segmented object, the extracted object feature amounts, and the object positions, together with individual tile positions, so that the amount of the image data of the target is reduced and stored, A tile processing module that processes each tile individually; C. An aggregation module that responds to the tile processing module, wherein the aggregation module aggregates the segmented objects stored in the data store, and the aggregation automatically combines objects that appear two or more times between adjacent tiles among the segmented objects, to obtain a work list, object positions, and feature amounts for the segmented objects stored in the data store, the aggregation module; D. An output assembly having content based on the adaptive sampling representation for tiles, tile positions, and the work list for segmented objects stored in the data store, wherein the output assembly reads out tiles from the content only once and is configured to form a complete image at the maximum resolution of a part or the whole of the biological object, and by reading out tiles only once, the need for additional computer memory and / or processor resources is alleviated with respect to storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof of the complete image, the output assembly; A computer-based biomedical image processing system comprising the above.

21. The computer-based biomedical image processing system according to claim 20, wherein the interface receives or accesses the target image data from a source device, computer memory, archive, or any combination thereof.

22. The computer-based biomedical image processing system according to claim 20, wherein the target image data includes a biological image or a medical image captured / generated by a source device using any of X-ray, computed tomography, positron emission tomography, optical microscopy, electron microscopy, optical coherence tomography, magnetic resonance imaging, ultrasonic waves, and other digital imaging techniques, and the tile processing module is configured to perform processing quasi-real-time from the capture of the biological image or medical image.

23. In the computer-based biomedical image processing system according to claim 20, the image data of the object further includes any one or a combination of color, annotation, text, graphic, temporal feature amount, etc., a computer-based biomedical image processing system.

24. In the computer-based biomedical image processing system according to claim 20, the living body is any one of a tissue sample, a whole mouse brain, a cross-section of a human brain, other parts of a mammalian brain, and other biological substances of the object, a computer-based biomedical image processing system.

25. In the computer-based biomedical image processing system according to claim 20, the tile processing module individually performs the adaptive sampling compression for different fluorescence channels of the image data of the object, a computer-based biomedical image processing system.

26. In the computer-based biomedical image processing system according to claim 20, the tile processing module performs the adaptive sampling compression on the image data of the object for various periods, a computer-based biomedical image processing system.

27. In the computer-based biomedical image processing system according to claim 20, the tile processing module uses the adaptive particle representation (APR) technology to compress the image data of the tile region with less information amount, a computer-based biomedical image processing system.

28. In the computer-based biomedical image processing system according to claim 20, the system can further read out and display a specific part of the image data of the object from the adaptive sampling representation stored in the data store, a computer-based biomedical image processing system.

29. In the computer-based biomedical image processing system according to claim 20, the complete image from the output assembly is optionally or alternatively displayed at a low resolution, a computer-based biomedical image processing system.

30. In the computer-based biomedical image processing system according to claim 20, further (i) Interface connection with a registration pipeline for aligning the volumes of various living organisms within a common coordinate system, and (ii) a registration interface that enables mapping of the segmented objects within the common coordinate system, A computer-based biomedical image processing system comprising the same.

31. In the computer-based biomedical image processing system according to claim 20, the tile processing module, for the adaptive sampling representation, (i) Image filtering processing and subsequent threshold processing, and (ii) Applying a classifier formed by any of machine learning algorithms (but not limited to these) such as random forest, support vector machine, neural network, etc. trained with similar data, A computer-based biomedical image processing system that segments an object by applying the classifier.

32. In the computer-based biomedical image processing system according to claim 20, when the aggregation module aggregates the segmented objects, the computer-based biomedical image processing system that automatically combines the objects with each other by a matching algorithm.

33. In the computer-based biomedical image processing system according to claim 20, the tile processing module further calculates the position of the segmented object using the tile positions stored in the data store.

34. In the computer-based biomedical image processing system according to claim 20, further, A display module connected to display the image of the living organism based on the stored adaptive sampling representation for the tile, the tile positions stored in the data store, and the data of the segmented objects stored in the data store, A computer-based biomedical image processing system comprising the same.

35. In the computer-based biomedical image processing system according to claim 20, the tile processing module stores the adaptive sampling representations of the respective tiles obtained by combining and cropping in the overlapping tile regions in the data store, and each of the stored tiles uniquely represents the physical position in the biological object by a single particle in the adaptive sampling representation of the tiles stored in the data store. A computer-based biomedical image processing system.

36. An image processing method implemented on a computer, comprising: A. For given image data of a biological object, along the axis of the third spatial dimension, a series of two-dimensional image units that are frames of n×m pixels of each parallel plane orthogonal to the axis are obtained in the depth direction, and each series of two-dimensional image units along each orthogonal axis is obtained in the depth direction. Each two-dimensional image unit of a series is a frame of n×m pixels of each parallel plane orthogonal to the corresponding axis of the same series, and different series have different orthogonal axes intersecting each plane at different tile positions. By the acquisition, a plurality of series functioning as tiles are obtained, representing an image volume obtained by imaging various parts of the biological object in three spatial dimensions. Each of the plurality of series is a separate tile representing a separate image volume. The process of acquisition, B. i) A sub-process of obtaining an adaptive sampling representation for the tile by compressing the image data in a tile region with low information content and maintaining the high resolution in a region with high information content, ii) A sub-process of obtaining the relative tile positions with respect to other tiles, iii) A sub-process of segmenting the adaptive sampling representation for the tile with respect to a target object, the segmentation including extracting object feature amounts and corresponding object positions, and iv) Instead of storing the source image of the biological object in a data store in a computer memory, the adaptive sampling representation for the tile, the obtained tile positions, the segmented object, the extracted object feature amounts, and the object positions are stored in the data store such that the given image data is stored with a reduced amount together with the individual tile positions. By a sub-process of storing in the data store, The process of automatically processing each tile individually by a computer processor, C. A process of aggregating the segmented objects stored in the data store, the aggregation including obtaining a work list for the segmented objects stored in the data store by automatically merging objects that appear two or more times between adjacent tiles among the segmented objects. D. A process of forming a complete image of the maximum resolution of part or all of the biological object from only one readout of a tile based on the adaptive sampling representation for the tile, the tile position, and the work list for the segmented objects stored in the data store, wherein performing only one readout of the tile alleviates the need for additional computer memory and / or processor resources with respect to storage, retrieval, communication, distribution, processing, analysis, display, and any combination thereof of the complete image. A method comprising the above.

37. In the method according to claim 36, for a target tile, process A and process B are performed during imaging of the source image, and process C is performed after imaging of the image corresponding to the target tile.

38. A computer-based biomedical image processing system, A. An interface for receiving or accessing target image data for a biological object, the interface obtaining, in the depth direction, a series of two-dimensional image units each of which is a frame of n×m pixels of each parallel plane orthogonal to the axis along the axis of the third spatial dimension, the obtaining including obtaining each series of two-dimensional image units in the depth direction along each orthogonal axis, each two-dimensional image unit of a series being a frame of n×m pixels of each parallel plane orthogonal to the corresponding axis of the series, different series having different orthogonal axes intersecting each plane at different tile positions, and the obtaining resulting in a plurality of series functioning as tiles representing an image volume of various parts of the biological object imaged in three spatial dimensions, each of the plurality of series being a separate tile representing a separate image volume. B. A tile processing module communicably connected to the interface, the tile processing module being automatically executed by a digital processor. i) Compress the image data of the tile area with less information volume and maintain the maximum resolution of the area with large information volume, thereby obtaining an adaptive sampling representation for the tile; ii) Determine the relative tile position with respect to other tiles; iii) Segment the adaptive sampling representation for the tile with respect to the target object, and the segmentation includes extracting object feature amounts and corresponding object positions; iv) Instead of storing the source image of the biological object in the data store in the computer memory, store the adaptive sampling representation for the tile, the determined tile position, the segmented object, the extracted object feature amounts, and the object positions in the data store so that the image data of the target is stored with its amount reduced, together with the individual tile positions; A tile processing module that processes each tile individually; C. An aggregation module that responds to the tile processing module, and the aggregation module aggregates the segmented objects stored in the data store, and the aggregation includes automatically merging the objects that appear two or more times between adjacent tiles among the segmented objects, to obtain a work list, object positions, and feature amounts for the segmented objects stored in the data store, including an aggregation module; D. An output assembly having content based on the adaptive sampling representation for the tile, the tile position, and the work list for the segmented object stored in the data store, and the output assembly is configured to read out the tile from the content only once and form a complete image with the maximum resolution of a part or the whole of the biological object. Since the tile is read out only once, the need for additional computer memory and / or processor resources is alleviated with respect to any of storage, retrieval, communication, distribution, processing, analysis, display, and combinations thereof of the complete image, including an output assembly; A computer-based biological medical image processing system comprising.

39. In the computer-based biomedical image processing system according to claim 38, the image data of the subject includes a biological image or a medical image captured and generated by a source device, and the tile processing module is configured to perform processing in near real-time from the capture of the biological image or the medical image. A computer-based biomedical image processing system.

Images