Variable compression, deconversion, and restoration of medical images based on diagnostic and therapeutic relevance

Abstract

The image processing platform acquires an input image of the sample and globally downconverts the input image into a downconverted image. After globally downconverting the input image, the image processing platform simultaneously globally upconverts the downconverted image into an upconverted image using a generative adversarial network (GAN) model configured to reconstruct an image including features corresponding to the sample and classify multiple regions of the downconverted image based on cellular morphology and / or diagnostic relevance. The image processing platform communicates the upconverted image to a communication network for delivery to a client device. TIFF2025533563000002.tif84147

Description

[Technical Field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 377,005, entitled "Variable Compression, De-Resolution, and Restoration of a Medical Image Based Upon Diagnostic and Therapeutic Relevance," filed September 23, 2022, which is incorporated herein by reference in its entirety.

[0002] Technical Field This application relates to the field of digital pathology, i.e., systems and methods for acquiring and processing electronic pathology slide images for their acquisition, encoding, compression, storage, transmission, reconstruction, display, navigation, evaluation, diagnosis, and annotation into the resulting pathology report. [Background technology]

[0003] background In the field of pathology, mounted slides of pathology specimens can be converted into digital images of the entire slide for subsequent use in computer-based diagnostic workflows. Examples of systems for performing such conversions are included in Bacus (US Pat. No. 8,625,920) and Soenksen (US Pat. No. 6,711,283).

[0004] However, after two decades of technological advances and slow but gradual market adoption, most pathologists still use optical microscopes rather than scanned slide images for their primary diagnostic workflow. One reason is that whole-slide digital images are so large—as much as 4.8 gigapixels. Equivalent to a 4K computer display with a resolution of 3840 x 2160 pixels, a typical whole-slide digital image would represent approximately 500 full screens of detailed visual content. Such vast image files generally load slowly and are cumbersome to navigate compared to the instantaneous movement of a slide at a human fingertip. Pathologists generally prefer to physically touch specimen slides with their own hands, which provides a direct and profound connection to the patient, given the high stakes of the diagnostic outcome.

[0005] In addition to the problems associated with capturing and storing whole-slide digital images, the diagnostic workflow involving viewing such images can be inefficient and unintuitive compared to the experience of viewing a physical specimen slide through a microscope. Current software solutions for digital diagnostic workflows typically follow conventions of computerized user interface design, such as click boxes, radio buttons, scroll bars, and text boxes.

[0006] To improve their value proposition, many of these software-based solutions include artificial intelligence (AI) to aid in the diagnostic process through cell, pattern, or feature recognition. However, most pathologists still choose not to outsource such crucial tasks to third-party software solutions. Furthermore, slide scanner equipment costs between $20,000 and $300,000, depending on features and capacity. This represents more than a year's salary for most pathologists in the United States. Senior pathologists within 10 years of retirement face diminishing returns on such expensive reinvention of their established careers. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] US$8,625,920 [Patent Document 2] US6,711,283 Summary of the Invention

[0008] overview The present disclosure addresses the above-mentioned needs by identifying, cataloging, and spatially mapping known cellular, intracellular, and extracellular morphologies to aid in the subsequent reconstruction and super-resolution of high-resolution images. The approach described herein further evaluates, quantifies, and spatially maps and defines regions of the original image to include two or more levels of diagnostic relevance. Each such region is then assigned an optimal compression level, generally inversely correlated with its diagnostic relevance. This takes advantage of file size benefits from the inherent fact that normal and normal cells are generally healthy and of low diagnostic relevance in diagnosing one or more pathologies. This same aforementioned regularity and / or normality makes healthy tissue regions and cellular morphologies more suitable for AI-assisted super-resolution circuitry, such as generative adversarial networks (GANs). By preferentially designating dedicated GANs for each subtype of tissue, cell, condition, or morphology, the approach described herein can more accurately reconstruct the original high-resolution image from its down-converted, pixel-shifted progeny.

[0009] An advantage of the approach described herein is the reduction of file sizes for storage and transmission. High-resolution images can be reproduced on demand using increasingly high amounts of power in mobile phone processors and neural network software and firmware implementations. This advantage exists in addition to the compression provided by file formats such as JPEG-2000, whose lowest compression settings approximate a lossless method, although the total number of pixels stored is reduced. [Brief explanation of the drawings]

[0010] For a better understanding of the various described implementations, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout the drawings, in which:

[0011] [Figure 1] 1 is a diagram of an image processing environment configured to apply relevance-based variable compression, de-resolution, and restoration of medical images, according to some implementations. [Figure 2A] FIG. 1 is a diagram of a medical image acquisition process, according to some implementations. [Figure 2B] FIG. 1 is a diagram of a medical image acquisition process, according to some implementations. [Figure 2C] FIG. 1 is a diagram of a medical image acquisition process, according to some implementations. [Figure 3] A diagram of the alpha layer and metadata generation process, according to some implementations. [Figure 4A] A diagram of the deconversion, compression, and pre-verification process according to some implementations. [Figure 4B] A diagram of the deconversion, compression, and pre-verification process according to some implementations. [Figure 4C] A diagram of the deconversion, compression, and pre-verification process according to some implementations. [Figure 5] A diagram of the super-resolution, decompression, and display process according to some implementations. [Figure 6] FIG. 1 is a diagram of a pixel shifting process, according to some implementations. [Figure 7] 1 is a flowchart of a process for applying variable relevance-based compression, deconversion, and restoration of medical images, according to some implementations. [Figure 8A] 1 is a diagram of a system for displaying and interacting with medical images, according to some implementations. [Figure 8B]1 is a diagram of a system for displaying and interacting with medical images, according to some implementations. [Figure 9A] 1 is a diagram of a device configured to interact with medical images, according to some implementations. [Figure 9B] 1 is a diagram of a device configured to interact with medical images, according to some implementations. [Figure 9C] 1 is a diagram of a device configured to interact with medical images, according to some implementations. [Figure 9D] 1 is a diagram of a device configured to interact with medical images, according to some implementations. [Figure 10] 1 is a diagram of a device configured to interact with medical images, according to some implementations. [Figure 11] 1 illustrates multiple modes of use of a device configured to interact with medical images, according to some implementations. [Figure 12] 1 illustrates modes of use of a device configured to interact with medical images, according to some implementations. [Figure 13] 1 is a diagram of a system for collaborative interaction with medical images, according to some implementations. [Figure 14] FIG. 1 is a diagram of a system for interacting with medical images using facial gestures, according to some implementations. [Figure 15] FIG. 1 is a diagram of a system for interacting with medical images using facial gestures, according to some implementations. [Figure 16] FIG. 1 is a diagram of a system for interacting with medical images using facial gestures, according to some implementations. DETAILED DESCRIPTION OF THE INVENTION

[0012] Detailed Description This disclosure describes systems and methods for using machine vision and AI for better image compression, transmission, super-resolution rendering, and subsequent diagnostic workflow of medical images. By pre-identifying, mapping, and indexing diagnostically relevant tissue features and image regions against a library of known cells, tissue types, and features, the systems and methods described herein enable faithful subsequent reconstruction of the original compressed image, as well as faster navigation of the image during the diagnostic workflow. Using one or more peripheral controls, the systems and methods described herein use a diagnostic feature index to snap each successive diagnostically relevant tissue feature or image region to the pathologist's focus of attention.

[0013] The present disclosure aims to isolate and extract diagnostically relevant (i.e., suspicious and / or potentially cancerous) cells, tissues, and image regions and provide them with less lossy compression than the remaining less diagnostically relevant cells, tissues, and image regions, which by definition are more normal and regular in their state, attributes, and morphology, individually and in aggregates. Such normal and regular image content is therefore more suitable for higher levels of compression of various types, such as wavelets, token libraries such as LZW, color compression, pixel-shift super-resolution, run-length coding, and others.

[0014] Generally, less relevant cellular and tissue features are more normal or regular, and therefore more predictable, and amenable to advantageous deconversion and compression, allowing the original image or near-original image to be preserved, with minimal or no compression applied, to ensure the highest level of accuracy for those image regions most critical to extracting the more relevant cells and tissues, diagnosing medical problems, and determining therapeutic course of action. By compressing less relevant image regions and preserving more relevant image regions, the systems and methods described herein optimally balance the trade-off between more workable image file size and pathologist confidence.

[0015] FIG. 1 is a diagram of an image processing environment 100 configured to apply relevance-based variable compression, deconversion, and restoration of medical images, according to some implementations.

[0016] Specifically, environment 100 illustrates an electronic network 130 that may be connected to server system 102, including hosting partners such as hospitals, laboratories, and / or physician offices. For example, physician servers, hospital servers, clinical test servers, research lab servers, and / or laboratory information systems may each be connected to electronic network 130, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. Server system 102 includes a processing device configured to implement an image processing platform 110, including an image acquisition module 112, an image mapping / classifier module 114, an image compression module 116, and a DICOM compliance engine 118, each of which is discussed in more detail below.

[0017] The environment 100 facilitates efficient and reliable remote access to medical images between the pathologist device 150 and the collaborator device 160. The devices 150 / 160 are electronic devices, sometimes referred to as client devices, associated with each user. The devices 150 / 160 may include, but are not limited to, smartphones, tablet computers, laptop computers, desktop computers, smart cards, voice assistant devices, or other known or yet-to-be-discovered technologies (e.g., combinations of hardware and software) having structure and / or capabilities similar to the mobile devices or computer peripherals described herein. In some implementations, the devices 150 / 160 may include peripherals such as dials configured to navigate areas of the medical images, the features of which are disclosed in more detail below. The devices 150 / 160 are communicatively coupled to the server system 102 using communications capabilities (e.g., modems, walkie-talkies, radios, etc.) for communicating over the network 130. AI partner device 140 may approximate the functionality of devices 150 / 160 or otherwise assist with certain aspects of image analysis, as described in more detail below.

[0018] The server system 102 is communicatively coupled to the devices 140-160 by one or more communication networks 130. The communication network(s) 130 are configured to convey communications (messages, signals, transmissions, etc.). The communications include various types of information and / or instructions, including, but not limited to, data, commands, bits, symbols, voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, and / or any combination thereof. The communication network(s) 130 use one or more communication protocols, such as Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), near field communication (NFC), ultra-wideband (UWB), radio frequency identification (RFID), infrared radio, inductive radio, ZigBee, Z-Wave, 6LoWPAN, Thread, 4G, 5G, and the like. Such protocols may be used to send and receive communications using one or more transmitters, receivers, or transceivers. For example, wired communications (e.g., wired serial communications) may use technologies suitable for wired communications, short-range communications (e.g., Bluetooth) may use technologies suitable for close-proximity communications, and long-range communications (e.g., GSM, CDMA, Wi-Fi, wide area networks (WANs), local area networks (LANs), etc.) may use technologies suitable for remote communications over some distance (e.g., via the Internet). In general, communications network(s) 130 may include or otherwise use any known or yet to be discovered wired or wireless communications technology.

[0019] The server system 102 (e.g., including one or more hospital servers, clinical test servers, research lab servers, and / or lab information systems) may generate or otherwise acquire images of one or more patient cytology samples, histopathology samples, slides of cytology samples, digitized images of histopathology sample slides, or any combination thereof. The server system 102 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsy or cytology information, etc. The server system 102 processes the digitized slide images and transmits the processed images to the devices 150 / 160 via the network 130. The server system 102 may include one or more storage devices 120 for storing the aforementioned images and processed image data. The server system 102 may also include processing devices, such as one or more processors, each including one or more processing cores, for processing the images and data stored on the storage device 120. The server system 102 may further include one or more machine learning tools or capabilities, the features of which are described in more detail below. Additionally or alternatively, the present disclosure (or portions of the systems and methods of the present disclosure) may be implemented on a local processing device (eg, a laptop).

[0020] The server system 102 may be implemented on one or more standalone data processing devices or a distributed network of computers. In some implementations, the server system 102 also employs various virtual devices and / or services from third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing and / or infrastructure resources for the server system 102. In some implementations, the server system 102 includes, but is not limited to, a handheld computer, a tablet computer, a laptop computer, a desktop computer, or a combination of any two or more of these or other data processing devices.

[0021] Storage 120 includes non-transitory computer-readable storage media, such as volatile memory (e.g., one or more random access memory devices) and / or non-volatile memory (e.g., one or more flash memory devices, magnetic disk storage devices, optical disk storage devices, or other non-volatile solid-state storage devices). The memory may include one or more storage devices located remotely from the processor(s). The memory stores programs (described herein as modules and corresponding to sets of instructions) that, when executed by the processor(s), cause the server system 102 to perform the functions described herein. The modules (e.g., 112-118) and data described herein need not be implemented as separate programs, procedures, modules, or data structures. Thus, various subsets of these modules and data may be combined or otherwise rearranged in various implementations.

[0022] FIG. 2A is a diagram of a medical image acquisition process 200 according to some implementations. In some implementations, one or more source images 202, 204 are scanned at a resolution corresponding to the capabilities of the scan camera used to scan the source images (referred to as the native scan resolution). While the diagram shows two images 202 and 204, other implementations may include one image or three or more images (e.g., three images, four images, etc.). For implementations in which multiple images are scanned, the multiple images may be pixel-shifted images. Specifically, each successive image may be offset by at least one pixel in at least one direction. This may dramatically reduce scan time and cut the cost of the scanner camera in half.

[0023] The source images 202 and 204 may be acquired by the image acquisition module 112 of the image processing platform 110 (FIG. 1). That is, the source images may be acquired locally at the hospital where the server system 102 is located, or remotely from the location of the server system 102.

[0024] Each pixel-shifted source image 202, 204 is a component of a final source image 210. In other words, the pixel-shifted source images 202, 204 are combined into a final source image 210. In some implementations, the final source image 210 is either a composite image constructed from optimal pixel(s) from within a non-planar volume z-stack, or a series of tiles that exceed the resolution of the original image sensor and therefore could not be incident on the image sensor of a scanning camera.

[0025] Final source image 210 includes one or more regions (e.g., regions 220a-220e), generally referred to as regions of interest, or more specifically as diagnostically relevant regions or therapeutically relevant regions. Such a region is diagnostically relevant, for example, if it contains cellular morphology relevant to a patient's diagnosis associated with tissue in image 210. Similarly, such a region is therapeutically relevant, for example, if it contains features relevant to a patient's treatment outcome associated with tissue in image 210. Throughout this disclosure, "diagnostic relevance" may refer to diagnostic relevance and / or therapeutic relevance.

[0026] In some implementations, the therapeutic nexus may include such factors that may reasonably help guide treatment selection, suitability for various drug testing or research programs, and the most appropriate specific location or institution for treatment. Stated differently, the therapeutic nexus corresponds to the match of a sample / patient with a research, drug trial, and / or treatment regimen.

[0027] In some implementations, diagnostic relevance is not limited to merely indicating the definitive presence or likelihood of cancer, dysplasia, fibrosis, inflammation, or other pathology, but also includes any feature within a sample that is relevant to a positive or negative determination of pathology, such as the presence or status of cancer, the degree of remission or minimal residual disease, or the degree and location of dysplasia, or the degree and composition of fibrosis or inflammation, whether such indication resides in the state or morphology of organelles, cells, tissues, protein matrices, fluids, organic polymers, or anatomical structures, or in secondary evidence provided through fixation, clearing, processing, sectioning, mounting, staining, and any other method implemented within a histology laboratory. An exemplary list of relevant features that the image mapping / classifier module 114 may index within an AI scanner includes cells, nuclei, organelles, lacunae, glands, blood vessels, signet ring cells, atypical polymorphisms, and atypical collagen structures.

[0028] In some implementations, diagnostic and / or therapeutic relevance may be based on quantifying several events of objects, cells, or conditions within a specific proximity or region. For example, diagnostic and / or therapeutic relevance may be based on how many cells of a specific type are within a certain proximity to each other or contained in a field of a defined size. For example, diagnostic and / or therapeutic relevance may be based on counting mitotic figures within a specific region. As another example, in breast cancer, one of the criteria for grading is the number of mitotic figures in a tumor in 10 high-power (40x) fields. If the microscope specifications are known, a 40x field can also be considered an area. Melanoma uses the number of mitoses per square millimeter. The mitotic count is important for grading malignant tumors and can help distinguish between benign and malignant tumors. Gastrointestinal stromal tumors and many other adenosarcomas / soft tissue tumors are stratified into benign, uncertain malignant potential, and malignant based on the number of mitotic figures, often 50 or more, in multiple high-power fields.

[0029] In some implementations, the relevance regions (also called zones, subsets, or portions) may be based on levels of relevance at multiple layers, types or contexts of relevance, or any combination thereof, as discussed in more detail below (see, e.g., FIG. 3).

[0030] FIG. 2B is a diagram of a medical image acquisition process 230 according to some implementations. The acquisition and processing of source images 202 and 204 is similar to that described above with reference to FIG. 2A. However, in process 230, final source image 212 includes multiple levels of regions of interest, or diagnostic and / or therapeutic relevance levels. For example, regions 220a-220e have a first diagnostic and / or therapeutic relevance level, regions 222a-222c have a second diagnostic and / or therapeutic relevance level lower than the first level, and regions 224a-224b have a third diagnostic and / or therapeutic relevance level lower than the second level. Regions of final source image 212 not included in regions 220a-220e, 222a-222c, and 224a-224b (e.g., region 226) have a diagnostic and / or therapeutic relevance level lower than the third level. The regions with the highest level of relevance (e.g., 220a) may be most useful to a pathologist in determining a patient's diagnosis or therapeutic course of action, while the regions with the lowest level of relevance (e.g., 226) may be least useful (or of no benefit) to a pathologist in determining a patient's diagnosis or therapeutic course of action.

[0031] FIG. 2C is a diagram of a medical image acquisition process 250 according to some implementations. The acquisition and processing of source images 202 and 204 are similar to those described above with reference to FIGS. 2A-2B. However, in process 250, the final source image may be visualized in a three-dimensional format, with regions separated in the z-axis by their respective diagnostic and / or therapeutic relevance levels. In this manner, the visualization of FIG. 2C of the final source image may resemble a topographical map, with the relevance levels of multiple regions of the image shown as heights. Such visualization displays a three-dimensional fly-through model of the specimen in the final source image, along with labels, snap-to-points, and / or relevance hierarchies, shown as various z-heights. In some implementations, the pixels most relevant to a given (x, y) location in the image may be identified in a volume z-stack.

[0032] In some implementations, the image mapping / classifier module 114 uses a third-party input aggregator (TPI or TPIA) to identify and map areas of relevance in the final source image. In some implementations, the image mapping / classifier module 114 uses relevance ratings provided by in-house systems or personnel to identify and map areas of relevance in the final source image.

[0033] In some implementations, the diagnostic and / or therapeutic association may be a composite image from multiple diagnostic and / or therapeutic association assessments provided from several different sources.

[0034] In some implementations, in analyzing the composite z-stack, the image mapping / classification module 114 evaluates diagnostic and / or therapeutic relevance regardless of focus clarity. A feature or image region may be highly diagnostically and / or therapeutically relevant compared to features above and below it in the z-axis, but may be out of focus or not have sharp focus. Some systems using selective processes within the z-stack may generally select the best focus, not diagnostic relevance, which is an entirely different attribute. The disclosed system creates a composite image with defocused features that can then be sharpened, for example, by an algorithm or neural network, particularly because they were initially determined and selected based on diagnostic and / or therapeutic relevance.

[0035] In some implementations, the relevance assessments of the samples in the final source image features and / or image regions (e.g., regions 220, 222, and 224) may be organized into hierarchical classes, categories, or associated metatags.

[0036] In some implementations, priority data (e.g., pre-designated spot locations or regions of areas of a slide or specimen features that a pathologist or oncologist deems important) and metadata (e.g., health information about the patient, or the community / environment in which the patient lives, works, or conducts business) can serve as input to a machine vision / learning system in the image mapping / classifier module 114 to evaluate and index specimen features and score the diagnostic and / or therapeutic relevance of various regions in the final source image.

[0037] In some implementations, the image mapping / classifier module 114 may respond to revisions by a pathologist to the relevance ratings assigned to image regions. In some implementations, such revisions may track an individual relevance model for the pathologist. In some implementations, such revisions may inform the continuous improvement of an overall model for determining the relevance of specific cell morphologies or other features of medical images. Multiple such profiles for constantly improving modeling and agreement at each level may be based on pathologist self-agreement, agreement within multi-practitioner pathology practices, and / or agreement to committee standards. In some implementations, there may be an overall shift of any or all of the above toward better sensitivity and specificity (thus advancing the field).

[0038] In some implementations, the image mapping / classifier module 114 generates a cell index for the sample in the source image and stores the cell index as metadata corresponding to the image. The data in the cell index may be mapped as a vector file associated with the coordinate values ​​of the image where certain features of the cell index are present. The metadata may be contained in one or more separate files contained in a file wrapper, in the file header of the image file, or stereographically within the image file.

[0039] A cell index is an accurate, complete, descriptive, prescriptive, quantitative, and qualitative assessment of a biopsy sample in a source image. The cell index may include intracellular, extracellular, qualitative, and / or quantitative attributes of the sample. The sample type may include any of bone, blood, fluid, tissue, etc. The cell index itself may be any of an index, census, compendium, map, list, etc. The cell index may also be referred to as a cell index and compression key (CICK). In some implementations, the cell index may include pre-annotations.

[0040] An exemplary list of cellular index features (indexable features of a specimen) included in the metadata for a given medical image is: abscess, absorptive cells (enterocytes), acid-fast bacilli / bacilli, acinar (alveolar) glands, alveoli / alveolar clusters, adipocytes, adipose tissue, adventitia, alveolar spaces, alveoli, amacrine cells, ameloblasts, apocrine cells, arrector pili muscle, arterioles, arteries, astrocytes, atherosclerotic plaques, abnormal mitosis / mitosis. , bacillus / bacillus, bacteria / bacteria, band, Barr body, basement membrane, basophil, basophilic, basophilic spot, bile duct, blast cell, blood vessel, bone, bone marrow, Bowman's capsule, Brunner's gland, Brush margin, Bunina body, Cabot ring, calcification, canalicular system, trabecular bone, capillary, capsule, myocardium, heart valve, cartilage, cell, cementum, central vein, centriole, chief cell, chondroblast, chondrocyte, chromatin, cilium / cilia, collagen, columnar cell, connective tissue, corpus a Rubican's cord, cord, Billroth's cord, cornea, corona radiata, corpus luteum, cortex, Cowdry body, crypt, intestinal crypt, cyst, cytoplasmic vacuole, placental cell, dentin, dermis, Descemet's membrane, duct, ductal epithelium, Dutcher's body, dystrophic calcification, eccrine duct, eccrine gland, elastic fiber, enamel, endocardium, endocrine cell, endometrial gland, endometriosis, endomysium, endoperiosteum, endothelial cell, endothelial cell group, enteroendocrine cell, eosinophil, eosinophilia, epithelial cell , epithelium, outer nerve membrane, epithelium, red blood cell (erythrocyte), exocrine cell, external elastic lamina, fascia, muscle bundle, fenestrated endothelium, fibrin, fibropurulent exudate, fibroblast, fibroma, fibrosis, fibrovascular core, fibrils, hair follicle, fruiting body, fungal hyphae / mycelium, fungal yeast, fungus, ganglion cell, gangrenous necrosis, gastric pit, biocenter, giant cell, gland, glial cell, glomerulus / glomerular cluster, goblet cell, granular follicle, granule, granulocyte, granuloma, granulocyte, stroma, hair, hair follicle, halo cell, Hassar's corpuscle, Haversian duct, Heine's corpuscle, Helicobacter / Helicobacter pylori / H, pylori, Helicobacter-like organism, hematogones, hemosiderin, hepatocyte, Hilm, histiocyte, Hofbauer cell, Howell-Jolly body, hyaline cartilage, hydroxyapatite, hyperpigmentation, hyphae / mycelium, hypnozoite, immunoblast, inclusion body, internal circular muscle, internal elastic lamina, interstitial cell of Cajal, interstitial cell of Leydig,Intranuclear inclusion body, islet cell, islet of Langerhans, mesangial cell, keratin, keratin pearl, keratinocyte, Kupffer cell (liver macrophage), bleb, lacuna, lamina propria, Langerhans cell, lens, leukocyte (white blood cell), Lewy body, Leydig cell, ligament, lipofuscin, loose connective tissue, cavity, intraluminal contour, intraluminal space, Luschka duct, lymphatic vessel, lymphoblast, lymphocyte, lymphoid follicle, lysosome, macrophage, maculae adenocarcinoma, maculae densa, mast cell, megakaryocyte, meiosis, melanin, membrane, Merkel cell, mesangial cell, mesenchymal tissue, mesenchymal cell, mesenchymal membrane, metamyelocyte, Michalis-Guttmann body, microcalcification, microorganism, microvilli, division / mitosis, mitotic figure, Molluscum body, monocytoblast, Monocyte, mucosa, mucous cell, muscle tissue, muscularis externa, muscularis mucosae, myeloblast, bone marrow cell, Meinenteric (Auerbach) plexus, Nabothian cyst, necrosis, nerve, nerve tissue, nerve cell, neutrophil, Nissl substance, nuclear membrane, nucleolus / nucleolus cluster, nucleus / nucleus cluster, oligodendrocyte, oocyte, ordinary connective tissue, organelle, osteoblast, osteoclast, osteocyte, osteoblastic tissue, longitudinal muscle externa, egg, oxyntic cell, oxyphilic cell, Pacinian corpuscle , pancreatic adenoid cells, Paneth cells, papilla / papillary cluster, papillary dermis, Pappenheimer's bodies, parasites, parasympathetic ganglion cells, parenchyma, parietal (oxynt) cells, periarterial lymphoid sheath, pericytes, pericalon, periosteum, Peyer's patches, phagocytes, phagocytosis, Pick bodies, pigment, plasma cells, cell membrane, Plasmodium, platelets, pleomorphic, alveolar cells, podocytes, polychromatic erythroblasts, portal vein trifurcation, proerythroblasts, promyelocytes, psammocytes, medulla, Pulmonary Kinkinger cells, Purkinje fibers, cone cells, red medulla, Reed-Sternberg cells, Reinke crystals, respiratory epithelium, rete ovary, processus reticularis, ridges of the rete testis, reticular dermis, reticulocytes, retinal pigment epithelium, Russell bodies, sarcorema, schistocytes, sebaceous glands, secretory epithelium, seminal vesicles, seminiferous tubules, seroma, serosa, serotonin cells, Sertoli cells, signet ring cells, simple columnar epithelium, simple cuboidal epithelium, simple squamous epithelium, sinus / sinusoid, sinusoid, skeletal muscle, smooth muscle cells, smidge cells, sperm precursor cells, spermatids, spermatogonia, spermatogonial clusters, spermatozoa, spindle cells, spirochetes, sporozoites, multilayered squamous epithelium, stroma, subcapsular space, subcutaneous adipose tissue, submucosa, submucosal (Meissner) plexus, submucosal glands, surface mucous cells, sweat glandsThese include synovial membrane, tendon, plate endplate, thrombus, thymocyte, treponeme, tubular gland, duct, urothelium, follicle, vascular network, vein, venule, villi, viral envelope body, Wharton-Finkeldey body, Wharton's jelly, white pulp, Wolffian duct, woven bone, and yeast. This list is provided for illustrative purposes and is not intended to be exhaustive or limit the scope of the present disclosure.

[0041] In some implementations, other data related to the above features may also be indexable, such as attributes and / or parameters of any of the foregoing features, including, but not limited to, width, height, thickness, diameter, optical density, color bias, opacity, polarization, dimensional distortion from normal, angular bias, rotation state, sharpness of focus, etc.

[0042] In some implementations, other types of features may also be indexable. For example, confounding or distorting features may be indexable, such as cracked slides, cracked coverslips, excess coverslip media, trapped air bubbles, smears, fingerprints, protein speck content, hair content, folded tissue edges, tissue wrinkles, non-repeating tissue thickness variations, recurring tissue thickness variations, dried top residue, dried bottom residue, misplaced labels, detached coverslips, extra glass slides, etc. These features may be associated with the medical image, but not with the sample itself. Thus, in some implementations, these features may be logged but are not included in the cell index.

[0043] Any subset of the aforementioned features in the cell index can be generated as a vector file or spline, which can describe an area encompassing multiple (e.g., thousands) cells in an efficient manner. Thus, in some implementations, an exemplary ratio of pixels to logged features can be at least 1,000 to 1. In some implementations, the cell index can include matching metatags, since some features may be more than one (e.g., a skin cell may be part of a gland wall). Because most normal / healthy tissues contain tens or hundreds of similar cells, the index can be run-length encoded, so that the net ratio should still be well below 1,000 to 1. Also, because at least half of the index may be unused for any given sample, the image mapping / classifier module 114 can use a contiguous per-sample raster, which can reduce the size of entries in the index to an 8-bit (1-byte) identifier. In this way, the per-sample vocabulary can be limited to well below 256 feature types.

[0044] 3 is a diagram of an alpha layer and metadata generation process 300, according to some implementations. In some implementations, hierarchical levels of diagnostic relevance are established through the system's machine vision analysis (implemented by the image mapping / classifier module 114), with each layer containing one or more regions of the overall image, and thus a percentage of the overall pixels. Such regions may then be separated into separate alpha layers, each containing a hierarchy of diagnostic relevance. These alpha layer images are then compressed using the optimal compression type for that composite image, with the compression level being inversely related to diagnostic relevance.

[0045] For example, in process 300, a medical image 302 (e.g., image 214, corresponding to FIG. 2C ) is acquired by image acquisition module 112 and mapped by image mapping / classifier module 114. As a result of the mapping, a metadata layer 303 corresponding to image layer 302 is generated, which includes a cell index including diagnostic relevance scores per region and per feature. This metadata layer may also include a recommended sequential diagnostic workflow (described in more detail below) including pre-annotations, or a composite image of such recommendations aggregated from multiple diagnostic sources.

[0046] Image 302 and cell index 303 are divided into three separate alpha layers, 304, 306, and 308, each defined with a selected compression type, format, ratio, and / or degree optimal for image fidelity. For example, layer 304 (comprising image file or layer 304a and metadata file or layer 304b) contains the most diagnostically relevant portions of the image (e.g., 220a-220e, FIG. 2C), layer 306 (comprising image file or layer 306a and metadata file or layer 306b) contains the moderately diagnostically relevant portions of the image (e.g., 222a-222c, FIG. 2C), and layer 308 (comprising image file or layer 308a and metadata file or layer 308b) contains the less diagnostically relevant portions of the image (e.g., 224a-224b, FIG. 2C).

[0047] For each layer, every region and feature may then be assigned a dedicated reconstruction super-resolution GAN (layers 314, 316, 318) from a palette of such dedicated GANs based on cellular attributes, morphology, coloration, pathology state, etc. (described in more detail below). In this and other methods, the metadata may be imperative rather than merely descriptive.

[0048] 4A is a diagram of a deconversion, compression, and pre-verification process 400a according to some implementations. In some implementations, the process 400a is performed by the image processing platform 110 of the server system 102, which includes the image mapping / classifier module 114 and the image compression module 116.

[0049] A source image 402 (e.g., corresponding to images 210, 212, or 214 of FIGS. 2A-2C, or image 302 of FIG. 3) is analyzed by mapping / classifier module 114 to determine regions of diagnostic and / or therapeutic relevance as described above. One or more regions having the highest level of relevance, or a level of relevance that meets a threshold, are optionally extracted into one or more alpha layers 404 (e.g., corresponding to layer 304a of FIG. 3).

[0050] The mapping / classifier module 114 generates metadata 406 for each layer. The image compression module 116 deconverts and / or compresses each of the image layers that have a lower diagnostic and / or therapeutic relevance than that of the layer 404 or that do not meet the relevance threshold, generating one or more downconverted and / or compressed images 408. Each downconverted and / or compressed image 408 corresponds to a metadata layer 406, which includes data that instructs an upsampling and / or decompression algorithm on how to restore the image. In this manner, this upsampling and / or decompression process is pre-verified by upsampling and / or decompressing (reconstructing) one or more of the downconverted and / or compressed images 408 using the corresponding metadata 406 to generate a reconstructed image 410.

[0051] The image processing platform 110 compares the reconstructed image 410 with the original image 402a without the extracted region 404 and determines the differences between the two images based on the comparison. Differences are expected due to the nature of deconverting, compressing, up-resolving, and / or decompressing the image file (e.g., using lossy algorithms). However, because the image 408 was sourced from a lower tier of diagnostic and / or therapeutic relevance, some lost detail (e.g., sharpness) may be acceptable due to the accuracy of the machine learning / vision models used to reconstruct the image at the client device. These machine learning / vision models are tested in a pre-verification phase, including this comparison step. If the differences exceed a threshold, the comparison step fails, and the process is repeated with the deconversion / compression step.

[0052] Upon failure, one or more machine learning / vision models are updated, and the image is deconverted and / or compressed again using the updated machine learning / vision models. One such example of a machine learning / vision model is a GAN. A GAN uses a generator circuit (e.g., a convolutional neural network (CNN)) to generate an image and a discriminator circuit (e.g., another CNN) to determine whether the generated image is true or false. A reconstruction circuit and / or algorithm 432 (also referred to as an upconversion and / or decompression circuit and / or algorithm 432) is used to reconstruct the image 408 into the image 410 and may be implemented as a generator network of the GAN, and a comparison circuit and / or algorithm 434 is used to compare the reconstructed image 410 with the original image 402a and may be implemented as a discriminator network of the GAN. In this way, each time the comparison results in a failure, the reconstruction circuit 432 learns and updates its generator model to provide a more realistic image 410 according to the updated generator model.

[0053] If the comparison of the reconstructed image 410 to the original image 402a is successful (e.g., the difference is less than a threshold), the image processing platform 110 packages the extraction layer(s) 404, the deconverted and / or compressed image(s) 408, and the latest metadata 406, including the cell index and the latest pre-validated version of the machine vision / learning model (e.g., a GAN model) for use in reconstructing the deconverted and / or compressed image(s) 408. The images 404, 408, and the metadata 406 are packaged into one or more files for transmission over the network 130 to one or more client devices 150 / 160.

[0054] In some implementations, the comparison of the reconstructed image 410 to the original image 402a may be close to successful but not actually successful. Stated another way, the difference may be below the failure threshold but above the success threshold by a threshold amount. Rather than continuing to refine the machine learning / vision model and requiring more time for pre-validation, the difference data 412 itself may be included in a file packaged for transmission to one or more client devices 150 / 160 over the network 130.

[0055] In some implementations, each image layer (e.g., 404 and 408) may be deconverted and / or compressed using a different algorithm depending on what is optimized for that layer. For example, different layers may be compressed using different compression ratios, compression methods, or compression times. In other words, each layer may not only be compressed using a different compression algorithm or type, but may also be compressed to a different degree. Because each layer is created based on diagnostic and / or therapeutic relevance, the re-resolution and compression of the image layer is based on the diagnostic and / or therapeutic relevance. Specifically, more relevant image layers (containing more relevant regions) may be deconverted and / or compressed to a greater extent than less relevant image layers (containing less relevant regions), and may even use entirely different re-resolution and / or compression algorithms.

[0056] In some implementations, the compression module 116 may generate a region map of stepwise variable compression for fidelity and diagnostic and / or therapeutic relevance for each image 408. In this manner, the image compression module 116 may perform gradient-variable compression for diagnostic and / or therapeutic relevance combined with gradient-variable deconversion for diagnostic and / or therapeutic relevance, followed by super-resolution of the cell-indexed medical image using a predetermined library of tissue-specific neural networks. These down-converted and compressed images are the result of several non-redundant, complementary empirical measurements and evaluations of the original image.

[0057] In some implementations, using a machine learning / vision model in the reconstruction step 432 involves mapping the indexed features of the cell index against a library or palette of dedicated machine learning / vision models. For example, in an implementation using a GAN model, the indexed features are mapped against a library or palette of dedicated GAN models. In other words, a GAN or any other dedicated tissue-specific, feature-type-specific, or morphology-specific machine learning model may be used to convert parametrically characterized instances of cell morphology into super-resolved pixels or vector graphic elements, which may then be rasterized into pixels. In some implementations, other machine learning / vision models may be used in addition to or as an alternative to a GAN model, such as stable diffusion or any other type of machine learning / vision model not yet known or discovered.

[0058] In some implementations, the aforementioned GAN palette can be implemented as an arbitrary modular library of machine learning / vision super-resolution models, each specialized by cell and / or tissue type, state, or morphology. For example, regions 220a-220e in FIG. 2A may be associated with different models in the palette of models, each specialized to reconstruct one of the cell and / or tissue types, states, or morphologies present in the respective region.

[0059] 4A is repeated on the client device(s) 150 / 160 using the same metadata and machine learning / vision models, which are then packaged with the image layers as described above in step 436. Thus, the machine learning / vision (e.g., GAN) palette-mapped images may be sent to the client device(s) for subsequent decompression and super-resolution.

[0060] In some implementations, pre-verification reconstruction (steps 432-434) implements a final image check to ensure that the image data sent to the client device(s) can be faithfully reconstructed to the same fidelity as the original image 402. Referring to FIG. 4A , if the check of the reconstructed image 410 does not result in sufficient fidelity (“fail”), a first loop (subsequent path “A”) isolates mismatched pixels and / or features in the image, tries different GAN or other machine learning / vision models for the particular mismatched pixels / features, and repeats the check (steps 432-434). This loop may be repeated multiple times until the check is successful (“succeeds”) and an image file is packaged for transmission to the client device(s).

[0061] In some implementations, after the loop iterates for a threshold amount of time, or once the difference in comparison step 434 falls below a threshold, the original (faithful) pixels contained in the difference (between images 410 and 402a) may be separated into a correction layer 412 that is packaged into a file for transmission to client device(s). Additionally or alternatively, the correction GAN may generate a fine correction alpha layer for a particular layer, region, or the entire image that is subject to the difference (between images 410 and 402a).

[0062] In some implementations, the diagnostic and / or therapeutically relevant alpha layer(s) 404, deconversion and / or compression layer(s) 408, cell quantification index and other metadata layer 406, GAN map (or other machine learning / vision mapping) layer, and optional correction layer 412 are all packaged, separated, and maintained within a single file wrapper and remotely hosted as a known image check key (KICK) file. In some implementations, the KICK file may represent approximately 30% of the original file size, a significant improvement for the purpose of optimizing limited storage resources by significantly reducing the storage load on the server system 102 and client device(s) 150 / 160. In some implementations, the original image 402 may be deleted from storage 120 after the KICK file is packaged, replaced by the KICK file itself, and made available for future viewing requests.

[0063] In some implementations, the diagnostic and / or therapeutically relevant alpha layer(s) 404, the cell quantification index and other metadata layer 406, the GAN map (or other machine learning / vision mapping) layer, and the optional correction layer 412 are packaged as a “key” file 414 separate from the downconversion and / or compression layer 408 of the main package file. In some implementations, this key file may be approximately ¼ the file size of the original image 402, and the main file may also be ¼ the original image 402, providing the client device(s) with a stronger reduction in their storage load. Thus, the client device(s) receive the diagnostic and / or therapeutically relevant key file for super-resolution and the main file for combining with the key file to create a complete image (e.g., appearing like the original image 402).

[0064] The following example illustrates the features described above with reference to FIG. 4A. The AI ​​engine (mapping / classification 114) performs quantitative mapping, cataloging (classification), and mapping of all features, morphology, organelles, nuclei, cell orientation, cell state, etc., of the input slide image 402. The most relevant features and / or regions are separated into an alpha layer 404, isolating the pristine elements of the source image (e.g., less than 30% of all pixels). The remaining features and / or regions are then also separated into an alpha layer, and each is deconverted and / or compressed based on its respective diagnostic / therapeutic relevance score. This takes advantage of the fact that healthy tissue is generally more normal and regular, and therefore more predictable for dedicated neural networks. A final check step verifies the reconstruction (comparison 434) and adjusts quality metrics until fidelity is perfect (or exceeds the predetermined threshold described above). Finally, a reduced bitrate compensation layer 412 is also created (but only if necessary).

[0065] 4B shows an alternative implementation of the deconversion, compression, and pre-verification process 400b, according to some implementations. Process 400b (FIG. 4B) is identical to process 400a (FIG. 4A) except for the placement and functionality of mapper / classifier 114.

[0066] The process 400b supports a first approach in which the input image 402 is first mapped by the mapping / classification module 114, which then guides the deconversion or compression process 116 and the extraction of the more / most relevant features and / or regions. Specifically, according to the classification and mapping of the relevant features and / or regions, the module 114 instructs the deconversion / compression module 116 which features and / or regions to process into the corresponding alpha layer and which parts to extract in the most relevant alpha layer 404.

[0067] Process 400b supports a second approach in which the input image 402 is globally deconverted in module 116, and the low-resolution image is used for classifier mapping, which then guides feature extraction and selective variable compression or up / down conversion or other processing (e.g., color reduction). Specifically, because the deconverted image 408 has less data to process, based on the globally deconverted input image 408, mapping / classifier module 114 can more efficiently determine relevant features and / or regions for extraction and subsequent down / up conversion or compression / decompression. This added efficiency saves time, allowing the input image 402 to be processed more quickly, with little or no impact on quality.

[0068] Process 400b supports any combination of the first and second approaches described above, such as a first (simpler) classifier at full resolution (as in the first approach), followed by a richer / complete classifier at lower resolution (as in the second approach).

[0069] 4C shows another implementation of a medical image processing scheme 400c, according to some implementations. Features of process 400c (FIG. 4C) that are identical to features in process 400a (FIG. 4A) and process 400b (FIG. 4B) are similarly labeled.

[0070] In process 400c, the input image 402 is divided into multiple tiles (unless the tiles are provided by an image scanner). The tile size for each image is based on the priorities of speed, quality, and compressibility. Once processed by the mapper / classifier 114 (as described above with reference to processes 400a and 400b), the input image (each tile) is deconverted and / or compressed. Because an entire image portion (e.g., an entire tile) is deconverted and / or compressed, this step may be referred to as global downconversion and / or global compression. Thus, the entire image (all of the tiles) is globally downconverted and / or compressed. In one embodiment, the resulting deconverted image layer 408 may be 50% or less of the size of the input image 402 (or more in other embodiments).

[0071] In some implementations, a full-resolution image may be reconstructed from portions of the image that have undergone each / all various levels of processing (deconversion / compression / etc.) by managing at the tile level. For example, highly processed tiles may be combined together with unchanged tiles. If this creates noticeable visual artifacts, a dithering mask may be used to mitigate the edges of one or more of the adjusted tiles.

[0072] The deconverted / compressed image data 408 is then upconverted and / or decompressed back to its original resolution / size, producing an output image 410 having the same resolution and / or size / quality as that of the input image 402. Because the entire image (all of the tiles) is upconverted and / or decompressed, this step may also be referred to as global upconversion and / or global decompression. As described above with reference to processes 400a and 400b, the upconvert / decompress module 432 uses a GAN to predictively improve the clarity of the image data 408, which results in an output image 410 that is at least as detailed as (and in some cases more detailed than) the original input image 402.

[0073] Simultaneously (in parallel) with upconverting / decompressing the image data 408 using module 432, mapper / classifier 114 analyzes the downconverted / compressed image data 408 (line G in FIG. 4C ) and directs subsequent processing. Specifically, if at least a portion (and in some cases, all) of the image data is subjected to feature cataloging and relevance classification while the image data is being downconverted / compressed, the mapping and classification process is more efficient, thereby saving time in generating a fully mapped and classified output image 410 from an unmapped and unclassified input image 402. In other words, the analysis in mapper / classifier module 114 may be much quicker because it is performed using image data at a lower resolution. Based on the classifier 114's aforementioned analysis, one or more portions of the input image 402 may be manipulated to provide higher-quality portions of the input image that correspond to diagnostically / therapeutically relevant features.

[0074] The upconverted / decompressed output image 410 has the same resolution and quality as the input image 402, but run-length coding works better on the resulting (clarified image), making the output image 410 more compressible. Also, in some implementations, the re-upconversion is performed by filling in predictable pixels (using a GAN or other AI upconversion process), with the original pixels removed during the deconversion / compression process. This approach provides sharpening of the output image 410, which may be better than the original input image 402. In other words, pixels missing as a result of the deconversion and / or compression are refilled with predicted pixels during the upconversion / decompression process, thereby increasing the number of predictable pixels by simply overwriting the previously removed pixels with pixels resulting from the prediction used by the upconversion / decompression module 432.

[0075] The output image 410 is provided (in some implementations, together with the metadata layer 406) to the client device(s) 150 / 160 via the packaging 436 and the network 130 (as described above with reference to processes 400a and 400b).

[0076] 5 is a diagram of a super-resolution, decompression, and display process 500 according to some implementations. Process 500 is performed on client device(s) 150 / 160 in response to receiving a KICK file or key and main file from server system 102 over network 130. The diagnostic and / or therapeutic relevance alpha layer(s) 404, deconvert and / or compress layer(s) 408, cell quantification index and other metadata layer 406, GAN map (or other machine learning / vision map) layer, and optional correction layers are unpackaged for separate processing. The deconversion and / or compression layer(s) 408 use the GAN Pap data in combination with metadata 406 (e.g., cell index) to super-resolve (also called up-conversion and / or decompression), thereby generating a reconstructed image 402a (corresponding to the final version of image 402a of FIG. 4 at server system 102 during the pre-verification process), which is combined with diagnostic and / or therapeutically relevant alpha layer(s) 404 to generate a reconstruction of the original image 402, which has the same level of fidelity as the original image acquired by image acquisition module 112 at server system 102.

[0077] The upconvert / decompress function upconverts and / or decompresses the image(s) 408 using the cell index metadata and / or dedicated GAN map (or other machine learning / vision map) received in file(s) from the server system 102. As an optional final upconversion process (e.g., after combining images 404 and 402a), the restored image 402 may be further upconverted beyond the original sensor resolution using the pixel-shifting resolution change function described herein (e.g., with reference to FIG. 6).

[0078] In some implementations, included in the metadata layer 406 are diagnostic workflow instructions, including the order in which to display the relevance regions (e.g., 220a, followed by 220b, followed by 220c, etc. (FIG. 2A)). In some implementations, the diagnostic workflow is predicted by TPI or in-house AI prediction algorithm(s). In this way, not only can the diagnosis outcome be predicted (e.g., in the form of diagnostic and / or treatment relevance regions of the medical image), but also the workflow for performing the diagnosis (the pathologist's diagnostic workflow). In other words, the prediction algorithm uses the client device(s) 150 / 160 to determine which regions of the medical image the pathologist wants to see first, second, etc.

[0079] By including diagnostic workflow instructions in the metadata layer 406, the image processing platform 110 can encode not only the medical image (e.g., the entire slide image), but also an animated or guided view of the pathologist's AI-predicted workflow, which is typically only 25%-50% of the entire image. In some implementations, the prediction algorithm determines not only the order of regions, but also the zoom level, angle, which regions to display adjacent to each other, etc.

[0080] In some implementations, during the actual diagnostic workflow (while the pathologist is viewing various regions of the image), additions or revisions to the workflow may be communicated to the client device(s) 150 / 160 and fed back to the predictive model at the server system 102. Such additions and revisions may be used to update the predictive workflow model used by the server system 102.

[0081] In some implementations, such additions or revisions may be associated with user-specific profiles, allowing each pathologist to personalize their own predictive workflow. These user-specific profiles may track individual relevance models corresponding to individual pathologists. These user-specific profiles may additionally or alternatively aid in the continuous improvement of the overall model used by the server system 102 for diagnostic workflow prediction for all pathologists. Thus, while some systems convert pathology slides into images or rich data, the disclosed system converts slides into a powerful, self-contained diagnostic workflow that is simple and efficient enough to function across user devices (e.g., smartphones) anytime, anywhere, allowing pathologists to review medical images (e.g., full slide images) without the need to travel to an office or use specialized viewing equipment.

[0082] For implementations in which an image having the original resolution is provided to the client device (e.g., as described above with reference to process 400c of Figure 4C), there is no need to upconvert the received image and unpackage the additional alpha layer(s) 404 and combine them with the upconverted received image as shown in Figure 5. In these scenarios, the image itself may simply be decompressed and / or provided directly to the client device.

[0083] FIG. 6 is a diagram of a pixel shifting process 600, according to some implementations. A source image (e.g., 402) is pixel shifted and downconverted into multiple images in a retro source proxy layer. For example, one group of 256 pixels may be converted into four pixel-shifted groups of 16 pixels. For each group of pixels, each pixel is a combined (e.g., averaged) version of 16 pixels from the source image. The retro source proxy layer may be deconverted image(s) 408 that are packaged and transmitted to client device(s) 150 / 160 as described herein with reference to FIGS. 4-5. To reconstruct an image, the retro source proxy groups of pixels may be upscaled and overlapping pixel values ​​may be combined (e.g., averaged) and overlapped to form a reconstructed image (e.g., staked in a pixel-shifted manner).

[0084] FIG. 7 is a flow diagram illustrating an example process 700 for compressing and transmitting, reconstructing, and presenting images for diagnostic annotation, according to some implementations. The process may be governed by instructions stored in computer memory or a non-transitory computer-readable storage medium (e.g., storage 120). The instructions may be included in one or more programs stored on the non-transitory computer-readable storage medium. When executed by one or more processors, the instructions cause the server system 102 to perform the process. The non-transitory computer-readable storage medium may include one or more solid-state storage devices (e.g., flash memory), magnetic or optical disk storage devices, or other non-volatile memory devices. The instructions may include source code, assembly language code, object code, or any other instruction format that can be interpreted by one or more processors. Some operations within the process may be combined, and the order of some operations may be changed.

[0085] Upon acquiring a medical image (e.g., a whole slide image) (e.g., 402, Figures 4A-4C), the AIDICOM compliance engine 118 of the server system 102 removes patient-specific data from the image. The server system 102 uses the aggregated TPI and score to identify (704) and score (706) tissue type to isolate diagnostically and / or therapeutically relevant TPI alpha layer(s) (e.g., 404, Figures 4A-4C). The server system 102 creates (708) a metadata layer (e.g., 406, Figures 4A-4C) to inform subsequent super-resolution. The server system 102 creates (710) a pixel-shifted, down-converted retro-source proxy layer (e.g., 408, Figures 4A-4C). The server system 102 tests (712), pre-verifies, and adds corrections to the super-resolution (e.g., steps 432, 434 and path "A" in Figures 4A-4C). The server system 102 determines the fidelity (714) and separates and retains the final key (e.g., steps 434 and 436, FIGS. 4A-4C). The server system 102 packages the smaller aggregate file into a new wrapper (716) (e.g., step 436, FIGS. 4A-4C).

[0086] In some implementations, the input image 402 described above with reference to Figures 4A-7 may be part of a z-stack consisting of multiple images for each corresponding z-height of the sample. Samples are typically prepared and imaged by flattening the z-stack into a single layer. By flattening the z-stack, the user loses access to z-field navigation and any insight that might be observed from the ability to utilize such navigation. The following discussion describes implementations for restoring and / or simulating a previously flattened z-stack, providing a true navigable z-field regeneration.

[0087] In scans where an intact z-stack (multiple images for each corresponding z-height) exists, the image processing platform 110 may capture only value-added pixels of a feature relative to pixels of the same feature on upper and lower layers. Such value-added pixels may include those with better focus, but the evaluation may also include diagnostic and therapeutic relevance. Thus, a value-added pixel may be a pixel with a focus corresponding to a predetermined sharpness threshold and / or a pixel that is part of a feature corresponding to a predetermined threshold of diagnostic or therapeutic relevance.

[0088] Because the image mapping module 114 catalogs the entire input image, the image processing platform 110 can determine and store the z-levels corresponding to each feature (and portions of each feature) in the image. As a result, the image processing platform 110 can determine which features are behind or on top of other features in the z-field. For example, the image processing platform 110 can determine which blood cells are above other blood cells and then run one or more predictive GAN models to predict pixel values ​​for the obscured portions of the underlying blood cells. Thus, the image processing platform 110 can recover a navigable z-field from a flat image.

[0089] In particular, pixels obscured by overlapping cells or other features can be restored using the techniques described herein, as long as those "underlying" pixels differ from the predicted model. Using this very economical small number of pixels retained as the alpha layer, the image processing platform 110 can approximate a true navigation experience in the z-axis, either using a control such as a focus knob (e.g., a control on the peripheral 803 described below) or using biometric navigation features (e.g., described below with reference to Figures 14-16). For example, a zoom gesture can lead to z-navigation when a predetermined or dynamically triggered maximum threshold is reached.

[0090] In some implementations, using a small number of the aforementioned very economical pixels retained as an alpha layer, the image processing platform 110 can create virtual slides from within a deep z-field, at an angle, or even at various non-planar virtual surfaces. This feature can be useful in 3D imaging, such as lattice light sheets, or in simulating or superimposing slide images overlaid on 3D radiographic image(s). This use case can include not only slide image features far beyond the resolution of the radiographic image, but also the transfer of staining from the slide image to adjacent radiographic pixels or voxels. From only a few contiguous sections on the slide image, such an approach can virtually generate a 3D "slide image" by interpolating or estimating voxels based on the contextual overlay of radiographic voxels and slide image pixels. Such 3D mocks (moving resolution elements) can then be utilized as a representation of the primary diagnostic workflow or used as the basis for rendering several "virtual slices" with most or all of the detail and usefulness otherwise afforded by only the stained tissue slide (vs. the radiographic image).

[0091] In some implementations, using the very economical small number of pixels mentioned above, retained as an alpha layer, image processing platform 110 can determine not only how deep multiple features are in the z-field, but also where they are relative to focus. Based on that, image processing platform 110 can computationally back out other optical aberrations (such as spherical aberration) and eliminate spectral differences associated with different distances from focus and different feature shapes, or can leave the spectral differences in place to give the user true navigation of the z-field. Furthermore, image processing platform 110 can place "underlying" features further behind a given feature by moving them further out of focus to provide even more of a true z-navigation effect.

[0092] The following section describes the economy of pixels retained by the predictive model (e.g., GAN mapping). In other words, the image processing platform may preserve only those pixels (within the image data provided to packager 436) that differ from the various predictions described above with reference to the GAN model in processes 400a, 400b, and 400c (Figures 4A-4C).

[0093] In some implementations, the image processing platform retains only pixels that differ from the GAN prediction. In some implementations, the image processing platform may selectively replace some pixels in a manner that enhances usability. Prediction efficiency is informed by an ever-growing library of models and parameters to characterize each pixel / feature and thereby reproduce authenticity.

[0094] Specifically, the predictive GAN-based reconstruction (“reconstruction”) of the image is checked for fidelity (e.g., as described above with reference to modules 432, 434, and 114) and, in some cases, fine-tuned to maximize fidelity until a specified threshold of fidelity is met, and the remainder (e.g., final correction layer 412, also called residual coding or error residual coding) is saved as a separate file or as a pyramidal image file (e.g., packaged in step 436), or can be included in any number of meta-layers contained in a file wrapper.

[0095] As the GAN model continually improves in accurately predicting what a super-resolution representation of any given cell or cell feature or biomarker will look like, the residuals become smaller and smaller. Thus, the image processing platform may not need to store any pixels except for those that differ from the prediction. This is particularly useful for reducing the file size of volumetric images (also called "z-stacks") or "voxels" of 3D radiological images. Thus, the image processing platform only needs to store those pixels that differ from the generative model's prediction, which becomes an increasingly accurate prediction.

[0096] This aforementioned "residual" efficiency is a key part of the image processing techniques described herein because the largest images at the highest magnifications will conversely have the largest ratio of pixels (or voxels) per cell, and the predictive models will perform best at predictively reconstructing "typical / normal / healthy" cells, leading to very high compression ratios.

[0097] For example, a typical blood smear is composed (primarily) of healthy red blood cells and suspicious white blood cells. Healthy red blood cells, while of very low diagnostic relevance, may outnumber white blood cells by 600 to 1. The GAN model described herein can highly accurately predict over 4,500 pixels for each red blood cell based on only 36 concise parameters, comprising approximately 72 bytes of data. This constitutes a compression ratio of approximately 99.9% for the red blood cell region of the image. Red blood cells outnumber white blood cells by 600 times, resulting in a crude potential compression of 99.9 × (600 / 601). Even with tissue models, compression ratios can exceed 70%.

[0098] The aforementioned error-residual efficiency applies to the volumetric images described above, including z-stack slide images, as well as the resulting light sheet microscopy and / or radiology volumetric images.

[0099] Stated another way, any pixel that the various predictive models described herein (working individually or in any combination) can accurately predict can then be deconverted (e.g., as described above with reference to modules 116 and 432) and faithfully reconstructed. This can include entire red blood cells, cell or nuclear edges, or organelles within cells, or chromatids within cells, and their respective granularities, Auer rods, mitotic chromosomes, etc. This can also include having alpha layers (image regions) for cells below various layers (volumetric images) in a z-stack. For all of the foregoing examples, the image processing platform need only preserve those pixels that deviate from the predictive models.

[0100] In general, the aforementioned error-residual efficiency applies to anything that can be imaged, including the readout of an NGS flow cell, chromosome karyotypes (sometimes located one-to-one), volumetric layers in a lattice lightsheet image, etc.

[0101] FIG. 8A is a diagram of a system 800 for displaying and interacting with medical images, according to some implementations. Server system 102 transmits image files (e.g., 414, FIGS. 4A-4C) over network 130 to client device 150 / 160. Client device 150 / 160 includes, for example, smartphone 801, optionally communicatively coupled to peripheral device 803 and display device 802 for interacting with and viewing the restored image (e.g., 402, FIG. 5). In some implementations, such as FIG. 8B, peripheral devices are not required, and the client device may be only smartphone 801, only display device 802, or smartphone 801 coupled to display device 802.

[0102] 9A-9D and 10A-10B are diagrams of peripherals (e.g., 803, FIG. 8) configured to interact with medical images, according to some implementations. The peripherals shown in these figures can be used to advance through the image field as part of a diagnostic workflow specified in a metadata layer (e.g., 406, FIGS. 4A-4C) associated with the image. Additional details regarding the peripherals are disclosed below.

[0103] 11 and 12A-12C illustrate several modes of use of a peripheral device (e.g., 803, FIG. 8) configured to interact with medical images, according to some implementations. Further details regarding these modes of use are disclosed below.

[0104] 13 is a diagram of a system 1300 for collaborative interaction with medical images, according to some implementations. In some implementations, peripheral movements at a first client device 150 are transmitted to one or more second client devices 160, thereby causing peripherals associated with the one or more second client devices 160 to perform the same movements as the peripherals at the first client device 150. In such implementations, a lead pathologist (150) may train others (160) to perform a diagnostic workflow in a manner that allows the others to have the same visual and tactile experience as the lead pathologist as the lead pathologist navigates image regions as part of the diagnostic workflow. Further details regarding these collaborative interactions are disclosed below.

[0105] 14-16 are system diagrams for interacting with medical images using facial gestures, according to some implementations. In some implementations, facial gestures may allow a user of a client device to control viewing and navigation of image regions of a medical image on a display. Further details regarding these systems are disclosed below.

[0106] Some embodiments of the present disclosure use AI to create downconverted and pixel-shifted pseudo-source or "retro-source proxy layer" images to improve compression for reconstructing and upconverting or super-resolving a suitably faithful facsimile of the original high-resolution source image or portions of that source, such as a tile or tiles or regions within a group of tiles or regions. Such embodiments may also retain reference portions of the original image for machine learning discriminator circuitry in the upconversion process. Such embodiments may also utilize such reference portions of the original image for machine learning discriminator circuitry in pre-verification and / or validation of the upconversion process.

[0107] Some embodiments of the present disclosure improve compression through the use of AI systems to analyze regions and features of high-resolution pathology slide images and / or biopsy samples and compare them to a continuously updated "known tissue library," which includes raster and / or vector and / or wavelet example data of various types of cells, cytoplasm, organelles, angiogenesis, tumors, cysts, lumens, glands, lacunae, lamina, and other diagnostically relevant features, as described above for the presence of various conditions such as metastasis, mitosis, miosis, carcinogenesis, and apoptosis. Such libraries may include isolated, specific, general, or probabilistic parameter data for examples such as morphology type, area, width, diameter, contrast, rotation state, distortion, aspect ratio, presence of protein, etc. Such libraries may include individual examples or parameters of said examples, such as statistical data such as median, mean, standard deviation, etc. Such libraries may include, for one or more examples therein, correlations between examples or between examples and various factors, such as those found in patient data and / or metadata. In one such embodiment, the AI ​​system records the resulting attributes and parameters associated with the cells or regions as one or more metadata layers, mapping said metadata to Cartesian coordinates of the region of the sample, and / or mapping said metadata to indexed locations of formations detected in the image or sample, and / or mapping said metadata to their pixel locations within the image or portions of the image. Such metadata is then subsequently used by the AI ​​system or by another AI system or subsystem to support subsequent decompression and / or reconstruction, and / or upconversion or super-resolution of a suitable faithful facsimile of the original image.Such aforementioned metadata may be retained and stored and / or transmitted and / or mined as image layers, or may be stereographically stored within the pixels of an image layer, or may simultaneously be retained and stored as data files or arrays such as XML or HTML, or as ASCII text files or DB2 or DBT or CSV or JSON or MDB, or other indexable and searchable and / or otherwise minable formats or data modalities.

[0108] Some embodiments of the present disclosure use the metadata layer of the analysis of the preceding embodiments to simultaneously reconstruct an upconverted or super-resolved facsimile of the original image to improve compression through the use of an AI system to guide the upconversion or super-resolution process. Such operations help ensure that such facsimiles are performed on and with sufficient fidelity and consistency with the original high-resolution image when and subsequently performed. One such embodiment may employ a library of individual, dedicated generative adversarial networks (GANs) for such super-resolution, each specialized for a certain cell type, tissue type, tissue or cellular condition, or other useful, individual, and diagnostically relevant aspect of the image and / or sample or portion thereof. In one such embodiment, the system determines which dedicated GAN most accurately approximates the original image or region of the image and associates, or "maps," it to the location or feature of the region or sample for which it was validated. Such a "GAN map" is then retained as metadata, either indexed as a dataset or retained as an image layer. When maintained as an image layer, such metadata may utilize compression such as run-length encoding (RLE) since such attributes tend to apply to many consecutive or adjacent pixels or regions or specimen features.

[0109] Some embodiments of the present disclosure use AI software and / or hardware systems to aggregate groups of adjacent pixels from an original or synthetic image into lower-resolution pseudo-pixels for a “downconverted” pseudo-source image (or “retro-source proxy”), then repeat the aggregation and shift the next pseudo-source image by a fraction of one aggregated pseudo-pixel size in an approximation of a traditional “pixel shifting” process. The system uses one or more available algorithms, GANs, or other types of neural networks to ensure that, when recombined and upconverted, the “retro-source” image faithfully reconstructs the original image. In such verification, the machine learning system uses portions of the original source image as references in a discriminator or comparison loop to reconstruct a reasonably faithful facsimile of the original high-resolution source, or portions of that source, such as a tile, or a tile or region within a group of tiles or regions.

[0110] Some embodiments of the present disclosure use an AI software system to computationally or algorithmically combine multiple optically matched exposures to cancel sensor noise and create a "noise-reduced" source image. The noise-reduced source is then downconverted by combining adjacent pixels into square clusters of four, nine, or sixteen. This process is repeated to generate a series of downconverted images, each shifted by a fraction of the clustered spurious pixels, typically by a shift distance corresponding to one of the original native source pixels. The AI ​​system performs the recombination and upconversion process and verifies that the product of the newly generated pixel-shifted "retro-source proxy" images faithfully reproduces or sufficiently approximates the original native resolution and / or noise-reduced multi-exposure composite image that was the source of the deconversion and pixel-shift process. One such embodiment has a file size advantage: the total size of the retro-source proxy layer images is proportional to the number (N) of such images, while the upconversion result has a file size proportional to the square of that number (N). The upconverted results are generated on demand, while the smaller retro-source proxy layer images (individually and in aggregate) are the versions of the target content that are stored and transmitted. Also, because the AI ​​pre-reverses the fidelity, such embodiments minimize the number of retro-source proxy images (N) required for sufficient fidelity.

[0111] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within tiles of a multi-focal plane source image(s) (also referred to as a "z-stack" image set) that represent preferred image quality and / or diagnostic relevance and / or suitability for reduced and optimized color for any given Cartesian coordinate location of the imaged specimen, and aggregate such selected pixels or regions of pixels into a pseudo-source image of features and / or portions of the specimen that could not be imaged by the sensor because they did not lie within the matching plane or matching line of the line-scan sensor.

[0112] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within a source image or within such aforementioned selectively aggregated pseudo-source images that represent a favorable fit for a tissue-specific and / or pathology-specific graphic token palette.

[0113] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within a source image or within such aforementioned selectively aggregated pseudo-source images that represent favorable suitability for compression by run-length encoding.

[0114] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within a source image or within such aforementioned selectively aggregated pseudo-source images, which represent a favorable fit for wavelet compression.

[0115] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within a source image or within such aforementioned selectively aggregated pseudo-source images that, when calculated and / or extracted from that source or pseudo-source, leave a residue that has favorable suitability for one or more types of compression.

[0116] Some embodiments of the present disclosure use AI to improve compression, comparing such aforementioned extracted layers to the original source image or a partially extracted pseudo-source image or an aggregated pseudo-source image, and generating correction factors that, when applied to the extracted layer and / or remaining layer(s), improve the fidelity of the resulting reconstructed and / or upconverted image.

[0117] Some embodiments of the present disclosure improve the transmission and cloud-hosted viewing of stored slide images by selectively caching more diagnostically relevant image portions or reference tiles at such locations or in such infrastructure, resulting in superior speed or lower latency for user-pathologists during diagnostic workflows or collaborative consultations.

[0118] Some embodiments of the present disclosure improve the transmission and cloud-hosted viewing of stored slide images by using an AI system or subsystem to predictively pre-load images or portions of images at such locations or by using infrastructure that can directly facilitate one or more suitable collaborative resources.

[0119] Some embodiments of the present disclosure improve the transmission and cloud-hosted viewing of stored slide images by selectively pre-loading down-converted entire slide images, or portions thereof, to such locations or such infrastructure, providing superior speed or reduced latency to user-pathologists during diagnostic workflows or collaborative consultations.

[0120] Some embodiments of the present disclosure improve diagnostic workflows by using an AI system or subsystem to select and / or isolate and / or extract and / or preserve diagnostically relevant reference portions of an original or pseudo-source image. Such portions and specimen features contained therein are pre-indexed to the reconstructed image, and those indexed regions are mapped to detent features of a rotary scroll wheel (e.g., 803) for fast and accurate navigation of a large number of such features and locations.

[0121] Some embodiments of the present disclosure use AI systems or subsystems to improve diagnostic workflows, providing hands-free navigation, region selection, and annotation via voice commands, speech-to-text annotation, and eye and face tracking, particularly through measurement and precise tracking of the vestibulo-ocular reflex. Complete navigation and annotation actions may be simultaneously shared with collaborating or virtual colleagues across a network and across a broader area, enabling a real-time consultation service exchange in which diagnostic services can be aggregated and disseminated to populations and regions lacking such resources. Such real-time collaborative diagnosis is distinct from second opinion networks such as those described by Soenksen (US Pat. No. 11,211,170) in that it provides increased skills and increased credibility for younger and / or non-Western personnel. Such mentorship is crucial for improving the quality of care, both practical and perceived, in emerging countries and / or economically challenged communities.

[0122] Some embodiments of the present disclosure use AI systems or subsystems that are preferentially adapted and / or dedicated to specific types or combinations of tissue, morphology, and / or pathology to improve the visual quality and reconstruction fidelity of resulting images. One embodiment of such a dedicated system may include a generative adversarial network (GAN) for upconverting cells of a given tissue type determined to exemplify polymorphism. Another embodiment of such a dedicated system may include a generative adversarial network (GAN) for upconverting healthy and regular cells of a given tissue type. Another embodiment of such a dedicated system may include a generative adversarial network (GAN) for upconverting cells of a given tissue type determined to exemplify metastasis.

[0123] Some embodiments of the present disclosure use an AI system or subsystem to map pixels or groups of pixels, portions of an image or pseudo-image, or cells or groups of cells, or Cartesian coordinates or defined regions of an imaged sample against a library and / or palette of such aforementioned dedicated tissue-specific, and / or morphology-specific or otherwise dedicated GANs to improve the visual quality and reconstruction fidelity of the resulting image.

[0124] Some embodiments of the present disclosure use an AI system or subsystem to map pixels or groups of pixels, portions of an image or pseudo-image, or cells or groups of cells, or Cartesian coordinates or defined regions of an imaged sample against a library and / or palette of tissue-specific and / or morphology-specific or otherwise dedicated graphic tokens to improve the visual quality and reconstruction fidelity of the resulting image. In one such embodiment, the aforementioned dedicated token library may be dynamically updated using an AI system or subsystem that detects recurring and / or widespread occurrences of potential new graphic tokens, establishes new such tokens using a GAN, and appends them to the existing token library and / or tokens. Such a system may then retroactively apply the improved and / or expanded token library to previously processed images or portions of images using a discriminator network and reference portions of the images to ensure good and / or satisfactory resulting reconstructed image quality.

[0125] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within a source image or a selectively aggregated pseudo-source image that, when computationally extracted from the source or pseudo-source, leave a residue that has a favorable fit to the reduced color depth, including, but not limited to, optimal paletted colors of hematoxylin and eosin stain (H&E) or other stains.

[0126] Some embodiments of the present disclosure improve compression by using AI to select regions of pixels within a source image or a selectively aggregated pseudo-source image that, when computationally extracted from the source or pseudo-source, would have favorable compatibility with reduced color depth, including, but not limited to, optimally paletted colors or algorithmically compressed color profiles for tissues processed with hematoxylin and eosin stain (H&E) or other stains.

[0127] Some embodiments of the present disclosure use a camera and AI system or subsystem to improve diagnostic workflows, enabling hands-free navigation and vestibulo-ocular region selection and annotation, where the user's gaze is intentionally fixed on a selector element, such as a crosshair or selection box or highlighted region, line, circle, point, or polygon, or a dim, blinking, or scintillating highlighted region, line, circle, point, or polygon, and the user's head and / or face is then intentionally moved to indicate an intent to move an image or a portion of an image into and / or under said selector element. The camera and AI system detects such VOR activity and shifts the displayed image accordingly. In some implementations, tracking of the user's head and / or face can be disengaged or disabled in response to receiving a user input that temporarily or permanently disables the head and / or face navigation features. During this time, the user can re-enable head and / or face tracking via a second user input that instructs the system to recenter their face (re-establish a new original image) and resume head and / or face tracking.

[0128] Some embodiments of the present disclosure improve diagnostic workflow using a camera and AI system or subsystem to enable hands-free highlighting and annotation via the vestibulo-ocular reflex, where a user's gaze is intentionally fixed on a portion of a displayed image, and then the user's head and / or face is intentionally moved to indicate an intent to highlight or select that portion of the image. The camera and AI system detects such VOR activity and selects, tracks, or selects that portion of the displayed image accordingly. Subsequent voice-to-text capture annotates the actively selected region or specimen feature as verbally indicated by the pathologist.

[0129] Some embodiments of the present disclosure improve diagnostic workflows using cameras, microphones, and AI systems or subsystems to enable hands-free navigation, highlighting, and annotation by combining the vestibulo-ocular reflex (VOR) with spoken commands such as "highlight," "select," "deselect," "annotate," "circle," "square," "navigation," "polygon," "spline," "touch paint," "new layer," "mark," "pin here," "pause," "save spot," "compare," "split view," or other such commands typical of graphic and / or text editing.

[0130] Some embodiments of the present disclosure use a system or subsystem consisting of a camera, microphone, and AI software system to improve diagnostic workflows, enabling hands-free navigation via head and / or facial movements in conjunction with voice commands such as "vanity mirror." In such a mode, a user indicates an intent to increase image magnification by tilting their head toward the display. The system detects and tracks this movement in real time and adjusts the displayed image accordingly. Similarly, a user indicates an intent to pan left by turning their head left, or to pan up by tilting their head up or down by tilting their head down. The system detects and tracks this movement in real time and adjusts the displayed image accordingly. In such a "vanity mirror" mode, a user can verbally instruct the system to apply, or increase or decrease, a "scaling factor" such that a gentle movement can induce a large shift in the displayed image, or vice versa. Similarly, a user can verbally instruct the system to "reverse" the relationship between their head movement and the resulting shift in the displayed image in the aforementioned mode. In such a "vanity mirror" mode, the user can verbally instruct the system to apply or increase or decrease a "stabilization factor" so that the displayed image shifts in a smooth, jitter-free manner, regardless of finer and / or less intentional movements of the user's face or head. Such stabilization is an important feature for users with degenerative neuromuscular conditions. In some implementations, the user's head and / or face tracking is disengaged in response to receiving a user input that temporarily or permanently disables the head and / or face navigation features of the user input. The user can re-enable head and / or face tracking during this time via a second user input that instructs the system to recenter their face (re-establish a new original image) and resume head and / or face tracking.

[0131] In some implementations, the user's eye gaze is fixed on a displayed fixed element, and simultaneous facial and head movements are used to command movement of the displayed moving element and / or selection of a selected element, which may be associated with an actuation command.

[0132] In some implementations, the commanded movement is of an image relative to a fixed cursor, selection box, painting tool, mask designator, magnified selection zone, or area-designating graphic display element.

[0133] In some implementations, the commanded movement is of a cursor, selection box, paint tool, mask designator, magnified selection zone, area designation graphic display element relative to a fixed image or portion of an image or sample area.

[0134] In some implementations, the commanded movement is of a file name, folder name, or icon or thumbnail representing a file or folder, or multiple files or folders, to a fixed cursor or selection box, or an enlarged selection zone, or a file or folder designation graphical display element.

[0135] In some implementations, the commanded movement is of a file- or folder-designating graphical display element, such as a cursor or selection box, or a magnified selection zone, or a fixed file or folder name, or an icon or thumbnail representing a file or folder, or multiple thereof.

[0136] In some implementations, the commanded movement is of a fixed cursor or selection box, or an enlarged selection zone, or a file or folder designation graphical display element, or of a command or command list, or of a hierarchical command category or icon or thumbnail or preview representing a command or command category or multiple thereof.

[0137] In some implementations, the commanded movement is of a cursor or selection box, or an enlarged selection zone, or a file- or folder-designated graphical display element to a fixed command or command list, or a hierarchical command category or icon or thumbnail or preview representing a command or command category, or multiple thereof.

[0138] In some implementations, the commanded movement is of a fixed cursor or selection box, or an enlarged selection zone, or a setting value specification or setting selection graphical display element, or of a setting or settings list, or a hierarchical settings category or icon or thumbnail or preview representing a setting or settings category or multiple thereof.

[0139] In some implementations, the commanded movement is of a selection or setting designation graphical display element to a cursor or selection box, or an enlarged selection zone, or a fixed setting or settings list, or a hierarchical settings category or icon or thumbnail or preview representing a setting value or settings category, or multiple thereof.

[0140] The activation command is voice command activation, push button, scroll wheel, keystroke, touchpad, touchscreen, intentional eye blink, foot switch, roller ball, non-verbal voice activation, or (each / all of the preceding).

[0141] Some embodiments of the present disclosure improve diagnostic workflows using an AI system or subsystem consisting of a smartphone with a two-part phone cradle: a lower, stationary base and an upper, movable cradle that holds the phone horizontally with the display facing up. As the user manipulates the phone with their fingertips, the system displays a magnified slide image on the phone's display, as if the phone were an extreme magnifying glass sliding over an actual specimen, or as if a slide were sliding under an optical microscope. The smartphone's rear camera senses the movement of the lower base passing underneath, which can be illuminated as needed by the smartphone's rear LED. Communication can be further sensed using the phone's onboard sensors or by a Bluetooth-paired peripheral device with functionality, features, and structure similar to a wireless optical scroll mouse. The upper cradle can facilitate smooth and precise movement with low-friction pads and / or rollers between the upper cradle and the lower base.

[0142] Some embodiments of the present disclosure may implement the aforementioned "tabletop" mode in which the lower base is a table or desk surface and the aforementioned scroll mouse features are integrated into the upper cradle.

[0143] Some embodiments of the present disclosure may implement the aforementioned "tabletop" mode, in which the upper cradle is a typical smartphone case.

[0144] Some embodiments of the present disclosure may implement the aforementioned "tabletop" mode in which the smartphone is configured at an incline or adjustable incline, i.e., not parallel to the underlying surface. Such incline may be oriented to facilitate more effective face tracking by the phone's front-facing camera.

[0145] Some embodiments of the present disclosure may implement AI diagnostic recommendations as draft annotations that the user can affirm, revise, or reject at their discretion. Such draft pre-annotations may be presented synonymously with collaborative peer annotations. Such annotations may be presented without distinction between the recommendations of one or more human peers, or the recommendations of an AI "virtual pathologist," or anonymized previous annotations of the same user / pathologist. The system may resubmit previously evaluated slides to the user to truly measure self-concordance. Such concordance testing may be performed continuously by the system for several parts of the workflow process.

[0146] Some embodiments of the present disclosure may implement the aforementioned "tabletop" mode, where the incremental selection, review, and annotation of pre-identified diagnostically relevant sample features is controlled by facial movements and / or VOR and / or the phone's touchscreen. In such a modality, a user may conceivably complete the entire diagnostic workflow without grabbing their hands from the handhold on the phone, as a proxy for traditional slide manipulation.

[0147] Some embodiments of the present disclosure may implement the aforementioned face tracking navigation in a manner that smoothly progresses through various pre-identified diagnostically relevant features, cells, locations, or annotations.

[0148] Some embodiments of the present disclosure may implement the aforementioned face tracking navigation in a "snapping" or "popping" manner, progressing through various features, cells, locations, or annotations of pre-identified diagnostic relevance in a non-linear fashion for each indexed feature, location, or annotation, in a manner similar to the navigation behavior associated with the aforementioned scroll wheel embodiment. Such "snap" or "pop" navigation helps facilitate user review of specimens and images. In such navigation, auditory and visual cues indicate the progressive selection of each location or feature, with such cues contextually determined by the system and / or user-configurable settings. In such embodiments, the system may temporarily or persistently modify the sensitivity and / or scale of facial motion tracking to promote a more stable or fluid review experience for the user. In such embodiments, navigation may progress between features and / or locations while remaining at or near a single magnification, or alternatively, may reduce magnification before proceeding to the next location, or navigation may proceed to visually approximate an apparent flight or bounding arc of the z-axis. In one such embodiment, the system may reduce, attenuate, or ignore forward-looking aspects of face tracking in a momentary, or temporary, or persistent, or modal manner.

[0149] Some embodiments of the present disclosure can implement the aforementioned "tabletop" mode while Miracasting the display output to a television.

[0150] Some embodiments of the present disclosure improve diagnostic workflows using a system or subsystem consisting of a camera, microphone, and AI software system to enable manual tracking or manual tracking control of navigation, highlighting, and annotations.

[0151] Some embodiments of the present disclosure improve diagnostic workflow using a system or subsystem consisting of a touch-sensitive sensor, camera, microphone, and AI software system to obtain the aforementioned control modalities in any combination with a touchscreen, such as a smartphone placed on a desk or tabletop surface or held in a vertical or semi-vertical cradle.

[0152] Some embodiments of the present disclosure improve diagnostic workflow using a system consisting of an AI software system or subsystem and a 5G smartphone wirelessly interfaced with a nearby large-screen television in a display mode known as "Miracast." In such embodiments, slide images are cloud-hosted and streamed over a 5G mobile network. Voice command and speech-to-text transcription of annotations are accomplished through the smartphone's capabilities. VOR and face tracking navigation and / or selection are also accomplished by the smartphone through one or more cameras and / or an infrared tracking dot pattern projector, thus displaying on the television.

[0153] Some embodiments of the present disclosure improve diagnostic workflow using a system consisting of an AI software system or subsystem, a 5G smartphone wirelessly interfaced with a nearby large screen television, one or more Bluetooth or WiFi peripherals paired with the smartphone, such as a scroll wheel, joystick, foot switch or variable foot pedal, trackball, simple selector button, mouse, keyboard, capacitive proximity sensor, infrared or ultrasonic motion detector or proximity sensor, one or more separate or integrated motion sensing MEMs, accelerometers or strain gauges, stylus, mouse, haptic VR gloves, wand, laser pointer, VR / AR display goggles, one or more speakers, one or more LED or LCD displays, headset microphone and / or headphones, remote control handset, reflective or fluorescent ball or tape, or other sensory or motion capture control or feedback device commercially available for mobile or desktop computing, for example.

[0154] Some embodiments of the present disclosure improve diagnostic workflow using a system consisting of an AI software system or subsystem, a 5G smartphone wirelessly interfaced with a nearby large-screen television, and a detent scroll wheel. In one such embodiment, each detent is indexed to a specific feature or region of diagnostic relevance within the imaged sample, such as a tissue feature, a Cartesian coordinate within the imaged sample, a highlight or annotated portion of the image, an annotation or external message, a hyperlinked external document or portion of a document or media file, an active chat session, or a collaborative resource. Such a scroll wheel modality allows for uniquely accurate, rapid, and efficient navigation of multiple discrete regions or features or image portions.

[0155] Some embodiments of the present disclosure improve diagnostic workflow using a dynamically detented scroll wheel that alters the audiovisual and tactile behavior of each indexed feature or area as it is reviewed and annotated in a manner that indicates progress and allows review and / or revision of that progress. Such dynamic scroll wheels comprise a brushless DC motor integrated with one or more electronic circuit boards and a rotatable knob or wheel, and may also include LEDs, push buttons, membrane buttons, touch sensors, OLED or LCD displays, palm rests, speakers or audio transducers, Hall effect sensors or strain gauges, microphones or piezoelectric transducers or sensors, optical or magnetic commutation sensors, or other elements of peripherals commercially available for mobile or desktop computing.

[0156] Some embodiments of the present disclosure improve diagnostic workflows using a dynamically detented scroll wheel, which uses a brushless DC motor for simulated and dynamically configurable dynamic behaviors such as momentum, resistive inertia, and soft damping. One such embodiment may incorporate a strain gauge or other sensor at the base of the scroll wheel to detect manual forces applied axially to the top center of the knob for XY navigation purposes. Such a sensor may also detect tapping on the top of the scroll wheel for select and deselect functions.

[0157] Some embodiments of the present disclosure improve diagnostic workflow using a dynamic scroll wheel, where the device dynamically varies the apparent inertia and / or soft damping of the wheel or knob through actuated mechanical features such as brushless or brushed DC motors or stepper motors, or servo motors, solenoids or electromagnets, or friction elements in conjunction with electronically actuated ferrofluids. In one such embodiment, the scroll wheel may move under the control of a remote collaborator or to indicate its progress for collaborative diagnostic and / or training purposes. Such simulated inertia may allow for customization of the device's feel to better suit different users or to reduce hand and wrist fatigue. The embodiments may also provide user-configurable detent strength for similar reasons of fatigue and / or user preference.

[0158] Some embodiments of the present disclosure improve collaborative diagnostic workflows by using an AI system or subsystem to select and interconnect one or more collaborative and available resources from a real-time registry of currently active pathologists and / or virtual pathologists within a cloud-hosted network. In one such embodiment, the dynamic scroll wheel described above may be used to obtain simultaneous feedback of opinions or assessments from multiple pathologists and / or virtual pathologists, which are tabulated and / or aggregated by the system computationally or by neural networks according to one or more consensus algorithms or other preferred criteria and practices. In such a mode, the scroll wheel may function as a remote tactile and haptic handshake by and between collaborating participants. Such collaborative sessions may be recorded for subsequent review.

[0159] Some embodiments of the present disclosure use an AI system or subsystem to improve self-concordance, measuring and evaluating discrepancies between a pathologist's diagnostic preferences and their tendency to diagnose similar or identical specimen images. Such re-review of previously diagnosed slide images may be triggered by the system in the context of a training consultation. Sample similarity and concordance cases may be based on the system's own classification of diagnostic relevance attributes or may follow the standards and practices of a medical board or other governing regulatory agency. The system may facilitate self-concordance improvement or consistent evaluation of a pathologist's review through subtle suggestive scintillation of nearby image portions of diagnostic relevance. The system may continually refine its own criteria for diagnostic relevance to more closely approximate the judgment and workflow patterns of individual pathologists until nearly every slide review results in the execution of recommended annotations and confirmation of diagnostic conclusions. The value proposition is productivity improvement, not "replacement" of pathologists.

[0160] Some embodiments of the present disclosure improve self-concordance using an AI system that compares diagnostically relevant features of each slide during diagnosis with previous similar features, images, and cases diagnosed by the pathologist and / or other well-respected pathologists and / or committee standards and practices. One such embodiment may pre-populate annotations with suggested text, which the user can freely confirm, revise, or reject. Through continuous monitoring of actual diagnostic workflow and machine learning, such a system may eventually approximate human diagnostic intuition sufficiently to be indistinguishable in terms of sensitivity and specificity.

[0161] The following description includes several implementations of systems and methods for implementing the concepts described above with reference to Figures 1-16. These implementations are provided as non-limiting examples.

[0162] In some implementations, a method includes, at a server system, acquiring an image of a sample (e.g., including a composite image derived from a volume z-stack or composed of pixels, regions, features selected for diagnostic or therapeutic relevance) (e.g., not necessarily requiring a slide; a tissue ribbon can be scanned directly without being mounted on a slide) (e.g., including specific circumstances related to one or more pharmaceutical protocols and matching suitable participant candidates thereto), identifying one or more cellular morphologies of the sample, mapping a plurality of regions of the image corresponding to the one or more cellular morphologies, assigning a diagnostic or therapeutic relevance level to each of the plurality of regions, compressing the plurality of regions to a compression level (e.g., or compression type / method, inversely correlated in terms of fidelity) that is inversely correlated to the assigned diagnostic or therapeutic relevance level for the region, receiving a request to view the image from a first client device, and in response to receiving the request to view the image from the first client device, sending metadata to the first client device including (i) the compressed plurality of regions and (ii) an indexing of the assigned diagnostic or therapeutic relevance levels of the plurality of regions.

[0163] In some implementations, assigning the diagnostic or therapeutic relevance level includes sending the image to one or more diagnostic machine vision systems (or human pathologist review); receiving diagnostic or therapeutic relevance data associated with a plurality of regions from the one or more diagnostic machine vision systems in response to sending the image; and aggregating the diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems, wherein the assignment of the diagnostic or therapeutic relevance level is based on the aggregated diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems (there may be various ways to combine several such inputs for best sensitivity, specificity, and agreement).

[0164] In some implementations, the method further includes extracting the plurality of regions into a plurality of separate alpha layers or images, and compressing the plurality of regions includes compressing the plurality of separate alpha layers or images, associating a plurality of portions of metadata with each of the plurality of separate alpha layers or images, and encoding or encrypting the plurality of portions of metadata into each of the plurality of separate alpha layers or images.

[0165] In some implementations, identifying one or more cellular morphologies of the sample includes compiling a cellular index of image features using a predetermined library of tissue-specific or pathology-specific neural networks.

[0166] In some implementations, assigning a diagnostic or therapeutic relevance level to each region includes assigning multiple hierarchies of diagnostic or therapeutic relevance, and compressing the multiple regions includes using a compression level corresponding to each hierarchy of the multiple hierarchies of diagnostic or therapeutic relevance.

[0167] In some implementations, the method further includes prioritizing the plurality of regions into an ordered sequence of individual image regions or sample features based on the diagnostic or therapeutic relevance of each region of the plurality of regions, and the metadata includes instructions for displaying the plurality of regions in a certain order based on the sequence.

[0168] In some implementations, the ordered sequence of individual image regions is optimized based on one or more of review efficiency, review completeness, directionality from one side of the image to the other, linear review of cell morphology, and categorical review of cell morphology.

[0169] In some implementations, the method further includes rendering the ordered individual image regions on the display as a three-dimensional fly-through rendering of the image, wherein a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to an assigned diagnostic or therapeutic relevance level of each region of the image.

[0170] In some implementations, the metadata includes parameter-based characterizations of cells, organelles, groups or regions of cells, cell states, or tissue morphology of the sample.

[0171] In some implementations, the metadata includes, for each region, a specification of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

[0172] In some implementations, the metadata includes, for each domain, one or more instances from a library of dedicated GAN models for subsequent reconstruction of the domain (such a GAN library may be organized into hierarchical classes and various categories and degrees of specialization).

[0173] In some implementations, compressing the plurality of regions includes deconverting a plurality of regions of the plurality of regions having a diagnostic or therapeutic relevance below a threshold, and preserving the original resolution of a plurality of regions of the plurality of regions having a diagnostic or therapeutic relevance that meets the threshold.

[0174] In some implementations, deconverting regions having diagnostic or therapeutic relevance below a threshold includes deconverting to a partially pixel-shifted retro source image layer for pixel-shift super-resolution upon subsequent recombination at the first client device.

[0175] In some implementations, the method further includes, before receiving a request to view the image from the first client device, using one or more dedicated GANs to decompress the plurality of regions into a plurality of reconstructed regions, comparing the plurality of reconstructed regions with pre-compressed versions of the plurality of regions, and determining differences between the reconstructed regions and the pre-compressed versions of the plurality of regions based on the comparison.

[0176] In some implementations, the method further includes, before receiving a request to view the image from the first client device, determining that a difference between the reconstructed region and the pre-compressed versions of the plurality of regions meets a threshold; updating one or more dedicated GANs based on the determination that the difference between the reconstructed region and the pre-compressed versions of the plurality of regions meets the threshold; and re-compressing the plurality of regions using a dedicated GAN from the one or more dedicated GANs updated for each region, wherein transmitting the compressed plurality of regions includes transmitting the re-compressed plurality of regions.

[0177] In some implementations, the method includes determining, before receiving a request to view the image from the first client device, that a difference between the reconstructed region and a pre-compressed version of the plurality of regions does not meet a threshold, and transmitting the compressed plurality of regions follows a determination that a difference between the reconstructed region and the pre-compressed version of the plurality of regions does not meet a threshold.

[0178] In some implementations, the method further includes storing, at the server system, the compressed regions and the metadata, and deleting the image before receiving a request to view the image from the first client device.

[0179] In some implementations, the method further includes packaging, at the server system, the compressed plurality of regions and the metadata into a file wrapper, and sending the compressed plurality of regions and the metadata to the first client device includes sending the file wrapper to the first client device.

[0180] In some implementations, the method further includes receiving, at the first client device, the compressed regions and metadata from the server system; decompressing the compressed regions and metadata; combining the decompressed regions with a reconstructed version of the image or requested portions thereof; adding characteristic data included in the metadata corresponding to features of the sample to corresponding regions of the reconstructed version of the image; and displaying the portions of the reconstructed version of the image in an order on a display integrated with or communicatively coupled to the first client device based on an assigned diagnostic or therapeutic relevance level specified by the metadata.

[0181] In some implementations, the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is lower than the first degree of diagnostic or therapeutic relevance, and compressing the plurality of regions includes compressing the first region using a first compression ratio of M:1 and compressing the second region using a second compression ratio of N:1, where N>M≧1.

[0182] In some implementations, the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is lower than the first degree of diagnostic or therapeutic relevance, and compressing the plurality of regions includes compressing the first region using a lossless compression algorithm and compressing the second region using a lossy compression algorithm.

[0183] In some implementations, the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance lower than the first degree of diagnostic or therapeutic relevance, and compressing the plurality of regions includes reducing the resolution of the first region to an Mth degree and reducing the resolution of the second region to an Nth degree, where N>M≧0.

[0184] In another aspect, a method for reconstructing and presenting images for compression and transmission, as well as diagnostic annotation, includes, at a server system including one or more processors, acquiring an image of a sample (e.g., including a composite image derived from a volume z-stack and composed of pixels, regions, or features selected for diagnostic or therapeutic relevance) (e.g., not necessarily requiring a slide; a tissue ribbon can be scanned directly without being mounted on a slide) (e.g., including specific conditions associated with one or more pharmaceutical protocols and matching suitable participant candidates thereto), identifying one or more cellular morphologies of the sample, mapping a plurality of regions of the image corresponding to the one or more cellular morphologies, assigning a respective diagnostic or therapeutic relevance level to the plurality of regions, reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance level to generate a plurality of processed regions, receiving a request to view the image from a first client device, and in response to receiving the request to view the image from the first client device, transmitting metadata to the first client device including an index of (i) the plurality of processed regions and (ii) the assigned diagnostic or therapeutic relevance levels of the plurality of processed regions.

[0185] In some implementations, reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance level includes reducing the resolution of at least one region of the plurality of regions, which includes inverse pixel shifting the at least one region.

[0186] In some implementations, inverse pixel shifting at least one region includes segmenting adjacent pixels of the image into a plurality of pixel groups; combining adjacent pixels of each pixel group of the plurality of pixel groups into a pixel group value (e.g., combining includes averaging or other mathematical functions or algorithms, including neural networks for predicting and mitigating DeBayer artifacts or sensor noise); segmenting adjacent pixels of the image into a plurality of shifted pixel groups; averaging adjacent pixels of each shifted pixel group of the plurality of shifted pixel groups into a shifted pixel group value; and replacing the adjacent pixels of the image with a plurality of layers, including (i) a first layer including pixel group values ​​for each pixel group and (ii) a second layer including shifted pixel group values ​​for each shifted pixel group.

[0187] In some implementations, assigning each diagnostic or therapeutic relevance level to the plurality of regions includes assigning a first diagnostic or therapeutic relevance to a first region of the plurality of regions, assigning a second diagnostic or therapeutic relevance lower than the first degree to a second region of the plurality of regions, and reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance levels, including reducing the resolution of the first region to an Mth degree and reducing the resolution of the second region to an Nth degree, where N>M≧0.

[0188] In some implementations, assigning each diagnostic or therapeutic relevance level to the plurality of regions includes assigning a first diagnostic or therapeutic relevance to a first region of the plurality of regions, assigning a second diagnostic or therapeutic relevance lower than the first level to a second region of the plurality of regions, and reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance levels, including maintaining the original resolution of the first region based on a determination that the diagnostic or therapeutic relevance level of the first region meets a threshold, and reducing the resolution of the second region based on a determination that the diagnostic or therapeutic relevance level of the second region does not meet the threshold.

[0189] In some implementations, reducing or maintaining the resolution of each of the plurality of regions includes deconverting regions of the plurality of regions that have diagnostic or therapeutic relevance below a threshold, and retaining the original resolution of regions of the plurality of regions that have diagnostic or therapeutic relevance that meet the threshold.

[0190] In some implementations, deconverting regions having diagnostic or therapeutic relevance below a threshold includes deconverting to a partially pixel-shifted retro source image layer for super-resolution upon subsequent recombination at the first client device.

[0191] In some implementations, assigning the diagnostic or therapeutic relevance level includes sending the image to one or more diagnostic machine vision systems (or for human pathologist review); receiving diagnostic or therapeutic relevance data associated with the plurality of regions from the one or more diagnostic machine vision systems in response to sending the image; and aggregating the diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems, wherein assigning the diagnostic or therapeutic relevance level is based on the aggregated diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems.

[0192] In some implementations, the method further includes extracting the plurality of regions into a plurality of discrete alpha layers or images, and reducing or maintaining the resolution of each of the plurality of regions includes reducing or maintaining the resolution of each of the plurality of discrete alpha layers or images, associating a plurality of portions of metadata with each of the plurality of discrete alpha layers or images, and encoding or encrypting the plurality of portions of metadata into the plurality of discrete alpha layers or images, respectively.

[0193] In some implementations, identifying one or more cellular morphologies of the sample includes compiling a cellular index of image features using a predetermined library of tissue-specific or pathology-specific neural networks.

[0194] In some implementations, assigning a diagnostic or therapeutic relevance level to each region includes assigning multiple layers of diagnostic or therapeutic relevance, and reducing or maintaining the resolution of each of the multiple regions includes reducing or maintaining the resolution of each using a degree of deconversion corresponding to each tier of the multiple tiers of diagnostic or therapeutic relevance.

[0195] In some implementations, the method further includes prioritizing the plurality of regions into an ordered sequence of individual image regions or sample features based on the diagnostic or therapeutic relevance of each region of the plurality of regions, and the metadata includes instructions for displaying the plurality of regions in a certain order based on the sequence.

[0196] In some implementations, the method further includes rendering the ordered individual image regions on the display as a three-dimensional fly-through rendering of the image, wherein a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to an assigned diagnostic or therapeutic relevance level of each region of the image.

[0197] In some implementations, the metadata includes parameter-based characterizations of cells, organelles, groups or regions of cells, cell states, or tissue morphology of the sample.

[0198] In some implementations, the metadata includes, for each region, a specification of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

[0199] In some implementations, the metadata includes, for each domain, one or more instances from a library of dedicated GAN models for subsequent reconstruction of the domain (such a GAN library may be organized into hierarchical classes and various categories and degrees of specialization).

[0200] In some implementations, the method further includes, before receiving a request to view the image from the first client device, upconverting the plurality of regions into a plurality of reconstructed regions using one or more dedicated GANs, comparing the plurality of reconstructed regions with original versions of the plurality of regions, and determining differences between the reconstructed regions and the original versions of the plurality of regions based on the comparison.

[0201] In some implementations, the method further includes, before receiving a request to view the image from the first client device, determining that a difference between the reconstructed region and original versions of the plurality of regions meets a threshold; updating one or more dedicated GANs based on the determination that the difference between the reconstructed region and the original versions of the plurality of regions meets the threshold; and, for each region, using a dedicated GAN from the updated one or more dedicated GANs to reduce or maintain the resolution of each of the plurality of regions, wherein transmitting the plurality of processed regions includes transmitting the plurality of regions with each of the reduced or maintained resolution.

[0202] In some implementations, the method includes determining, before receiving a request to view the image from the first client device, that a difference between the reconstructed region and an original version of the plurality of regions does not meet a threshold, and transmitting the plurality of processed regions follows a determination that the difference between the reconstructed region and the original version of the plurality of regions does not meet a threshold.

[0203] In some implementations, the method further includes storing, at the server system, the plurality of processed regions and metadata, and deleting the image before receiving a request to view the image from the first client device.

[0204] In some implementations, the method further includes packaging, at the server system, the plurality of processed regions and the metadata into a file wrapper, and sending the plurality of processed regions and the metadata to the first client device includes sending the file wrapper to the first client device.

[0205] In some implementations, the method further includes receiving, at the first client device, a plurality of processed regions and metadata from the server system; upconverting at least a subset of the plurality of processed regions and metadata; combining the upconverted regions with a reconstructed version of the image; adding characteristic data contained in the metadata corresponding to features of the sample to corresponding regions of the reconstructed version of the image; and displaying portions of the reconstructed version of the image in an order on a display integrated with or communicatively coupled to the first client device based on an assigned diagnostic or therapeutic relevance level specified by the metadata.

[0206] In some implementations, the method further includes compressing the plurality of regions using, for each region, a compression level that is inversely correlated to an assigned diagnostic or therapeutic relevance level for the region.

[0207] In some implementations, the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is lower than the first degree of diagnostic or therapeutic relevance, and compressing the plurality of regions includes compressing the first region using a first compression ratio of M:1 and compressing the second region using a second compression ratio of N:1, where N>M≧1.

[0208] In some implementations, the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is lower than the first degree of diagnostic or therapeutic relevance, and compressing the plurality of regions includes compressing the first region using a lossless compression algorithm and compressing the second region using a lossy compression algorithm.

[0209] In another aspect, a method for reconstructing and presenting images for compression and transmission, as well as diagnostic annotation, includes, in a server system including one or more processors, acquiring an image of a sample (e.g., including a composite image derived from a volume z-stack and comprised of pixels, regions, or features selected for diagnostic or therapeutic relevance) (e.g., not necessarily requiring a slide; tissue ribbons may be scanned directly without slide mounting) (e.g., including specific circumstances related to one or more pharmaceutical protocols and matching suitable candidate participants thereto), identifying one or more cellular morphologies of the sample, mapping a plurality of regions of the image corresponding to the one or more cellular morphologies, and converting at least a subset of the plurality of regions into a plurality of compressed or deconvolved images. the plurality of compressed or deconverted image segments; determining a respective generative adversarial network (GAN) model corresponding to each cellular morphology associated with each compressed or deconverted image segment of the plurality of compressed or deconverted image segments; assigning each GAN model to each compressed or deconverted image segment; receiving a request to view the image from the first client device; and in response to receiving the request to view the image from the first client device, transmitting (i) the plurality of compressed or deconverted image segments and (ii) each GAN model assigned to the plurality of compressed or deconverted image segments to the first client device.

[0210] In some implementations, the method further includes constructing, at the server system, a map of each GAN model assigned to the plurality of compressed or deconverted image segments, wherein a segment of the map of each GAN model is linked to a corresponding image segment of the plurality of compressed or deconverted image segments, and transmitting each GAN model includes transmitting the map of each GAN model.

[0211] In some implementations, the method further includes, at the server system, continuing to compress using a lossless compression algorithm or maintain the original resolution of at least one region of the plurality of regions and determining and assigning a respective GAN model for the at least one region of the plurality of regions, and in response to receiving a request to view the image from the first client device, (iii) sending the at least one region compressed with the lossless compression algorithm or having the original resolution maintained to the first client device algorithm.

[0212] In some implementations, the method further includes, at the server system, assigning each diagnostic or therapeutic relevance level to the plurality of regions, determining that at least one region of the plurality of regions meets a threshold for diagnostic or therapeutic relevance, and determining that a subset of the plurality of regions does not meet the threshold for diagnostic or therapeutic relevance, wherein compressing using a lossless compression algorithm or maintaining the original resolution of at least one region of the plurality of regions is in accordance with the determination that at least one region of the plurality of regions meets the threshold for diagnostic or therapeutic relevance, compressing or deconverting the subset of the plurality of regions, and assigning a respective GAN model to each compressed or deconverted image segment is in accordance with the determination that the subset of the plurality of regions does not meet the threshold for diagnostic or therapeutic relevance.

[0213] In some implementations, identifying one or more cellular morphologies of the sample includes compiling a cellular index of image features using a predetermined library of tissue-specific or pathology-specific neural networks.

[0214] In some implementations, the compressing or deconverting includes deconverting a subset of the multiple regions into a partially pixel-shifted retro source image layer for pixel-shift super-resolution upon subsequent recombination at the first client device.

[0215] In some implementations, the method further includes, before receiving a request to view the image from the first client device, using each GAN model to decompress or super-resolve the subset of regions into a plurality of reconstructed regions, comparing the plurality of reconstructed regions with pre-compressed or pre-deconverted versions of the subset of regions, and determining, based on the comparison, differences between the reconstructed regions and the pre-compressed or pre-deconverted versions of the subset of regions.

[0216] In some implementations, the method further includes, before receiving a request to view the image from the first client device, determining that a difference between the reconstructed region and a pre-compressed or pre-deconverted version of the subset of the region satisfies a threshold; updating each GAN model based on the determination that the difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region satisfies the threshold; and recompressing or re-deconverting the subset of the plurality of regions using each updated GAN model, wherein transmitting the plurality of compressed or deconverted image segments includes transmitting the recompressed or re-deconverted subset of the plurality of regions.

[0217] In some implementations, the method further includes determining, before receiving a request to view the image from the first client device, that a difference between the reconstructed region and a pre-compressed or pre-deconverted version of the subset of the region does not meet a threshold, and transmitting the plurality of compressed or deconverted image segments follows the determination that the difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region does not meet the threshold.

[0218] In some implementations, the method further includes storing, at the server system, the plurality of compressed or deconverted image segments and respective GAN models assigned to the plurality of compressed or deconverted image segments, and deleting the image before receiving a request to view the image from the first client device.

[0219] In some implementations, the method further includes packaging, at the server system, the plurality of compressed or deconverted image segments and each GAN model assigned to the plurality of compressed or deconverted image segments into a file wrapper, and transmitting the plurality of compressed or deconverted image segments and each GAN model assigned to the plurality of compressed or deconverted image segments to the first client device includes transmitting the file wrapper to the first client device.

[0220] In some implementations, the method further includes receiving, at the first client device, from the server system, a plurality of compressed or deconverted image segments and respective GAN models assigned to the plurality of compressed or deconverted image segments; decompressing or super-resolving the compressed or deconverted image segments using each GAN model assigned to the plurality of compressed or deconverted image segments; combining the decompressed or super-resolved image segments with a reconstructed version of the image or requested portions thereof; and displaying the plurality of portions of the reconstructed version of the image on a display integrated with or communicatively coupled to the first client device.

[0221] In another aspect, a method for processing and transmitting images for diagnostic analysis includes, at a server system including one or more processors, obtaining an input image of a sample; globally downconverting the input image to a downconverted image; globally downconverting the input image to the downconverted image and then globally upconverting the downconverted image to an upconverted image using a generative adversarial network (GAN) model configured to reconstruct an image including features corresponding to the sample; classifying a plurality of regions of the downconverted image based on cellular morphology and / or diagnostic relevance; and communicating the upconverted image over a communication network for delivery to a client device.

[0222] In some implementations, the method further includes dividing the input image into a plurality of tiles, and downconverting the input image as a whole includes downconverting each of the plurality of tiles, and upconverting the downconverted image as a whole includes upconverting each of the plurality of tiles.

[0223] In some implementations, globally upconverting the downconverted image includes using a GAN model to predictively improve clarity of the downconverted image. In some implementations, globally upconverting the downconverted image includes restoring deleted pixels by predicting pixel values ​​corresponding to the deleted pixels using the GAN model. In some implementations, globally upconverting the downconverted image includes overwriting deconverted pixel values ​​with pixel values ​​predicted by the GAN model.

[0224] In some implementations, the method further includes compressing the upconverted image using a run-length encoding scheme before transmitting the upconverted image to a communication network.

[0225] In some implementations, the method further includes manipulating a portion of the input image for subsequent processing based on the classification of the plurality of regions, hi some implementations, the subsequent processing includes globally downconverting the input image having the manipulated portion and simultaneously globally upconverting and classifying the plurality of regions of the globally downconverted image.

[0226] In some implementations, the system includes one or more processors of a server or client device and memory storing instructions that, when executed by the one or more processors, cause the server or client device to perform any of the methods described above.

[0227] In some implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by a server or a client device, cause the server or client device to perform any of the methods described above.

[0228] In another aspect, a method for processing and transmitting images for diagnostic analysis includes, in a server system including one or more processors, acquiring an input image of a sample, the input image including image data representing a flattened z-stack; classifying spectral differences of a plurality of features of the input image; and assigning z-levels of the z-stack to each of the plurality of features based on the classification, wherein one or more first z-levels are assigned to a first subset of the plurality of features (e.g., blood cells at lower z-levels) and one or more second z-levels are assigned to a second subset of the plurality of features (e.g., blood cells at higher z-levels). assigning one or more first z-levels below one or more second z-levels, thereby obscuring portions of a first subset of features (e.g., at least a portion of lower blood cells are obscured by at least a portion of upper blood cells); predicting pixel values ​​associated with the obscured portions of the first subset of features using a generative adversarial network (GAN) model configured to reconstruct image features; generating three-dimensional (3D) image data including the predicted pixel values ​​and including image data from the one or more first z-levels and the one or more second z-levels, thereby representing a virtually reconstructed 3D z-stack; and providing the generated 3D image data for display on a client device.

[0229] In some implementations, generating the 3D image data includes selecting a plurality of pixel values ​​across a plurality of z-levels and including at least a portion of the predicted pixel values ​​that meet a predetermined sharpness threshold, and replacing pixel values ​​corresponding to the obscured pixels with the selected pixel values.

[0230] In some implementations, generating the 3D image data includes selecting a plurality of pixel values ​​across a plurality of z-levels, the plurality of pixel values ​​including at least a portion of predicted pixel values ​​that meet a predetermined threshold of diagnostic or therapeutic relevance, and replacing pixel values ​​corresponding to obscured pixels with the selected pixel values.

[0231] In some implementations, classifying the spectral differences includes classifying the feature boundaries based on which spectral portion is most dominant.

[0232] In some implementations, providing the generated 3D image data to the display includes approximating navigation through a z-field that includes the z-stack by mapping multiple z-levels of the z-stack to respective control levels associated with a control user input element at the client device. In some implementations, the control user input element is a slider, a knob, a zoom control, or a z-field navigation control. In some implementations, the approximate navigation through the z-stack is triggered after a zoom threshold is met. In some implementations, generating the 3D image data includes generating a virtual slide or a non-planar virtual surface at an angle that bisects the multiple z-levels.

[0233] In some implementations, the system includes one or more processors of a server or client device and memory storing instructions that, when executed by the one or more processors, cause the server or client device to perform any of the methods described above.

[0234] In some implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by a server or a client device, cause the server or client device to perform any of the methods described above.

[0235] Those skilled in the art will appreciate that changes could be made to the exemplary embodiments described above without departing from the broad inventive concept thereof. It is therefore understood that the invention is not limited to the exemplary embodiments shown and described, but is intended to cover modifications within the spirit and scope of the invention as defined by the appended claims.

[0236] For example, certain features of the exemplary embodiments may or may not be part of the claimed invention, different components as opposed to those specifically mentioned may implement at least some of the features described herein, and features of the disclosed embodiments may be combined.

[0237] As used herein, the terms "about" and "approximately" can refer to + or -10% of the referenced value. For example, "about 9" is understood to include 8.2 and 9.9.

[0238] It will be appreciated that at least some of the figures and descriptions of the invention have been simplified to focus on elements relevant to a clear understanding of the invention, but for clarity, other elements that one skilled in the art would understand may also comprise part of the invention, however, because such elements are well known in the art and because they do not necessarily facilitate a better understanding of the invention, descriptions of such elements are not provided herein.

[0239] It should be understood that although terms such as "first" and "second" may be used herein to describe various elements, these elements should not be limited by these terms; these terms are used only to distinguish one element from another.

[0240] For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without changing the meaning of the description, so long as all occurrences of "first element" are consistently re-named and all occurrences of the second element are consistently re-named. A first element and a second element are both elements, but are not the same element.

[0241] As used herein, the term "if" may be interpreted, optionally depending on the context, to mean "when," or "in response to determining," or "in response to detecting," or "according to determining." Similarly, the phrase "when determined," or "when [described condition or event] is detected," is interpreted, optionally depending on the context, to mean "when determined," or "in response to determining," or "when [described condition or event] is detected," "in response to detecting [described condition or event]," or "in response to determining [described condition or event]."

[0242] The terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the scope of the claims. For example, the image processing concepts described above can be used for non-medical images in addition to, or as an alternative to, the medical imaging example described above. Any image data, regardless of its content (medical or non-medical), can be processed by the image processing platform described herein using the same functions and modules.

[0243] As used in the description of implementations and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0244] As used herein, the term "and / or" will be understood to refer to and encompass any and all possible combinations of one or more of the associated listed items.

[0245] It will be further understood that as used herein, the terms "comprises" and / or "comprising" specify the presence of stated features, integers, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, operations, elements, components, and / or groups thereof.

[0246] As used herein, the term "if" may be interpreted to mean "when" or "upon," or "in response to determining," or "in response to detecting," the preceding described condition is true, depending on the context.

[0247] Similarly, the phrases "when it is determined (preceded by a stated condition)," or "if (preceded by a stated condition)," or "when (preceded by a stated condition)," or "when," or "when," may be interpreted to mean that the preceding stated condition is true "upon determining," or "in response to determining," or "upon detecting," or "in response to detecting," depending on the context.

[0248] Furthermore, to the extent that the method does not rely on the particular order of steps set forth herein, the particular order of steps should not be construed as a limitation on the claims. Claims directed to the methods of the present invention should not be limited to performing those steps in the order described, and one of ordinary skill in the art can readily appreciate that the steps may be varied and still remain within the spirit and scope of the invention.

Claims

1. 1. A method for compressing, transmitting, reconstructing, and presenting images for diagnostic annotation, comprising: The method comprises:

1. A server system including one or more processors, acquiring an image of the sample; identifying one or more cellular morphologies of said sample; mapping a plurality of regions of the image corresponding to the one or more cell morphologies; assigning a diagnostic or therapeutic relevance level to each region of the plurality of regions; compressing the plurality of regions using, for each region, a compression level that is inversely related to the assigned diagnostic or therapeutic relevance level for that region; receiving a request to view the image from a first client device; in response to receiving the request to view the image from the first client device, transmitting to the first client device (i) the compressed plurality of regions and (ii) metadata including an index of the assigned diagnostic or therapeutic relevance level of the plurality of regions; A method comprising:

2. assigning said diagnostic or therapeutic relevance level, transmitting the images to one or more diagnostic machine vision systems; receiving, in response to transmitting the image, from the one or more diagnostic machine vision systems, diagnostic or treatment-related data associated with the plurality of regions; aggregating the diagnostic or treatment-related data received from the one or more diagnostic machine vision systems; Including, wherein the step of assigning the diagnostic or treatment relevance level is based on the aggregated diagnostic or treatment relevance data received from the one or more diagnostic machine vision systems. The method of claim 1.

3. extracting the regions into separate alpha layers or images; extracting, wherein compressing the plurality of regions comprises compressing the plurality of individual alpha layers or images; associating a plurality of portions of the metadata with each of the plurality of individual alpha layers or images; encoding or encrypting said portions of said metadata into said plurality of separate alpha layers or images, respectively; 10. The method of any one of the preceding claims, further comprising:

4. 10. The method of claim 1, wherein identifying the one or more cellular morphologies of the sample comprises compiling a cellular index of features of the image using a predetermined library of tissue-specific or pathology-specific neural networks.

5. assigning a level of diagnostic or therapeutic relevance to each region includes assigning multiple tiers of diagnostic or therapeutic relevance; the step of compressing the plurality of regions includes using a compression level corresponding to each tier of the plurality of tiers of diagnostic or therapeutic relevance.

10. A method according to any one of the preceding claims.

6. prioritizing the plurality of regions into an ordered sequence of individual image regions or sample features based on the diagnostic or therapeutic relevance of each region of the plurality of regions; the metadata includes instructions for displaying the plurality of regions in an order based on the sequence; 10. A method according to any one of the preceding claims.

7. the ordered sequence of distinct image regions comprising: Review efficiency, Review completeness, the orientation of the image from one side to the other; Linear review of cell morphology, and Cell Morphology Category Review The method of claim 6 , wherein the optimization is based on one or more of:

8. rendering the ordered individual image regions on the display as a three-dimensional fly-through rendering of the image; a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to the assigned diagnostic or therapeutic relevance level of each region of the image. The method of claim 6.

9. 10. The method of claim 1, wherein the metadata comprises parameter-based characterizations of cells, organelles, groups or regions of cells, cell states, or tissue morphology of the sample.

10. 10. The method of claim 1, wherein the metadata includes, for each region, a specification of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

11. The method of claim 10 , wherein the metadata includes, for each region, one or more instances from a library of dedicated GAN models for subsequent reconstruction of the region.

12. compressing the plurality of regions de-resolving regions of the plurality of regions having diagnostic or therapeutic relevance below a threshold; preserving the original resolution of a plurality of regions of said plurality of regions having diagnostic or therapeutic relevance that meets said threshold; 10. The method of any one of the preceding claims, comprising:

13. 13. The method of claim 12, wherein deconverting the regions having diagnostic or therapeutic relevance below the threshold comprises deconverting them into a partially pixel-shifted retro source image layer for pixel-shift super-resolution upon subsequent recombination at the first client device.

14. before receiving the request to view the image from the first client device; decompressing the plurality of regions into a plurality of reconstruction regions using one or more dedicated GANs; comparing the reconstructed regions with pre-compressed versions of the regions; determining differences between the reconstructed region and the pre-compressed versions of the plurality of regions based on the comparing; 10. The method of any one of the preceding claims, further comprising:

15. before receiving the request to view the image from the first client device; determining that the difference between the reconstructed region and the pre-compressed versions of the plurality of regions meets a threshold; updating the one or more dedicated GANs based on the determination that the difference between the reconstructed region and the pre-compressed versions of the plurality of regions meets the threshold; recompressing the plurality of regions using, for each region, a dedicated GAN of the updated one or more dedicated GANs; further comprising transmitting the compressed plurality of regions includes transmitting the recompressed plurality of regions.

15. The method of claim 14.

16. before receiving the request to view the image from the first client device; determining that the difference between the reconstructed region and the pre-compressed versions of the plurality of regions does not meet the threshold; further comprising transmitting the compressed plurality of regions is pursuant to the determination that the difference between the reconstructed region and the pre-compressed version of the plurality of regions does not meet the threshold.

15. The method of claim 14.

17. In the server system, storing the compressed regions and the metadata; deleting the image before receiving the request to view the image from the first client device; 10. The method of any one of the preceding claims, further comprising:

18. In the server system, packaging the compressed regions and the metadata into a file wrapper; further comprising transmitting the compressed regions and the metadata to the first client device includes transmitting the file wrapper to the first client device.

10. A method according to any one of the preceding claims.

19. On the first client device, receiving the compressed regions and the metadata from the server system; decompressing the compressed regions and the metadata; combining the decompressed region with a reconstructed version of the image or requested portion thereof; adding characteristic data included in the metadata corresponding to features of the specimen to corresponding regions of the reconstructed version of the image; displaying portions of the reconstructed version of the image in an order based on the assigned diagnostic or therapeutic relevance level specified by the metadata on a display integrated with or communicatively coupled to the first client device; 10. The method of any one of the preceding claims, further comprising:

20. the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is less than the first degree of diagnostic or therapeutic relevance; compressing the plurality of regions compressing the first region using a first compression ratio of M:1; compressing the second region using a second compression ratio of N:1; and N>M≧1; 10. A method according to any one of the preceding claims.

21. the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is less than the first degree of diagnostic or therapeutic relevance; compressing the plurality of regions compressing the first region using a lossless compression algorithm; compressing the second region using a lossy compression algorithm; 10. The method of any one of the preceding claims, comprising:

22. the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is less than the first degree of diagnostic or therapeutic relevance; compressing the plurality of regions reducing the resolution of the first region to an Mth degree; reducing the resolution of the second region to an Nth degree; 5. The method of claim 1, wherein N>M≧0.

23. one or more processors on the server or client device; instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods of claims 1 to 22. and memory to store A system comprising:

24. Instructions which, when executed by a server or a client device, cause said server or said client device to perform any of the methods of claims 1 to 22. A non-transitory computer-readable storage medium for storing the

25. 1. A method for compressing and transmitting, reconstructing and presenting images for diagnostic annotation, comprising: The method comprises:

1. A server system including one or more processors, acquiring an image of the sample; identifying one or more cellular morphologies of said sample; mapping a plurality of regions of the image corresponding to the one or more cell morphologies; assigning respective diagnostic or therapeutic relevance levels to said plurality of regions; reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance level to generate a plurality of processed regions; receiving a request to view the image from a first client device; In response to receiving the request to view the image from the first client device, transmitting metadata to the first client device including (i) the plurality of processed regions and (ii) an index of the assigned diagnostic or therapeutic relevance levels of the plurality of processed regions; A method comprising:

26. 26. The method of claim 25, wherein reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance level comprises reducing the resolution of at least one region of the plurality of regions, which comprises inverse pixel shifting the at least one region.

27. inverse pixel shifting the at least one region; segmenting adjacent pixels of the image into a plurality of pixel groups; combining adjacent pixels of each pixel group of the plurality of pixel groups with a pixel group value; segmenting adjacent pixels of the image into a plurality of shifted pixel groups; averaging adjacent pixels of each shifted pixel group of the plurality of shifted pixel groups into a shifted pixel group value; replacing the adjacent pixels of the image with a plurality of layers, the replacement including: (i) a first layer including pixel group values ​​for each pixel group; and (ii) a second layer including shifted pixel group values ​​for each shifted pixel group.

27. The method of claim 26, comprising:

28. assigning respective diagnostic or therapeutic relevance levels to the plurality of regions, assigning a first degree of diagnostic or therapeutic relevance to a first region of the plurality of regions; assigning a second degree of diagnostic or therapeutic relevance to a second region of the plurality of regions that is lower than the first degree; and Including, reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance level; reducing the resolution of the first region to an Mth degree; reducing the resolution of the second region to an Nth degree; and N>M≧0; 10. A method according to any one of the preceding claims.

29. assigning respective diagnostic or therapeutic relevance levels to the plurality of regions, assigning a first degree of diagnostic or therapeutic relevance to a first region of the plurality of regions; assigning a second degree of diagnostic or therapeutic relevance to a second region of the plurality of regions that is lower than the first degree; and Including, reducing or maintaining the resolution of each of the plurality of regions based on the assigned diagnostic or therapeutic relevance level; maintaining the original resolution of the first region based on a determination that the diagnostic or therapeutic relevance level of the first region meets a threshold; and reducing the resolution of the second region based on a determination that the diagnostic or therapeutic relevance level of the second region does not meet the threshold; and 10. The method of any one of the preceding claims, comprising:

30. Reducing or maintaining the resolution of each of the plurality of regions, deconverting regions of said plurality of regions having diagnostic or therapeutic relevance below a threshold; preserving the original resolution of a plurality of regions of said plurality of regions having diagnostic or therapeutic relevance that meets said threshold; 10. The method of any one of the preceding claims, comprising:

31. 31. The method of claim 30, wherein deconverting the regions having diagnostic or therapeutic relevance below the threshold comprises deconverting to a partially pixel-shifted retro source image layer for super-resolution upon subsequent recombination at the first client device.

32. assigning said diagnostic or therapeutic relevance level, transmitting the images to one or more diagnostic machine vision systems; receiving, in response to transmitting the image, from the one or more diagnostic machine vision systems, diagnostic or treatment-related data associated with the plurality of regions; aggregating the diagnostic or treatment-related data received from the one or more diagnostic machine vision systems; Including, wherein the step of assigning the diagnostic or treatment relevance level is based on the aggregated diagnostic or treatment relevance data received from the one or more diagnostic machine vision systems.

10. A method according to any one of the preceding claims.

33. extracting the regions into separate alpha layers or images; extracting, wherein reducing or maintaining the resolution of each of the plurality of regions comprises reducing or maintaining the resolution of each of the plurality of individual alpha layers or images; associating a plurality of portions of the metadata with each of the plurality of individual alpha layers or images; encoding or encrypting said portions of said metadata into said plurality of separate alpha layers or images, respectively; 10. The method of any one of the preceding claims, further comprising:

34. 10. The method of claim 1, wherein identifying the one or more cellular morphologies of the sample comprises compiling a cellular index of features of the image using a predetermined library of tissue-specific or pathology-specific neural networks.

35. assigning a level of diagnostic or therapeutic relevance to each region includes assigning multiple tiers of diagnostic or therapeutic relevance; The step of reducing or maintaining the resolution of each of the plurality of regions includes reducing or maintaining the resolution of each of the plurality of diagnostic or therapeutically relevant hierarchies using a degree of deconversion corresponding to each of the plurality of hierarchies.

10. A method according to any one of the preceding claims.

36. prioritizing the plurality of regions into an ordered sequence of individual image regions or sample features based on the diagnostic or therapeutic relevance of each region of the plurality of regions; the metadata includes instructions for displaying the plurality of regions in an order based on the sequence; 10. A method according to any one of the preceding claims.

37. rendering the ordered individual image regions on the display as a three-dimensional fly-through rendering of the image; a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to the assigned diagnostic or therapeutic relevance level of each region of the image.

37. The method of claim 36.

38. 10. The method of claim 1, wherein the metadata comprises parameter-based characterizations of cells, organelles, groups or regions of cells, cell states, or tissue morphology of the sample.

39. 10. The method of claim 1, wherein the metadata includes, for each region, a specification of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

40. 40. The method of claim 39, wherein the metadata includes, for each region, one or more instances from a library of dedicated GAN models for subsequent reconstruction of the region.

41. before receiving the request to view the image from the first client device; up-resolving the multiple domains into multiple reconstruction domains using one or more dedicated GANs; comparing the reconstructed regions with original versions of the regions; determining differences between the reconstructed region and the original versions of the plurality of regions based on the comparing; 10. The method of any one of the preceding claims, further comprising:

42. before receiving the request to view the image from the first client device; determining that the difference between the reconstructed region and the original versions of the plurality of regions meets a threshold; updating the one or more dedicated GANs based on the determination that the difference between the reconstructed region and the original versions of the plurality of regions meets the threshold; for each region, using a dedicated GAN of the updated one or more dedicated GANs to reduce or maintain the resolution of each of the plurality of regions; further comprising the step of transmitting the plurality of processed regions includes transmitting the plurality of regions with the respective resolutions reduced or maintained.

42. The method of claim 41.

43. before receiving the request to view the image from the first client device; determining that the difference between the reconstructed region and the original versions of the plurality of regions does not meet the threshold; further comprising transmitting the plurality of processed regions is pursuant to the determination that the difference between the reconstructed region and the original version of the plurality of regions does not satisfy the threshold.

42. The method of claim 41.

44. In the server system, storing the plurality of processed regions and the metadata; deleting the image before receiving the request to view the image from the first client device; 10. The method of any one of the preceding claims, further comprising:

45. In the server system, packaging the plurality of processed regions and the metadata into a file wrapper; further comprising transmitting the plurality of processed regions and the metadata to the first client device includes transmitting the file wrapper to the first client device.

10. A method according to any one of the preceding claims.

46. On the first client device, receiving the plurality of processed regions and the metadata from the server system; upconverting the plurality of processed regions and at least a subset of the metadata; combining the upconverted region with a reconstructed version of the image; adding characteristic data included in the metadata corresponding to features of the specimen to corresponding regions of the reconstructed version of the image; displaying portions of the reconstructed version of the image in an order based on the assigned diagnostic or therapeutic relevance level specified by the metadata on a display integrated with or communicatively coupled to the first client device; 10. The method of any one of the preceding claims, comprising:

47. compressing the plurality of regions using, for each region, a compression level that is inversely related to the assigned diagnostic or therapeutic relevance level for that region.

10. The method of any one of the preceding claims, further comprising:

48. the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is less than the first degree of diagnostic or therapeutic relevance; compressing the plurality of regions compressing the first region using a first compression ratio of M:1; compressing the second region using a second compression ratio of N:1; and N>M≧1; 48. The method of claim 47.

49. the plurality of regions includes a first region having a first degree of diagnostic or therapeutic relevance and a second region having a second degree of diagnostic or therapeutic relevance that is less than the first degree of diagnostic or therapeutic relevance; compressing the plurality of regions compressing the first region using a lossless compression algorithm; compressing the second region using a lossy compression algorithm; 48. The method of claim 47, comprising:

50. one or more processors on the server or client device; instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods of claims 25 to 49. and memory to store A system comprising:

51. Instructions which, when executed by a server or a client device, cause said server or said client device to perform any of the methods of claims 25 to 49. A non-transitory computer-readable storage medium for storing the

52. 1. A method for compressing and transmitting, reconstructing and presenting images for diagnostic annotation, comprising: The method comprises:

1. A server system including one or more processors, acquiring an image of the sample; identifying one or more cellular morphologies of said sample; mapping a plurality of regions of the image corresponding to the one or more cell morphologies; compressing or deconverting at least a subset of the plurality of regions into a plurality of compressed or deconverted image segments; determining a respective generative adversarial network (GAN) model corresponding to a respective cell morphology associated with each compressed or deconverted image segment of the plurality of compressed or deconverted image segments; assigning each of the GAN models to each of the compressed or deconverted image segments; receiving a request to view the image from a first client device; In response to receiving the request to view the image from the first client device, transmitting (i) the plurality of compressed or deconverted image segments and (ii) the respective GAN models assigned to the plurality of compressed or deconverted image segments to the first client device; A method comprising:

53. In the server system, constructing a map of the GAN models assigned to each of the plurality of compressed or deconverted image segments, wherein segments of the map of the GAN models are linked to corresponding image segments of the plurality of compressed or deconverted image segments. further comprising wherein transmitting each GAN model comprises transmitting the map of each GAN model.

10. A method according to any one of the preceding claims.

54. In the server system, compressing using a lossless compression algorithm or preserving the original resolution of at least one of said plurality of regions; continuing to determine and assign a respective GAN model to the at least one region of the plurality of regions; In response to receiving the request to view the image from the first client device, (iii) sending the at least one region compressed with the lossless compression algorithm or with the original resolution maintained to the first client device algorithm; 10. The method of any one of the preceding claims, further comprising:

55. In the server system, assigning respective diagnostic or therapeutic relevance levels to said plurality of regions; determining that the at least one region of the plurality of regions meets a threshold of diagnostic or therapeutic relevance; determining that the subset of the plurality of regions does not meet the threshold of diagnostic or therapeutic relevance; further comprising compressing using a lossless compression algorithm or maintaining the original resolution of the at least one region of the plurality of regions is responsive to the determination that the at least one region of the plurality of regions meets the diagnostic or therapeutic relevance threshold; compressing or deconverting the subset of the plurality of regions and assigning the respective GAN models to the respective compressed or deconverted image segments is pursuant to the determination that the subset of the plurality of regions does not meet the threshold of diagnostic or therapeutic relevance.

55. The method of claim 54.

56. 10. The method of claim 1, wherein identifying the one or more cellular morphologies of the sample comprises compiling a cellular index of features of the image using a predetermined library of tissue-specific or pathology-specific neural networks.

57. 10. The method of claim 1, wherein the compressing or deconverting step includes partially deconverting the subset of the plurality of regions into a pixel-shifted retro source image layer for pixel-shift super-resolution upon subsequent recombination at the first client device.

58. before receiving the request to view the image from the first client device; using each of the GAN models to decompress or super-resolve the subset of regions into multiple reconstruction regions; comparing the plurality of reconstructed regions with pre-compressed or pre-deconverted versions of a subset of the regions; determining a difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region based on the comparing; 10. The method of any one of the preceding claims, further comprising:

59. before receiving the request to view the image from the first client device; determining that the difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region meets a threshold; updating the respective GAN model based on the determination that the difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region satisfies the threshold; recompressing or re-deconverting the subset of the plurality of regions using each of the updated GAN models; further comprising transmitting the plurality of compressed or deconverted image segments includes transmitting the recompressed or redeconverted subset of the plurality of regions.

59. The method of claim 58.

60. before receiving the request to view the image from the first client device; determining that the difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region does not meet the threshold; transmitting the plurality of compressed or deconverted image segments is pursuant to the determination that the difference between the reconstructed region and the pre-compressed or pre-deconverted version of the subset of the region does not satisfy the threshold.

59. The method of claim 58.

61. In the server system, storing the plurality of compressed or deconverted image segments and the respective GAN models assigned to the plurality of compressed or deconverted image segments; deleting the image before receiving the request to view the image from the first client device; 10. The method of any one of the preceding claims, further comprising:

62. In the server system, packaging the plurality of compressed or deconverted image segments and the respective GAN models assigned to the plurality of compressed or deconverted image segments into a file wrapper; further comprising transmitting the plurality of compressed or deconverted image segments and the respective GAN models assigned to the plurality of compressed or deconverted image segments to the first client device includes transmitting the file wrapper to the first client device.

10. A method according to any one of the preceding claims.

63. On the first client device, receiving the plurality of compressed or deconverted image segments and the respective GAN models assigned to the plurality of compressed or deconverted image segments from the server system; decompressing or super-resolving the compressed or deconverted image segments using the respective GAN models assigned to the plurality of compressed or deconverted image segments; combining the decompressed or super-resolved image segment with a reconstructed version of the image or requested portion thereof; displaying portions of the reconstructed version of the image on a display integrated with or communicatively coupled to the first client device; 10. The method of any one of the preceding claims, further comprising:

64. one or more processors on the server or client device; instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods of claims 50 to 63. and memory to store A system comprising:

65. Instructions which, when executed by a server or a client device, cause said server or said client device to perform any of the methods of claims 52 to 63. A non-transitory computer-readable storage medium for storing the

66. 1. A method for processing and transmitting images for diagnostic analysis, comprising: The method comprises:

1. A server system including one or more processors, acquiring an input image of the sample; globally down-converting the input image to a down-resolved image; After globally down-converting the input image to the down-converted image, simultaneously: globally up-converting the down-converted image into an up-converted image using a generative adversarial network (GAN) model configured to reconstruct an image including features corresponding to the specimen; classifying a plurality of regions of the down-converted image based on cell morphology and / or diagnostic relevance; communicating the upconverted image to a communications network for delivery to a client device; A method comprising:

67. further comprising dividing the input image into a plurality of layers; globally down-converting the input image includes down-converting each of the plurality of layers; the step of totally up-converting the down-converted image includes up-converting each of the plurality of layers.

10. A method according to any one of the preceding claims.

68. 10. The method of claim 1, wherein the step of globally up-converting the down-converted image comprises using the GAN model to predictively improve clarity of the down-converted image.

69. 10. The method of claim 1, wherein the step of globally up-converting the down-converted image comprises restoring deleted pixels by predicting pixel values ​​corresponding to the deleted pixels using the GAN model.

70. 10. The method of claim 1, wherein the step of globally upconverting the downconverted image comprises overwriting deconverted pixel values ​​with pixel values ​​predicted by the GAN model.

71. compressing the upconverted image using a run-length encoding scheme before transmitting the upconverted image to the communications network.

10. The method of any one of the preceding claims, further comprising:

72. manipulating a portion of the input image for subsequent processing based on the classification of the plurality of regions.

10. The method of any one of the preceding claims, further comprising:

73. The subsequent processing down-converting the input image again in its entirety, including the manipulated portion; Simultaneously, globally up-convert and classify a plurality of regions of the again globally down-converted image.

73. The method of claim 72, comprising:

74. one or more processors on the server or client device; instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods of claims 66 to 73. and memory to store A system comprising:

75. Instructions which, when executed by a server or a client device, cause said server or said client device to perform any of the methods of claims 66 to 73. A non-transitory computer-readable storage medium for storing the

76. 1. A method for processing and transmitting images for diagnostic analysis, comprising: The method comprises:

1. A server system including one or more processors, acquiring an input image of the sample, the input image comprising image data representing a flattened z-stack; classifying spectral differences of a plurality of features of the input image; assigning z-levels of the z-stack to each of the plurality of features based on the classifying, the z-stack comprising assigning one or more first z-levels to a first subset of the plurality of features and one or more second z-levels to a second subset of the plurality of features, the one or more first z-levels being below the one or more second z-levels, thereby obscuring portions of the first subset of the plurality of features; predicting pixel values ​​associated with the obscured portions of the first subset of features using a generative adversarial network (GAN) model configured to reconstruct image features; generating three-dimensional (3D) image data that includes the predicted pixel values ​​and that includes image data from the one or more first z-levels and the one or more second z-levels, thereby representing a virtually reconstructed 3D z-stack; providing the generated 3D image data for display on a client device; A method comprising:

77. generating the 3D image data, selecting a plurality of pixel values ​​across a plurality of said z-levels, the plurality of pixel values ​​including at least a portion of said predicted pixel values ​​that satisfy a predetermined sharpness threshold; replacing pixel values ​​corresponding to obscured pixels with the selected pixel values; 10. The method of any one of the preceding claims, comprising:

78. generating the 3D image data, selecting a plurality of pixel values ​​across a plurality of said z-levels, the plurality of pixel values ​​comprising at least a portion of said predicted pixel values ​​meeting a predetermined threshold of diagnostic or therapeutic relevance; replacing pixel values ​​corresponding to obscured pixels with the selected pixel values; 10. The method of any one of the preceding claims, comprising:

79. 10. The method of claim 1, wherein classifying the spectral differences comprises classifying the feature boundaries based on which spectral portion is most dominant.

80. 10. The method of claim 9, wherein providing the generated 3D image data for display comprises approximating navigation through a z-field comprising the z-stack by mapping multiple z-levels of the z-stack to respective control levels associated with control user input elements at the client device.

81. 81. The method of claim 80, wherein the control user input element is a slider, a knob, a zoom control, or a z-field navigation control.

82. 81. The method of claim 80, wherein approximating navigation through the z-stack is triggered after a zoom threshold is met.

83. 10. The method of claim 1, wherein generating the 3D image data comprises generating a virtual slide or a non-planar virtual surface at an angle that bisects a plurality of the z-levels.

84. one or more processors on the server or client device; instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods of claims 76 to 83. and memory to store A system comprising:

85. Instructions which, when executed by a server or a client device, cause said server or said client device to perform any of the methods of claims 76 to 83. A non-transitory computer-readable storage medium for storing the

Images